ThinkOnward has developed a powerful, open-source machine learning model designed for geophysical applications. Learn more about how the model was developed and how you can access and use it.

In machine learning (ML), foundation models (foundation models) serve as the starting point for training an ML model to become artificial intelligence (AI). A foundation model is a large, pre-trained machine learning model that has been trained on vast, diverse datasets. Its purpose is to enable the model to learn from these extensive data sets. By fine-tuning a foundation model on specialized data, you can further refine its abilities and enhance its capabilities. In short, a well-developed foundation model can significantly boost the accuracy, speed, and cost-effectiveness of AI by providing a strong starting point for model development.

For upstream oil and gas, an area of keen interest is geophysics. The sheer volume and complexity of seismic data makes it an excellent candidate for transformation by ML models and AI.

Basis for the ThinkOnward geophysical foundation model (GFM)

• ThinkOnward has developed a geophysical foundation model that is based on foundation model development and addresses some critical points:
• A paper by Sheng et al. (2023) defines a seismic foundation model, which lays the groundwork for the first peer-reviewed geophysics-based foundation model.
• The seismic foundation model builds on the vision transformer masked autoencoder proposed by Meta in a paper by He et al. (2021).
• There are two key issues with the seismic foundation model:
  • Size of the required input images. Because of the architecture of the model, the seismic input images used to train the model must be 224 x 224 pixels, which are not realistic image sizes for 2D seismic data.
  • Masking method for seismic images used during pre-training of the model. Masking is the process of hiding or blocking parts of a data set to help a neural network focus on relevant information. The method in the seismic foundation model depends on the size of the images and is limited to square patches.

These issues are key because the fundamental component of a seismic data set is a seismic trace, a 1D representation (a line or “squiggle”) of the data recorded for one channel.

The ThinkOnward geophysical foundation model

For our geophysical foundation model, ThinkOnward adjusted the data masking process of the seismic foundation model to traces and adjusted the internal architecture of the model to accommodate traces (instead of the 224x 224-pixel patches) (Figure 1). With these changes, the ThinkOnward geophysical foundation model accepts seismic images of any size for pre-training and fine-tuning, while honoring the fundamental component of seismic data, the seismic trace.

With the trace-based masking and data loading pipeline (Figure 2), we trained the backbone (a model that extracts and encodes features from input data) on 450 synthetic seismic volumes, which were released as part of the ThinkOnward Patch the Planet Challenge. After pre-training the backbone, we then designed and built a regression head for the vision transformer so that we can fine-tune for a variety of downstream tasks, such as interpolating missing traces or identifying horizons.

Released on Huggingface (an AI community at https://huggingface.co/), the ThinkOnward geophysical foundation model is a powerful, open-source machine learning model designed for geophysical applications, with fully open access and open weights, making it easier than ever to integrate cutting-edge technology into your projects.

To learn more, watch our recent webinar

• Key Features of the geophysical foundation model: Learn how this public, open-access model can enhance your geophysical and machine learning work.
• Demonstration: See firsthand how to install and start using the geophysical foundation model for ThinkOnward projects, challenges, and your own needs.
• Real-World Applications: Discover the potential of the geophysical foundation model in solving complex geophysical problems and how you can incorporate it into your workflow.

Click here to view this webinar. 

References

He, K. et al. (2021). Masked Autoencoders Are Scalable Vision Learners.

Sheng, H. et al. (2023). Seismic Foundation Model (SFM): a new generation deep learning model in geophysics. 

AAPG IMAGE 2024 poster session; Geophysical Foundation Model: Unmasking the Traces