In the United Arab Emirates (UAE), extreme weather events disrupt daily life, delaying flights, endangering transportation, and complicating urban planning. High daytime temperatures limit human activity outdoors, while dense nighttime fog is a frequent cause of severe and often fatal car crashes. Meanwhile, 2024 saw the heaviest precipitation event in the country in 75 years, causing severe floods in the usually dry region.
G42, a leading AI and cloud computing company based in UAE, developed a cutting-edge regional generative AI forecasting system, capable of predicting a wide range of meteorological events at a resolution of up to 200 m in the UAE. In this post, we show how G42 built this system by using NVIDIA GPUs and the Earth-2 platform.
Importance of regional forecasting in the UAE
In the UAE, regional weather forecasting plays a critical role in addressing the challenges posed by extreme weather events, such as severe rain, dense fog, dust storms, and intense heat, which can cause infrastructure damage and disrupt transportation, logistics, and public safety. Beyond these immediate concerns, localized weather forecasting is also essential for the energy transition, as it informs the integration of renewable energy sources into the grid and supports overall grid stability.
Global weather models lack the resolution needed to capture the fine-scale dynamics that impact specific regions, making tailored solutions indispensable. Regional weather models are used to produce higher-resolution forecasts for limited areas but are expensive and energy intensive to run.
To address this, G42 harnessed the NVIDIA Earth-2 platform to create an AI-driven regional forecasting pipeline. Developers at G42 developed a custom regional model for the UAE region using the CorrDiff architecture in Earth-2’s PhysicsNeMo framework for downscaling the global weather forecast data. Inception, the AI research and development company within G42 trained the model, while Space42 created the training data.
Connecting the custom CorrDiff to the pretrained NVIDIA FourCastNet global weather prediction model enabled G42 to produce country-wide predictions at a 2-km resolution in minutes, compared to hours using traditional numerical weather prediction methods.
Using a second variant of the CorrDiff model, the Inception developers also improved the resolution of the input data by another factor of 10. This variant, trained to output 200-m resolution weather data in and around the city of Abu Dhabi, is used as a second step in the forecast pipeline and enables hyperlocal forecasts at the city level. It captures the influence of geographic features such as Abu Dhabi’s coastal islands and urban development on the formation and behavior of local weather patterns, providing a better understanding of the complex interactions between terrain, urbanization, and weather.
For their use case, the G42 team adapted the Earth-2 NVIDIA Omniverse blueprint, which is a reference for building AI weather workflows from model training to interactive visualization. Keeping this end-to-end picture in mind helps converge on solutions that create value for G42’s customers.
G42’s AI downscaling pipeline
The local forecasting model for the UAE is trained and deployed on G42’s cluster of NVIDIA H100 GPUs, consisting of nodes of eight H100s each. CorrDiff training was performed on multiple nodes, using up to 64 GPUs, but this number can be scaled up or down based on GPU availability. The CorrDiff training recipe automatically scales to multiple GPUs and nodes and adapts to any number of GPUs up to the batch size used.
As the CorrDiff steps run independently, they can be executed in parallel on multiple GPUs. The distributed inference pipeline is built using the Earth2Studio Enterprise package.
Because diffusion models such as CorrDiff are relatively expensive to inference, we found that using one H100 GPU to inference FourCastNet and temporal interpolation together, while using seven H100 GPUs to handle CorrDiff, was the optimal ratio for this workflow. The distribution code makes use of the automated distribution capabilities of NVIDIA PhysicsNeMo and PyTorch, taking advantage of fast NVLink data transfers between the GPUs.
The AI forecast pipeline model accelerated by NVIDIA H100 GPUs produces one day of forecast in 170 GPU seconds using the nested 2-km and 200-m models, compared to the 960 CPU core hours needed to produce one day of the equivalent 200-m resolution of data with the Weather Research and Forecasting (WRF) model.
Data preparation for CorrDiff model training
The data required for training a CorrDiff model includes both coarse-resolution input and high-resolution target datasets. G42 used input data that consists of ERA5 reanalysis and generated high-resolution data through in-house simulations.
The training input data was sourced from ECMWF ERA5 reanalysis data at 0.25° spatial resolution, obtained from the Copernicus Climate Data Store. This dataset provides high-quality, global meteorological information and serves as the baseline input for training the model. The ERA5 data is also available at hourly intervals, ensuring consistency with the target data and proper temporal alignment.
The target data includes two sets of simulations at different spatial resolutions:
- 2 km covering the UAE and beyond
- 200 m for Abu Dhabi city
These datasets were simulated using the WRF model, which has been regionally adapted to enhance accuracy in local atmospheric conditions. The simulations span five years (2019–2023), with data available at hourly intervals. The WRF model is driven by boundary conditions from the Global Forecast System (GFS) Global Data Assimilation System (GDAS) datasets.
Training a custom CorrDiff dataloader in NVIDIA PhysicsNeMo
The UAE regional CorrDiff model was trained at a 2-km resolution for the entire UAE and at a 200-m resolution for Abu Dhabi city, enabling fine-scale weather forecasting tailored to both regional and urban environments. To train this model, G42 leveraged the CorrDiff training recipe in the PhysicsNeMo framework. This example is designed to enable anyone with high-resolution weather data to train their own customized generative downscaling model.
Weather data can arrive in a multitude of different structures, labeling and formats. The training example in PhysicsNeMo is designed to help enable users to integrate their own data pipes into the CorrDiff training recipe. Two dataloaders are included as examples:
- One to replicate the training for the Taiwan model from the original CorrDiff paper
- An implementation called CorrDiff-Mini, which uses a simple dataset and a smaller network to implement a lightweight CorrDiff.
Different dataloaders can be specified by editing a configuration file. The CorrDiff-Mini dataloader is a good starting point for implementing new custom dataloaders.
Creating a high-resolution regional forecast pipeline in Earth2Studio
G42 built the custom high-resolution weather forecast of the UAE using several steps, from data retrieval to inferencing with FourCastNet, interpolating to higher temporal resolution and downscaling with CorrDiff models. Here’s how to set up such a pipeline in Earth2Studio and how G42 customized their pipeline for their specific use case.
Data retrieval
A forecast starts from an analysis—the best estimate of the state of the atmosphere at a given moment.
In the G42 pipeline, the initial condition (IC) is retrieved from GFS as shown, but Earth2Studio also includes interfaces to other data sources such as ERA5.
from earth2studio.data import GFSdata = GFS() |
Forecasting
For global forecasting, G42 uses the FourCastNet (SFNO) model supported in Earth2Studio:
from earth2studio.models.px import SFNO?fc_model = SFNO.load_model(SFNO.load_default_package())? |
FourCastNet takes the initial state of the atmosphere as input and predicts the desired number of future states at 6-hour intervals with 0.25° of spatial resolution. For more information about installing Earth2Studio with SFNO support, see the Install topic in the Earth2Studio documentation.
Interpolation
The AI-based weather interpolation model receives FourCastNet’s output as a time-series forecast at a 6-hour temporal resolution and estimates the intermediate weather states to improve the resolution to 1 hour. In Earth2Studio, use the interpolation model by wrapping another forecast model.
from earth2studio.enterprise.models.px import ForecastInterpolation?interp_model = ForecastInterpolation.load_model(???? Package(interp_package_path, cache=False),???? fc_model=fc_model?) |
The interpolated model has an interface identical to any other forecast model, enabling it to be used in any workflow requiring a forecast model.
Downscaling
The CorrDiff model is loaded similarly to the forecast model, except using model packages created from the custom CorrDiff models trained in PhysicsNeMo.
from earth2studio.enterprise.models.dx.corrdiff import CorrDiffcorrdiff_model = CorrDiff.load_model( Package(corrdiff_package_path, cache=False) |
The code example outlines creating the model for a basic CorrDiff workflow, but Earth2Studio CorrDiff wrapper can also be customized to add more advanced features.
G42 used this to link together two nested CorrDiff models: the 2-km CorrDiff model for the UAE and the 200-m CorrDiff model for Abu Dhabi city and the surrounding areas. They also included in their model the calculation of a fog index, a heuristic indicator of the probability and thickness of fog, as well as a customized handling of precipitation to improve the quality of rain outputs.
Running the workflow
With all the components in place, run a workflow to put everything together and produce the local high-resolution forecast:
from earth2studio.run import diagnostic?from earth2studio.io.netcdf4 import NetCDF4Backend?io = NetCDF4Backend(output_file)?time = ["2025-03-17T00:00:00"]?timesteps = 168?diagnostic(???? time, timesteps, interp_model, corrdiff_model, data, io,???? output_prognostic=False?) |
The forecast start time is specified as 2025-03-17T00:00:00, along with the desired number of time steps to forecast. Each time step represents an hour, assuming interpolation is set to a 1-hour interval. For example, specifying 168 time steps generates a weather forecast covering one week at 1-hour intervals. The output is saved into a NetCDF4 format file.
Visualization
The Earth-2 Omniverse blueprint provides a reference implementation for building visualization pipelines using NVIDIA Omniverse SDK and microservices. Omniverse is a platform for building digital twins, integrating interactive 3D scenes with accelerated physics simulations and AI. It enables real-time collaboration across teams and data interoperability based on the OpenUSD file format.
Omniverse Kit was used to combine the fog and wind predictions generated by the UAE-specific CorrDiff models with 3D building information (Figure 2). G42 customized the reference application from the Earth-2 blueprint to build the interactive interface that enables users to explore global-scale atmospheric data.
Results
G42 pushed the limits of the Earth-2 stack by successfully training two custom versions of CorrDiff to achieve regional weather simulations of temperature, winds, humidity and precipitation at a 2-km resolution over UAE and its surroundings and at a 200-m resolution in the region near Abu Dhabi. They use these models to produce full end-to-end weather forecasts, demonstrating the platform’s ability to deliver regional forecasts.
CorrDiff has learned to reproduce at high resolution many of the characteristic features of the meteorology of the region. Figure 3 shows an example of the wind forecast at a 2-km resolution. CorrDiff shows the impact of buildings in Abu Dhabi city on slowing down the afternoon sea breeze.
In Figure 3, the color indicates the wind speed, while the streamlines show the wind direction.
Demonstrating the improvement in resolution as you zoom in from the 0.25° grid to the city scale, Figure 4 shows the input 2-m temperature (left) and the same variable generated by CorrDiff at two resolutions: 2 km per pixel (center) and 200 m per pixel (right). The red box in the center plot indicates the area shown in the 200-m view on the right.
While the 2-km simulation provides a comprehensive forecast for the entire UAE and its surroundings, the 200-m simulation zooms in and captures finer details around the city of Abu Dhabi and its vicinity.
G42’s customized CorrDiff implementation combines the outputs to produce a fog forecast at the city scale. The 200-m resolution highlights subtle variations in fog probability, critical for improving transportation safety, urban planning, and energy optimization in the region.
In April 2024, the UAE experienced its heaviest rainfall in 75 years with an accumulated rainfall exceeding 250 mm in less than 24 hours. A new version of the UAE CorrDiff model, optimized for precipitation output at a 2-km resolution, demonstrated an improved ability to capture the rainfall structure of this extreme weather event compared to its previous iteration.
The precipitation amounts generated by CorrDiff align broadly with observed data from the UAE, and G42 is conducting further analysis to evaluate the statistical properties of the simulated rainfall against the actual event. These advancements highlight the potential of CorrDiff to model rare and impactful weather phenomena with increasing precision.
Key benefits to governments and industry
Simulating weather events at high resolution is crucial for regional forecasting, offering direct benefits to local governments and industries, particularly in transportation, safety, and power grid stability. Timely access to these simulations enables authorities to take proactive measures to manage traffic, reduce accidents, and optimize public safety during hazardous weather conditions. Accurate weather forecasts at super-resolution also enable energy providers to anticipate demand fluctuations and maintain grid stability, especially during extreme temperature events.
With traditional simulations, achieving this level of detail would take considerable time and resources. However, using NVIDIA AI-powered technology significantly accelerates the process, enabling super-resolution simulations such as those generated by G42 based on Earth-2’s stack at unprecedented speeds. It’s now possible to provide actionable forecasts faster and more efficiently than ever before.
Combined with ensemble forecasting methods, the speed of AI-based regional forecasting also makes it possible to characterize a wide range of possible scenarios of future weather, increasing the probability of identifying the chance of an upcoming extreme event.
