THATS-DEEP: TEmperature and Humidity ATmospheric Sounding DEtail-Enhanced Product

Official implementation for the 3D Deep Neural Network (DNN) model described in the paper:
"Detail Enhancement of AIRS/AMSU Temperature and Moisture Profiles Using a 3D Deep Neural Network"
Published in Artificial Intelligence for the Earth Systems (2023).

Paper Authors: Adam B. Milstein (a), Joseph A. Santanello (b), and William J. Blackwell (a).
(a) MIT Lincoln Laboratory, Lexington, Massachusetts, USA (b) NASA Goddard Spaceflight Center, Greenbelt, Maryland

https://journals.ametsoc.org/view/journals/aies/2/2/AIES-D-22-0037.1.xml

POC for this code: Adam Milstein, milstein [at] ll [dot] mit [dot] edu

Spaceborne hyperspectral infrared and microwave sounders like AIRS and AMSU provide vital global atmospheric data. However, traditional Level 2 vertical profile retrievals often suffer from limited vertical resolution, smoothing errors, and noise. This leaves them unable to accurately represent sharp lower-troposphere thermodynamic structures.

This project uses a 3D Deep Neural Network (DNN) to reduce noise and sharpen structural details across entire atmospheric granules. By resolving key missing features—such as capping inversions at the top of the Planetary Boundary Layer (PBL)—the model significantly improves the accuracy of PBL height estimations over land.

The model treats atmospheric vertical levels as structural dimensions rather than independent channels:

• Inputs: Standard Level 2 temperature and moisture grids retrieved from AIRS/AMSU sounders. Specifically, the AIRS2SUP AIRS Level 2 Support Product Version 7, can be used with the provided code examples.
• Core Network: A 3D Convolutional Neural Network (built on a 3D U-Net style backbone) that captures spatial-vertical correlations.
• Outputs: Enhanced, noise-reduced 3D thermodynamic fields with physically consistent atmospheric gradients.

Ensure you have the required dependencies installed.

Three imported, open source Python packages are not included in the repo: pytorch, h5py, and numpy Add these to your environment using conda or pip.

Then, clone the repo:

# Clone the repository
git clone https://github.com/mit-ll/THATSDEEP.git

Then, change working directory to the UNET directory.

To run the model for inference on real AIRS/AMSU Level 2 operational products, an AIRS2SUP version 7 file, which is the name for the AIRS Level 2 support product, needed used as input. These are obtained from: https://www.earthdata.nasa.gov/data/catalog/ges-disc-airs2sup-7.0 and are HDF4 files. A saved model checkpoint will need to be loaded by the inference script, test_with_airs_inputs.py. For example, checkpoints/model_epoch_90.pth is the default that this script looks for. This checkpoint is available here as one of the asset files for release 1.0.0, as is a zipfile containing two example AIRS2SUP input files.

To train the model, the training set files is needed. These file, which are large Matlab matfiles, will be made available following a release review in the coming weeks. The training set files are: 'resized_images_era_56_48_32_A1.mat' and 'ANCILLARY_48.mat'. The first one contains the actual training images, namely preprocessed 3D granules of temperature and humidity as an ND-matlab array with a training/validation/testing split, while the second file contains corresponding additional variables such as surface pressure, the retrieval quality flags, land fraction, and zenith angles.

To train the 3D Deep Neural Network, using the preprocessed training dataset files (placed in relative path ../Data), run:

python trainer.py

To run the trained model on sample Level 2 sounder granules to enhance them:

python test_with_airs_inputs.py

The input files are expected in a subdirectory called AIRS2SUP.

The enhanced granules will be saved in a Matlab matfile, test_days.mat. The saved variables are: "temp_enhanced", "q_log_enhanced" (the log of water vapor mixing ratio in kg/kg), and "press_levels", the pressure levels for each position. The outputs use sigma levels like those used by ECMWF. (The bottom 56 levels of the 91-level grid are used.)

If you use this codebase or find the methodology helpful in your research, please cite the paper:

@article{milstein2023detail,
  title={Detail Enhancement of AIRS/AMSU Temperature and Moisture Profiles Using a 3D Deep Neural Network},
  author={Milstein, Adam B., Santanello, Joseph A., and Blackwell, William J.},
  journal={Artificial Intelligence for the Earth Systems},
  volume={2},
  number={2},
  pages={AIES-D-22-0037.1},
  year={2023},
  publisher={American Meteorological Society},
  doi={10.1175/AIES-D-22-0037.1}
}