Atmospheric Toolbox

OMI NO2 L3 Data Merge Issues

Hi Sander, I’m able to merge the OMI SO2 Data product easily in Python using the following code:

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsharao/Downloads/OMI_HCHO'
OutFolder = r'/Users/varsharao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        latitude = coda.fetch(pf, '/HDFEOS/GRIDS/OMI_Total_Column_Amount_SO2/Data_Fields/Latitude')
        latitude = latitude[:,0]
        longitude = coda.fetch(pf, '/HDFEOS/GRIDS/OMI_Total_Column_Amount_SO2/Data_Fields/Longitude')
        longitude = longitude[0,:]
        so2 = coda.fetch(pf, '/HDFEOS/GRIDS/OMI_Total_Column_Amount_SO2/Data_Fields/ColumnAmountSO2_PBL')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.latitude = harp.Variable(latitude, ["latitude"])
    product.latitude.unit = "degree_north"
    product.longitude = harp.Variable(longitude, ["longitude"])
    product.longitude.unit = "degree_east"
    product.so2_column_number_density = harp.Variable(so2, ["latitude", "longitude"])
    product.so2_column_number_density.unit = "DU"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin();squash(time, (latitude,longitude))")
OutFile =  OutFolder + File[19:23] + File [24:26] + ".nc"

harp.export_product(average, OutFile)

However, when I use similar code for the OMNO2d_003 product (NO2 L3) I have to remove lat/long because this isn’t included in the product for some reason and therefore I get the error:

runfile(’/Users/varsha.rao/Dropbox/untitled3.py’, wdir=’/Users/varsha.rao/Dropbox’)
Traceback (most recent call last):

File “/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py”, line 997, in _export_product
_export_variable(name, product[name], c_product)

File “/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py”, line 862, in _export_variable
raise Error(“dimensions missing or incomplete”)

Error: dimensions missing or incomplete

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File “/Users/varsha.rao/Dropbox/untitled3.py”, line 30, in
average = harp.execute_operations(productlist, post_operations=“bin();squash(time)”)

File “/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py”, line 1373, in execute_operations
_export_product(product, c_product_ptr[0])

File “/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py”, line 999, in _export_product
raise Error(“variable ‘%r’ could not be exported (%s)” % (name, str(_error)))

Error: variable ‘‘no2_column_number_density’’ could not be exported (dimensions missing or incomplete)

This is the code I’m running for the NO2 average:

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsha.rao/Downloads/2017_PeakLockdown/'
OutFolder = r'/Users/varsha.rao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        no2 = coda.fetch(pf, '/HDFEOS/GRIDS/ColumnAmountNO2/Data_Fields/ColumnAmountNO2TropCloudScreened')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.no2_column_number_density = harp.Variable(no2)
    product.no2_column_number_density.unit = "molec/cm2"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin();squash(time)")
OutFile =  OutFolder + File[19:23] + File [24:26] + ".nc"

harp.export_product(average, OutFile)

You still have to give the dimensions their names for the no2_column_number_density variable:

product.no2_column_number_density = harp.Variable(no2, ["latitude", "longitude"])

Also, you should leave out the squash operation if you don’t pass any variables as second parameter.

Hi Sander,

Thanks for your help this works but the issue is it creates an array with lat between 0-719 and long. between 0-1439 instead of the normal -180 - 180 and -90 to 90. Is there anyway to fix this in the python code? This issue prevents me from plotting in Panoply/QGIS as it’s not georeferenced so the plot isn’t on the globe. (In QGIS it was off to the side whereas in panoply it’s not a georeferenced plot.

Here is the code I’m using:

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsha.rao/Downloads/Jan2017_NO2/'
OutFolder = r'/Users/varsha.rao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        no2 = coda.fetch(pf, '/HDFEOS/GRIDS/ColumnAmountNO2/Data_Fields/ColumnAmountNO2TropCloudScreened')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.no2_column_number_density = harp.Variable(no2, ["longitude", "latitude"])
    product.no2_column_number_density.unit = "molec/cm2"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin()")
OutFile =  OutFolder + File[19:23] + File [24:26] + ".nc"

harp.export_product(average, OutFile)

This is because you are no longer including the latitude/longitude axis variables.
You should construct them manually. For instance:

product.latitude = harp.Variable(numpy.arange(-89.75, 89.75, 0.25), ["latitude"])
product.latitude.unit = "degree_north"
product.longitude = harp.Variable(numpy.arange(-179.75, 179.75, 0.25), ["longitude"])
product.longitude.unit = "degree_east"

I am not sure what the offset is for the OMI L3 grid, so you might have to start the grid at longitude 0 instead of -180.

Hi Sander

Using this code:

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsha.rao/Downloads/Research_3/NO2/2017/2017_PreLockdown/'
OutFolder = r'/Users/varsha.rao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        no2 = coda.fetch(pf, '/HDFEOS/GRIDS/ColumnAmountNO2/Data_Fields/ColumnAmountNO2TropCloudScreened')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.latitude = harp.Variable(numpy.arange(-89.75, 89.75, 0.25), ["latitude"])
    product.latitude.unit = "degree_north"
    product.longitude = harp.Variable(numpy.arange(-179.75, 179.75, 0.25), ["longitude"])
    product.longitude.unit = "degree_east"
    product.no2_column_number_density = harp.Variable(no2, ["longitude", "latitude"])
    product.no2_column_number_density.unit = "molec/cm2"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin()")
OutFile =  OutFolder + "Jan2017_NO2.nc"

harp.export_product(average, OutFile)

I get the following error:

runfile('/Users/varsha.rao/Dropbox/OMI_NO2_L3_Average.py', wdir='/Users/varsha.rao/Dropbox')
Traceback (most recent call last):

  File "/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py", line 997, in _export_product
    _export_variable(name, product[name], c_product)

  File "/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py", line 901, in _export_variable
    raise CLibraryError()

CLibraryError: dimension 0 (of type 'longitude') of variable 'no2_column_number_density' is incompatible with product; variable = 720, product = 1438 (/Users/jenkins/workspace/CondaPackages/conda3-osx-64/miniconda3/conda-bld/harp_1589906340121/work/libharp/harp-product.c:1300)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/varsha.rao/Dropbox/OMI_NO2_L3_Average.py", line 34, in <module>
    average = harp.execute_operations(productlist, post_operations="bin()")

  File "/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py", line 1373, in execute_operations
    _export_product(product, c_product_ptr[0])

  File "/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py", line 999, in _export_product
    raise Error("variable '%r' could not be exported (%s)" % (name, str(_error)))

Error: variable ''no2_column_number_density'' could not be exported (dimension 0 (of type 'longitude') of variable 'no2_column_number_density' is incompatible with product; variable = 720, product = 1438 (/Users/jenkins/workspace/CondaPackages/conda3-osx-64/miniconda3/conda-bld/harp_1589906340121/work/libharp/harp-product.c:1300))

Here’s a screenshot of what a averaged L3 file by NASA’s lat/long looks like:

You need to make sure that the arange output matches the headers in your table. This looks to be: numpy.arange(-89.875, 90, 0.25) and numpy.arange(-179.875, 180, 0.25)
Try to get familiar with Numpy. This is not really HARP specific.

Hi Sander, this returned the same error. I think this has to do with the latitude bounds. Longitude seems to go from -179.875 to 179.875 whereas latitude goes from -89.875 to 89.87501. As you can see in the image the latitude values are 0.00001 off of a round number.

This is an image of the opposite end of the array for the official NASA L3 averaged product.

It transitions from exact to 0.00001 off here:

The lat/lon dimensions are likely swapped for the NO2 grid, try: harp.Variable(no2, ["latitude", "longitude"])

Hi Sander, yes this was one issue. Then I had to squash time, long and lat. Here’s my code now which works well and I’ve compared it to the real data. I have two more questions but I will post them in a new thread.

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsha.rao/Downloads/Research_3/NO2/2017/2017_PreLockdown/'
OutFolder = r'/Users/varsha.rao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        no2 = coda.fetch(pf, '/HDFEOS/GRIDS/ColumnAmountNO2/Data_Fields/ColumnAmountNO2TropCloudScreened')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.latitude = harp.Variable(numpy.arange(-89.875, 90, 0.25), ["latitude"])
    product.latitude.unit = "degree_north"
    product.longitude = harp.Variable(numpy.arange(-179.875, 180, 0.25), ["longitude"])
    product.longitude.unit = "degree_east"
    product.no2_column_number_density = harp.Variable(no2, ["latitude", "longitude"])
    product.no2_column_number_density.unit = "molec/cm2"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin();squash(time, (latitude,longitude))")
OutFile =  OutFolder + "2017_PreLockdown_Avg.nc"

harp.export_product(average, OutFile)