OMI NO2 L3 Data Merge Issues

HARP
1

Hi Sander, I’m able to merge the OMI SO2 Data product easily in Python using the following code:

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsharao/Downloads/OMI_HCHO'
OutFolder = r'/Users/varsharao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        latitude = coda.fetch(pf, '/HDFEOS/GRIDS/OMI_Total_Column_Amount_SO2/Data_Fields/Latitude')
        latitude = latitude[:,0]
        longitude = coda.fetch(pf, '/HDFEOS/GRIDS/OMI_Total_Column_Amount_SO2/Data_Fields/Longitude')
        longitude = longitude[0,:]
        so2 = coda.fetch(pf, '/HDFEOS/GRIDS/OMI_Total_Column_Amount_SO2/Data_Fields/ColumnAmountSO2_PBL')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.latitude = harp.Variable(latitude, ["latitude"])
    product.latitude.unit = "degree_north"
    product.longitude = harp.Variable(longitude, ["longitude"])
    product.longitude.unit = "degree_east"
    product.so2_column_number_density = harp.Variable(so2, ["latitude", "longitude"])
    product.so2_column_number_density.unit = "DU"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin();squash(time, (latitude,longitude))")
OutFile =  OutFolder + File[19:23] + File [24:26] + ".nc"

harp.export_product(average, OutFile)

However, when I use similar code for the OMNO2d_003 product (NO2 L3) I have to remove lat/long because this isn’t included in the product for some reason and therefore I get the error:

runfile(’/Users/varsha.rao/Dropbox/untitled3.py’, wdir=’/Users/varsha.rao/Dropbox’)
Traceback (most recent call last):

File “/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py”, line 997, in _export_product
_export_variable(name, product[name], c_product)

File “/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py”, line 862, in _export_variable
raise Error(“dimensions missing or incomplete”)

Error: dimensions missing or incomplete

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File “/Users/varsha.rao/Dropbox/untitled3.py”, line 30, in
average = harp.execute_operations(productlist, post_operations=“bin();squash(time)”)

File “/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py”, line 1373, in execute_operations
_export_product(product, c_product_ptr[0])

File “/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py”, line 999, in _export_product
raise Error(“variable ‘%r’ could not be exported (%s)” % (name, str(_error)))

Error: variable ‘‘no2_column_number_density’’ could not be exported (dimensions missing or incomplete)

This is the code I’m running for the NO2 average:

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsha.rao/Downloads/2017_PeakLockdown/'
OutFolder = r'/Users/varsha.rao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        no2 = coda.fetch(pf, '/HDFEOS/GRIDS/ColumnAmountNO2/Data_Fields/ColumnAmountNO2TropCloudScreened')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.no2_column_number_density = harp.Variable(no2)
    product.no2_column_number_density.unit = "molec/cm2"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin();squash(time)")
OutFile =  OutFolder + File[19:23] + File [24:26] + ".nc"

harp.export_product(average, OutFile)
2

You still have to give the dimensions their names for the no2_column_number_density variable:

product.no2_column_number_density = harp.Variable(no2, ["latitude", "longitude"])

Also, you should leave out the squash operation if you don’t pass any variables as second parameter.

3

Hi Sander,

Thanks for your help this works but the issue is it creates an array with lat between 0-719 and long. between 0-1439 instead of the normal -180 - 180 and -90 to 90. Is there anyway to fix this in the python code? This issue prevents me from plotting in Panoply/QGIS as it’s not georeferenced so the plot isn’t on the globe. (In QGIS it was off to the side whereas in panoply it’s not a georeferenced plot.

Here is the code I’m using:

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsha.rao/Downloads/Jan2017_NO2/'
OutFolder = r'/Users/varsha.rao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        no2 = coda.fetch(pf, '/HDFEOS/GRIDS/ColumnAmountNO2/Data_Fields/ColumnAmountNO2TropCloudScreened')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.no2_column_number_density = harp.Variable(no2, ["longitude", "latitude"])
    product.no2_column_number_density.unit = "molec/cm2"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin()")
OutFile =  OutFolder + File[19:23] + File [24:26] + ".nc"

harp.export_product(average, OutFile)
5

This is because you are no longer including the latitude/longitude axis variables.
You should construct them manually. For instance:

product.latitude = harp.Variable(numpy.arange(-89.75, 89.75, 0.25), ["latitude"])
product.latitude.unit = "degree_north"
product.longitude = harp.Variable(numpy.arange(-179.75, 179.75, 0.25), ["longitude"])
product.longitude.unit = "degree_east"

I am not sure what the offset is for the OMI L3 grid, so you might have to start the grid at longitude 0 instead of -180.

6

Hi Sander

Using this code:

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsha.rao/Downloads/Research_3/NO2/2017/2017_PreLockdown/'
OutFolder = r'/Users/varsha.rao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        no2 = coda.fetch(pf, '/HDFEOS/GRIDS/ColumnAmountNO2/Data_Fields/ColumnAmountNO2TropCloudScreened')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.latitude = harp.Variable(numpy.arange(-89.75, 89.75, 0.25), ["latitude"])
    product.latitude.unit = "degree_north"
    product.longitude = harp.Variable(numpy.arange(-179.75, 179.75, 0.25), ["longitude"])
    product.longitude.unit = "degree_east"
    product.no2_column_number_density = harp.Variable(no2, ["longitude", "latitude"])
    product.no2_column_number_density.unit = "molec/cm2"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin()")
OutFile =  OutFolder + "Jan2017_NO2.nc"

harp.export_product(average, OutFile)

I get the following error:

runfile('/Users/varsha.rao/Dropbox/OMI_NO2_L3_Average.py', wdir='/Users/varsha.rao/Dropbox')
Traceback (most recent call last):

  File "/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py", line 997, in _export_product
    _export_variable(name, product[name], c_product)

  File "/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py", line 901, in _export_variable
    raise CLibraryError()

CLibraryError: dimension 0 (of type 'longitude') of variable 'no2_column_number_density' is incompatible with product; variable = 720, product = 1438 (/Users/jenkins/workspace/CondaPackages/conda3-osx-64/miniconda3/conda-bld/harp_1589906340121/work/libharp/harp-product.c:1300)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/varsha.rao/Dropbox/OMI_NO2_L3_Average.py", line 34, in <module>
    average = harp.execute_operations(productlist, post_operations="bin()")

  File "/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py", line 1373, in execute_operations
    _export_product(product, c_product_ptr[0])

  File "/Users/varsha.rao/opt/anaconda3/envs/visan/lib/python3.7/site-packages/harp/_harppy.py", line 999, in _export_product
    raise Error("variable '%r' could not be exported (%s)" % (name, str(_error)))

Error: variable ''no2_column_number_density'' could not be exported (dimension 0 (of type 'longitude') of variable 'no2_column_number_density' is incompatible with product; variable = 720, product = 1438 (/Users/jenkins/workspace/CondaPackages/conda3-osx-64/miniconda3/conda-bld/harp_1589906340121/work/libharp/harp-product.c:1300))

Here’s a screenshot of what a averaged L3 file by NASA’s lat/long looks like:

Screen Shot 2020-06-28 at 10.47.39 PM
Screen Shot 2020-06-28 at 10.47.39 PM1910×842 104 KB

7

You need to make sure that the arange output matches the headers in your table. This looks to be: numpy.arange(-89.875, 90, 0.25) and numpy.arange(-179.875, 180, 0.25)
Try to get familiar with Numpy. This is not really HARP specific.

8

Hi Sander, this returned the same error. I think this has to do with the latitude bounds. Longitude seems to go from -179.875 to 179.875 whereas latitude goes from -89.875 to 89.87501. As you can see in the image the latitude values are 0.00001 off of a round number.

Screen Shot 2020-06-28 at 11.31.57 PM
Screen Shot 2020-06-28 at 11.31.57 PM1894×842 39.9 KB

This is an image of the opposite end of the array for the official NASA L3 averaged product.

9

It transitions from exact to 0.00001 off here:

Screen Shot 2020-06-28 at 11.34.53 PM
Screen Shot 2020-06-28 at 11.34.53 PM2102×872 112 KB

10

The lat/lon dimensions are likely swapped for the NO2 grid, try: harp.Variable(no2, ["latitude", "longitude"])

11

Hi Sander, yes this was one issue. Then I had to squash time, long and lat. Here’s my code now which works well and I’ve compared it to the real data. I have two more questions but I will post them in a new thread.

import coda
import harp
import numpy
from os import listdir

#Local variables 

productlist = []
Folder = r'/Users/varsha.rao/Downloads/Research_3/NO2/2017/2017_PreLockdown/'
OutFolder = r'/Users/varsha.rao/Downloads/'
 
#Function for listing files
def list_files1(directory, extension):
    return (f for f in listdir(directory) if f.endswith('.' + extension))

#Adding files to a list
FileList = list_files1(Folder, "he5")
for File in FileList: 
    InFile = Folder + File
    pf = coda.open(InFile)
    try:
        no2 = coda.fetch(pf, '/HDFEOS/GRIDS/ColumnAmountNO2/Data_Fields/ColumnAmountNO2TropCloudScreened')
    finally:
        coda.close(pf)
    product = harp.Product()
    product.latitude = harp.Variable(numpy.arange(-89.875, 90, 0.25), ["latitude"])
    product.latitude.unit = "degree_north"
    product.longitude = harp.Variable(numpy.arange(-179.875, 180, 0.25), ["longitude"])
    product.longitude.unit = "degree_east"
    product.no2_column_number_density = harp.Variable(no2, ["latitude", "longitude"])
    product.no2_column_number_density.unit = "molec/cm2"
   
    productlist.append(product)
average = harp.execute_operations(productlist, post_operations="bin();squash(time, (latitude,longitude))")
OutFile =  OutFolder + "2017_PreLockdown_Avg.nc"

harp.export_product(average, OutFile)