Problem with HARP/Python within a loop

HARP
1

Hi! i’m getting an error message in python when i want to extract the data from a harp imported product into an array, i dont know why this is happening. This is the line with the error:
Z = np.array(SCI_CH4_filt.CH4_column_number_density.data)
AttributeError: Object 'Product' does not have attribute 'CH4_column_number_density'

SCI_CH4_filt is the product I get after applying filters and bin_spatial() with harp. Inside it has the CH4_column_number_density field with the data that I am interested in extracting.

The reason I am totally lost is because the exact same line does not throw any errors and works perfectly within a test code for a single file. The error arises when I run the same code (exactly the same) but inside a for loop to extract data from multiple files, one at a time. The error arises for all files.

Has anyone encountered this problem maybe? Perhaps this is not the place for the query, if that is the case I will delete the comment of course, thank you.

2

Can you maybe show the full code that you are running in the for loop?

And normally Python just stops when an error occurs, so I am guessing you are only getting this error for the first product where this happens (not all of them). Unless you are using a try/except block to catch the error and continue.

3

Yes, here I paste the code :slightly_smiling_face:. I’m using try/except because every so often a file throws a NoDataError when calling the CH4 dataset to apply filters. But when I test this code it throws me the except message for all the files, that’s why I had to test without the try/except block to see what is happening (and there I found the AttributeError)

for i in SCIAfiles:

    #  try:  # puede haber archivos sin datos (NoDataError)

SCI_CH4 = harp.import_product(i, options='dataset=nad_ir1_ch4')  

product = coda.open(i)  # abro el mismo archivo desde coda

SCI_CH4.err_CH4 = harp.Variable(coda.fetch(product, "nad_ir1_ch4",-1,"non_linear_fit_param_err",0), ["time"]) 
SCI_CH4.err_CO2 = harp.Variable(coda.fetch(product, "nad_ir1_ch4",-1,"non_linear_fit_param_err",1), ["time"]) 
SCI_CH4.err_H2O = harp.Variable(coda.fetch(product, "nad_ir1_ch4",-1,"non_linear_fit_param_err",2), ["time"])  
coda.close(product) 

SCI_CH4_filt = harp.execute_operations(SCI_CH4,
                                       operations='scan_direction_type == "forward";'
                                                  'latitude > -59 [degree_north];'
                                                  'latitude < -14 [degree_north];'
                                                  'longitude > -80 [degree_east];'
                                                  'longitude < -47 [degree_east];'
                                                  'CH4_column_number_density_validity=&49152;'
                                                  'err_CH4>=0;err_CO2>=0;err_H2O>=0;'
                                                  'err_CH4<0.005;err_CO2<0.01;'
                                                  'keep(longitude_bounds, latitude_bounds,
                                                  'CH4_column_number_density);'
                                                  'bin_spatial(86,-58,0.5,62,-79,0.5)')

# Genero una grilla/array
x = np.linspace(-48, -79, 62)  # (lon_left, lon_right, ncols)
y = np.linspace(-15, -58, 86)  # (lat_bottom, lat_top, nrows)
X, Y = np.meshgrid(x, y)

# Guardo los datos de CH4 en un array
Z = np.array(SCI_CH4_filt.CH4_column_number_density.data)

The code continues, what I do is bring the lat/lon grid and the CH4 data to a GeoTIFF. In order not to take up so much space here I copy this first part which is the one that throws me the error in the last line. The strange thing is that exactly the same code tested in a single file (that is, not inside the for loop) works fine. But inside the for loop that same file also throws an error, that’s why I don’t understand.

I am using HARP 1.13 (the latest) in PyCharm. Yesterday I decided to try using harp.export_product() to directly export the SCI_CH4_filt product as a NetCDF, it worked with the first test files but now when I want to do it it throws me another error :joy::

in export_product raise CLibraryError()
harp._harppy.CLibraryError: Permission denied (D:/Tesina/SCIAMACHY/SCIAMACHY_L3/L3_diarios_py/SCI_CH4_L3_20020802_011749.nc)

But I didn’t change any line! so, I don’t know why it stoped working haha.
I appreciate any help or recommendation!

4

Hello everyone! I was able to fix it, I removed the latitude longitude filter at the beginning (inside harp.execute_operations()) and it works. I don’t know exactly why, but the same problem appeared in R, apparently that filter leaves me the harp product empty, that’s why I couldn’t call the CH4_column_number_density data.

As for the other error (harp._harppy.CLibraryError: Permission denied), it was simply a working directory issue (I assigned the folder where I wanted to save the NetCDF as the working directory and voila).

I leave this here in case it helps someone else, as well as reading other topics on this forum has helped me a lot! :muscle:t4:

2 Likes