Atmospheric Toolbox

Problem with HARP/Python within a loop

Hi! i’m getting an error message in python when i want to extract the data from a harp imported product into an array, i dont know why this is happening. This is the line with the error:
Z = np.array(SCI_CH4_filt.CH4_column_number_density.data)
AttributeError: Object 'Product' does not have attribute 'CH4_column_number_density'

SCI_CH4_filt is the product I get after applying filters and bin_spatial() with harp. Inside it has the CH4_column_number_density field with the data that I am interested in extracting.

The reason I am totally lost is because the exact same line does not throw any errors and works perfectly within a test code for a single file. The error arises when I run the same code (exactly the same) but inside a for loop to extract data from multiple files, one at a time. The error arises for all files.

Has anyone encountered this problem maybe? Perhaps this is not the place for the query, if that is the case I will delete the comment of course, thank you.

Can you maybe show the full code that you are running in the for loop?

And normally Python just stops when an error occurs, so I am guessing you are only getting this error for the first product where this happens (not all of them). Unless you are using a try/except block to catch the error and continue.

Yes, here I paste the code :slightly_smiling_face:. I’m using try/except because every so often a file throws a NoDataError when calling the CH4 dataset to apply filters. But when I test this code it throws me the except message for all the files, that’s why I had to test without the try/except block to see what is happening (and there I found the AttributeError)

for i in SCIAfiles:
    #  try:  # puede haber archivos sin datos (NoDataError)

SCI_CH4 = harp.import_product(i, options='dataset=nad_ir1_ch4')  

product = coda.open(i)  # abro el mismo archivo desde coda

SCI_CH4.err_CH4 = harp.Variable(coda.fetch(product, "nad_ir1_ch4",-1,"non_linear_fit_param_err",0), ["time"]) 
SCI_CH4.err_CO2 = harp.Variable(coda.fetch(product, "nad_ir1_ch4",-1,"non_linear_fit_param_err",1), ["time"]) 
SCI_CH4.err_H2O = harp.Variable(coda.fetch(product, "nad_ir1_ch4",-1,"non_linear_fit_param_err",2), ["time"])  
coda.close(product) 

SCI_CH4_filt = harp.execute_operations(SCI_CH4,
                                       operations='scan_direction_type == "forward";'
                                                  'latitude > -59 [degree_north];'
                                                  'latitude < -14 [degree_north];'
                                                  'longitude > -80 [degree_east];'
                                                  'longitude < -47 [degree_east];'
                                                  'CH4_column_number_density_validity=&49152;'
                                                  'err_CH4>=0;err_CO2>=0;err_H2O>=0;'
                                                  'err_CH4<0.005;err_CO2<0.01;'
                                                  'keep(longitude_bounds, latitude_bounds,
                                                  'CH4_column_number_density);'
                                                  'bin_spatial(86,-58,0.5,62,-79,0.5)')

# Genero una grilla/array
x = np.linspace(-48, -79, 62)  # (lon_left, lon_right, ncols)
y = np.linspace(-15, -58, 86)  # (lat_bottom, lat_top, nrows)
X, Y = np.meshgrid(x, y)

# Guardo los datos de CH4 en un array
Z = np.array(SCI_CH4_filt.CH4_column_number_density.data)

The code continues, what I do is bring the lat/lon grid and the CH4 data to a GeoTIFF. In order not to take up so much space here I copy this first part which is the one that throws me the error in the last line. The strange thing is that exactly the same code tested in a single file (that is, not inside the for loop) works fine. But inside the for loop that same file also throws an error, that’s why I don’t understand.

I am using HARP 1.13 (the latest) in PyCharm. Yesterday I decided to try using harp.export_product() to directly export the SCI_CH4_filt product as a NetCDF, it worked with the first test files but now when I want to do it it throws me another error :joy::

in export_product raise CLibraryError()
harp._harppy.CLibraryError: Permission denied (D:/Tesina/SCIAMACHY/SCIAMACHY_L3/L3_diarios_py/SCI_CH4_L3_20020802_011749.nc)

But I didn’t change any line! so, I don’t know why it stoped working haha.
I appreciate any help or recommendation!

Hello everyone! I was able to fix it, I removed the latitude longitude filter at the beginning (inside harp.execute_operations()) and it works. I don’t know exactly why, but the same problem appeared in R, apparently that filter leaves me the harp product empty, that’s why I couldn’t call the CH4_column_number_density data.

As for the other error (harp._harppy.CLibraryError: Permission denied), it was simply a working directory issue (I assigned the folder where I wanted to save the NetCDF as the working directory and voila).

I leave this here in case it helps someone else, as well as reading other topics on this forum has helped me a lot! :muscle:t4: