I want to regrid S5P PAL AOT product using harp and the support for the same was extended in the recent harp 1.19 release. But when I execute the below commands in the command line, I get ‘segmentation fault’ error.
harpmerge -a “aerosol_optical_depth_validity>49;keep(latitude_bounds,longitude_bounds,datetime_start,datetime_length,aerosol_optical_depth);derive(datetime_stop {time} [days since 2000-01-01]);derive(datetime_start [days since 2000-01-01]);bin_spatial(42,30,1,70,-25,1);derive(latitude {latitude});derive(longitude {longitude});” S5P_PALAER.nc test.nc
When I traced the error, it points to the following line in the package
File “/opt/tools/harp/1.19/lib/python3.7/site-packages/harp/_harppy.py”, line 1256 in import_product
Am I doing something wrong here? I executed the below commands for few files in the period January 2020. To be specific, over certain files during 1st and 2nd Jan,2020 that overlap over Europe.
I also tried doing it the pythonic way using conda harp installation in jupyter-notebook but the execution of harp.import_product() for PAL_AER_OT products seem to kill the notebook kernel.
Could someone please advice.
This is a memory issue that recently got fixed. This fix will be included in the next HARP release.
For the moment you can work around the issue by excluding the qa value from your ingestion by using exclude(aerosol_optical_depth_validity) in your HARP import operations.
Hi Sander,
Thank you for the response.
So I tried the above suggested work around for excluding the aerosol_optical_depth_validity variable, but it doesn’t seem to work.
harpmerge -a “aerosol_optical_depth_validity>49;keep(latitude_bounds,longitude_bounds,datetime_start,datetime_length,aerosol_optical_depth);derive(datetime_stop {time} [days since 2000-01-01]);derive(datetime_start [days since 2000-01-01]);exclude(aerosol_optical_depth_validity);bin_spatial(42,30,1,70,-25,1);derive(latitude {latitude});derive(longitude {longitude})”.
gives ‘Segmentation Fault’. But yes, if I remove the initial filtering of qa_value (aerosol_optical_depth_validity>49) it works without issues. So if my understanding is right, we currently cannot filter based on qa values?
You cannot exclude the variable and try to filter on it. The workaround is to not use the variable at all (i.e. it means that you can’t perform any quality flag filtering with HARP for now). The idea is to put the exclude operation at the start as a general recommendation.
In your specific case, you should remove the aerosol_optical_depth_validity filter (and you can then also remove the exclude operation, since your keep operation already excludes it).
Okay thanks, that makes it clear.