Hi,
I’d like to merge .nc files with harp and am wondering if this code here is correct? Sometimes the mean for a month consists of null values which got me thinking. Is it the effect of very high QA value (75)?
Dear Kolanov,
I am assuming that you are merging S5P/TROPOMI NO2 data? then the validity > 75 is absolutely correct, see also recommendations here: S5P MPC Product Readme Nitrogen Dioxide (esa.int)
page 7.
I am thinking that, statistically, what you are aiming for is not quite appropriate. I would suggest that you first ingest all the orbital files within that month using:
product = harp.import_product(InFile, operations = 'tropospheric_NO2_column_number_density_validity>75;
latitude>=39.;latitude<=41.;longitude>=20.;longitude<=22;
derive(tropospheric_NO2_column_number_density[Pmolec/cm2]);
keep(latitude, longitude, latitude_bounds, longitude_bounds, tropospheric_NO2_column_number_density)
')
productlist.append(product)
where the list productlist now has all the valid points between the lat/lon I have chosen. And THEN you do a binning of everything within that month. If your month is a winter month, i.e. loads of clouds, you may not find a valid pixel within your narrow 0.01x0.01 grid.
average = harp.execute_operations(productlist,
operations = ‘bin_spatial(25,39.55,0.01,30,20.70,0.01);derive(longitude {longitude});derive(latitude {latitude})’,
post_operations = ‘bin();squash(time, (latitude,longitude));
exclude(latitude_bounds, longitude_bounds,latitude_bounds_weight,longitude_bounds_weight,latitude_weight,longitude_wei
ght,weight,count)’)
Recall that the gridding performed by harp is based on the central lat/lon of the valid pixels, it does not account of parts of the pixel overlapping the original TROPOMI pixel.
Good luck!
MariLiza
The best approach is actually not to ingest all data first, but let harp do the merging while you ingest all products. The reduce_operations parameter that was introduced in HARP 1.10 makes this very efficient. The official S5P AAI Siberia Smoke use case shows you how to do this.
Also, the comment of Mariliza on the gridding approach is not correct. HARP does actually take into account the overlap of each satellite pixel with each grid pixel. The ingested HARP product will need to contain latitude_bounds and longitude_bounds variables to achieve this, but that is the case when you ingest S5P products.
1 Like
Thank you very much for replies.
So I’m assuming it should look like this:
But for some reason I get an error with this and can’t figure out why.
What error are you getting?
An exception “Merging failed”.
@sander.niemeijer Thanks for letting me know I was incorrect in my assumption! Did harp always account for the pixel overlap or was this introduced in 1.14? in any case, kudos to you as ever.
Please remove the try/except. You need to provide the harp exception. This will likely already tell you the cause of the problem
This has been there since the first implementation.
Nevermind, I had a typo in harp method “import_products” instead of “import_product”.
Thank you very much for help.