I have a question regarding the use of averaging kernels with NO2 products in HARP.
The S5P ground processing chain for NO2 L2 products uses an Apriori NO2 profile from TM5 model. This model has a ground resolution of 1°x1° which can lead to underestimation of NO2 column in areas where NO2 concentration is higher. The product user manual (page 22) and experts advise to use more precise apriori profiles like the CAMS-regional European forecast for example. This model has a ground resolution of 0.1°x0.1° on 8 altitude levels. Here is what I want to achieve:
Both images are from the presentation of Henk Eskes on NO2 satellite applications during the second joint school on Atmospheric composition at which you were also a trainer if I am not wrong.
Before using CAMS model data and S5P products altogether, I wanted to reproject S5P NO2 products on a 0.1°x0.1° grid using bin_spatial() and also regrid the S5P vertical levels from the 34 TM5 pressure layers to 8 altitude levels with derive(altitude {latitude,longitude,vertical}) and a regrid() operation. However bin_spatial() removes the tropospheric_NO2_column_number_density_avk from the imported product. I tried to use a derive(tropospheric_NO2_column_number_density_avk {latitude,longitude,vertical}) to keep it within the imported product and it does not work. The derive() operation on altitude does not work either. Moreover, the harpdump–list-derivations applied to the S5P L2 NO2 product does not offer the possibility of deriving the tropospheric averaging kernel or the altitude.
I also feel that I could use an operation like derive_smoothed_column() operation to achieve what I want. But it apparently needs to be used on collocated products which seems to indicate a previous use of harpcollocate on S5P L2 NO2 and CAMS model product. However, the use of harpcollocate is not completely clear to me and I could not find examples in the documentation.
Is there a way to perform what I want with HARP? E.g using the CAMS AQ forecast NO2 model profile as an apriori profile to recompute the TROPOMI NO2 tropospheric NO2 column? I am pretty sure there is but I have not been able to find my way through it yet. I am kindly asking for your support here.
What you are asking for is not a simple process. You won’t be able to do this with HARP alone in any case.
First. The way to do this is to bring the CAMS data onto the satellite grid. You need the AMF and AVK as they have been used for the retrieval, so you shouldn’t regrid them (not even considering that the meaning of a regridded AMF or AVK is a bit dubious).
You need to get a model grid that matches the time of the single orbit that you want to adapt.
You can either use the nearest in time (easiest approach, and most often used) or perform a temporal regridding of the nearest model grids.
For collocating in space you will need to extract a model profile for each satellite pixel. Again, you can take the profile from the nearest grid point, or you can bi-linearly interpolate the four surrounding grid point data to an averaged profile at the center position of the pixel.
You can theoretically use harpcollocate to find the nearest grid point for each satellite pixel by using flatten() to remove the latitude and longitude on the model data. You then have a flat list of grid points (each with its own lat/lon) on which you can perform a -nx point_distance with harpcollocate. However, this is likely going to be very slow. If you are just looking at a small area on the map and prefilter the satellite data to this area before collocating, then this may still be an option though.
Also, for this to work, you need to bring the CAMS data into HARP format. HARP provides import routines for the CAMS global data in GRIB format, but currently does not support CAMS regional data in either GRIB or netCDF format. So you would have to create such a converter yourself.
Once you have both the model profile and satellite pixel information, you need to regrid the model profile to a partial column profile on the satellite vertical pressure grid. If you have the collocation results from above, you could use the HARP regrid() operation using the collocation result file as parameter. Otherwise you will have to do this yourself. Note that the HARP regrid() operation doesn’t do anything to compensate for any surface pressure difference between the model and satellite grid. I also have my questions about the “simply modify one of the pressures to match the other” statement in the PUM. This is not advisable if you already have partial columns. You should probably do the replacement before going from concentrations to partial columns (or somehow adjust the height difference in the partial column values).
Finally, you would then have to perform the steps as provided in the formula yourself. HARP already provides all the quantities that you need from the S5P product. $A^{trop}$ = tropospheric_NO2_column_number_density_avk, $M^{trop}$ = tropospheric_NO2_column_number_density_amf, $N_{v}^{trop}$ = tropospheric_NO2_column_number_density. It is the $x_{m,l}'$ that is the tricky one to get.
Thank you so much for the fast and detailed reply. From what I had read in the PUM and from the NO2 presentation, I thought the process was much simpler than this. And now I am also figuring out that I was taking the problem from the wrong end when trying to bring S5P data to the CAMS grid.
For training purposes, I wanted to maybe develop an exercise on computing the tropospheric NO2 column using S5P tropospheric averaging kernel and air mass factor and CAMS AQ forecast profile. I am now realizing that it is maybe too ambitious.
Anyway, thank you again for taking some time to explain the correct way to do this to me. I might use it for the future and I definitely learned a lot from your reply.
I’m writing the similar code for deriving NO2 based on the results of WRF-Chem.
Here’s my steps:
1. Iterate through dates
2. Read s5p (TROPOMI L2) data by "Satpy"
3. Read wrf-chem files nearest to s5p swath
4. Calculate the AMFs
4.1 Interpolate wrf profiles to TM5-MP pressure
4.2 Average (area_weighted) profiles to s5p pixels by "xESMF"
4.3 Use LUT to calculate Box-AMFs:
bamf_clr and bamf_cld
4.4 Use Box-AMFs to calculate AMF
5. Calculate no2Trop
6. Save to NetCDF files
There’re still some bugs in the codes and I will open the GitHub repository once it’s published.
