If I am reading this correctly, http://stcorp.github.io/harp/doc/html/ingestions/S5P_L2_O3.html, the ring scale factor and the fitted root mean square are not extracted.
Would be very useful, and necessary for whoever wants to use HARP for S5P TOC studies.
In HARP we don’t ingest the ring_scale_factor or fitted_root_mean_square since these variables do not fall into any of the naming conventions in HARP (they are very algorithm specific and therefore cannot be ‘harmonised’).
For the ingestions that are builtin into HARP we have strict rules. First because of some special optimisations that we have when you perform filtering and use large datasets (the ingestions are quite complex internally to try to make them fast). But we also need to make sure that the ingestions are exemplary. So some types of quantities will never end up in there.
If this quality filtering approach for O3 would have been permanent, we might have solved this by hardcoding the filtering within the HARP import function (to be turned on as an ingestion option). But I know that there will be an update of the L2 product at some point that allows filtering based on just the qa_value again (although this may take a while), so we are dealing with a temporary situation.
HARP assumes that all filtering for S5P L2 can be done on the qa_value which is the original intent of that variable. So we are dealing with an anomalous situation.
The role of HARP is not to solve the temporary problems that are in products. Those should be solved within the products themselves.
There are currently two solutions. You can create your own conversion of the S5P L2 product into HARP format (performing the filtering yourself) and then continue on that HARP file using the HARP library to perform whatever HARP operation. This is already a valid approach to support any type of file format that is not supported as ingestion by the HARP software itself.
The second approach, which is currently more practical, is to patch the qa_value values in your O3 products such that they become 100/1.0 if they match this custom filter or 0 if they don’t (properly taking into account the scale factor attribute that is applicable on this variable). You can then ingest the products using HARP again and apply a filter on the validity variable as normal.
It shouldn’t be too difficult to create a python script that uses the HDF5 (or netCDF4) library to do patch the O3 products.
I know that this approach is currently used by Google EarthEngine to work around this issue.