As you know we are working with the S5P_PAL__L2__AER_OT data files. As the AOT is reported on five wavelengths, the validity parameter also has a second dimension, spectral, of 5.
Are you planning to introduce a way to filter the orbital files in such a way that each of the 5 AOTs is filtered with its equivalent validity parameter via harp? as you can imagine, if we just ask for *_validity>50 during ingestion, curious outputs are provided.
Of course we are using a second step, keeping the validity flag and post-filtering, so it is not a curcial issue right now. It might however be a useful feature to have if say someone wishes to ingest and spatiotemporally bin all in one go using harp.
To do this you will have to ingest the product 5 times individually for each wavelength, but you can actually already do this with the current HARP release:
harpdump -a 'index(spectral)==1;squash(spectral, (aerosol_optical_depth, aerosol_optical_depth_validity, single_scattering_albedo));aerosol_optical_depth_validity>50' S5P_PAL__L2__AER_OT_20231001T013131_20231001T031301_30910_03_010100_20231009T222210.nc
And with the just released HARP 1.20.1 version you can actually filter on wavelength:
harpdump -a 'wavelength==380;squash(spectral, (aerosol_optical_depth, aerosol_optical_depth_validity, single_scattering_albedo));aerosol_optical_depth_validity>50' S5P_PAL__L2__AER_OT_20231001T013131_20231001T031301_30910_03_010100_20231009T222210.nc
Excellent! Our sys admin is about to install 1.20.1 so we just need a little loop for the five wavelengths and we are good to go.