Hi,
I have just processed GOME-2 L1b data (reprocessed GOME-2 data from the recently published FDR) and wanted to combine and compare it with the results of QA4ECV. To do this, I first analysed the same L1B file with the same algorithm on my local PC (with harp v1.11) and then on an HPC cluster (harp v1.17) and got different results. I attached a plot of the course of the latitude for both versions.
Do the different harp versions merge the data differently?
Kind regards,
Christian
It looks like some sorting issue.
Can you let me know which L1b product you are showing? We might then be able to see if we can reproduce this.
Also, did you use an operations as part of the harp import? (if so, please provide them)
An examplary file would be
GOME_xxx_1B_M02_20080103043558Z_20080103062058Z_R_O_20201222112133Z_0300.nat
from the L1B fundamental data record (EUMETSAT Product Navigator).
I don’t use an operation, but I read/open the file as follows:
pro = harp.import_product(gome2fn, options='band=band-4')
lat = pro.latitude.data
lon = pro.longitude.data
I can’t seem to reproduce this with HARP 1.17 on my machine.
I have no clue what is going on here.
Do you have the same out-of-order problem if you plot datetime or index?
Are you also sure that you are using the exact same L1b file? Because the product consists of a sequence of records and if those records in the product are out of order, then harp will likely ingest them out of order in the same way.
I have just noticed that it is a problem between different Pandas versions (1.3.3 vs 1.5.2). When I finally merge the fit results from the DOAS analysis, it seems that pd.concat automatically sorts in the older version, even though this parameter is set to False per default. However, when I explicitly set this parameter in the new version, everything looks fine again.
I am very sorry to have caused you any inconvenience.
No problem. Glad you solved it.