Curious harpmerge issue with gridded data

HARP
1

Good morning Sander,

I am running python 3.11.7 with harp 1.23 on the operational EUMETSAT IASI data on our HPC system, linux. For the EUMETSAT ACSAF project.

I am creating daily gridded files [how original.]

However, some of them contain all the weights of all parameters, e.g.
[mariliza@aristotle6 SO2_yearly_gridded]$ ncdump -h /mnt/lapaero_b/groups/lap_aero/IASI_files/SO2/Output/SO2_daily_gridded/IASI-A/2019/IASI_SO2_IASI-A_20190604.nc
netcdf IASI_SO2_IASI-A_20190604 {
dimensions:
time = 1 ;
latitude = 720 ;
longitude = 1440 ;
spectral = 5 ;
variables:
double sza(time, latitude, longitude) ;
sza:units = “degree” ;
sza:_FillValue = NaN ;
double so2_bt(time, latitude, longitude) ;
so2_bt:units = “K” ;
so2_bt:_FillValue = NaN ;
double so2_vcd_alt(time, latitude, longitude, spectral) ;
so2_vcd_alt:units = “” ;
so2_vcd_alt:_FillValue = NaN ;
double so2_alt(time, latitude, longitude) ;
so2_alt:units = “” ;
so2_alt:_FillValue = NaN ;
double so2_vcd(time, latitude, longitude) ;
so2_vcd:units = “” ;
so2_vcd:_FillValue = NaN ;
float so2_alt_weight(time, latitude, longitude) ;
so2_alt_weight:_FillValue = NaNf ;
float so2_vcd_weight(time, latitude, longitude) ;
so2_vcd_weight:_FillValue = NaNf ;
double longitude(longitude) ;
longitude:units = “degree_east” ;
longitude:_FillValue = NaN ;
double latitude(latitude) ;
latitude:units = “degree_north” ;
latitude:_FillValue = NaN ;
float so2_vcd_alt_weight(time, latitude, longitude, spectral) ;
so2_vcd_alt_weight:_FillValue = NaNf ;

// global attributes:
:Conventions = “HARP-1.0” ;

While others do not

[mariliza@aristotle6 SO2_yearly_gridded]$ ncdump -h /mnt/lapaero_b/groups/lap_aero/IASI_files/SO2/Output/SO2_daily_gridded/IASI-A/2019/IASI_SO2_IASI-A_20190601.nc
netcdf IASI_SO2_IASI-A_20190601 {
dimensions:
time = 1 ;
latitude = 720 ;
longitude = 1440 ;
spectral = 5 ;
variables:
double sza(time, latitude, longitude) ;
sza:units = “degree” ;
sza:_FillValue = NaN ;
double so2_bt(time, latitude, longitude) ;
so2_bt:units = “K” ;
so2_bt:_FillValue = NaN ;
double so2_vcd_alt(time, latitude, longitude, spectral) ;
so2_vcd_alt:units = “” ;
so2_vcd_alt:_FillValue = NaN ;
double so2_alt(time, latitude, longitude) ;
so2_alt:units = “” ;
so2_alt:_FillValue = NaN ;
double so2_vcd(time, latitude, longitude) ;
so2_vcd:units = “” ;
so2_vcd:_FillValue = NaN ;
float so2_alt_weight(time, latitude, longitude) ;
so2_alt_weight:_FillValue = NaNf ;
float so2_vcd_weight(time, latitude, longitude) ;
so2_vcd_weight:_FillValue = NaNf ;
double longitude(longitude) ;
longitude:units = “degree_east” ;
longitude:_FillValue = NaN ;
double latitude(latitude) ;
latitude:units = “degree_north” ;
latitude:_FillValue = NaN ;

// global attributes:
:Conventions = “HARP-1.0” ;

I.e. so2_vcd_alt_weight is missing. As a result, I cannot merge them into monthly, seasonal. annual files using the command line harpmerge as it complains, for e.g.
(harp-env) [mariliza@aristotle4 NewPrograms]$ harpmerge /mnt/lapaero_b/groups/lap_aero/IASI_files/SO2/Output/SO2_daily_gridded/IASI-A/2019/201906.nc output.nc
ERROR: products don’t both have variable ‘so2_vcd_alt_weight’ (while merging ‘/mnt/lapaero_b/groups/lap_aero/IASI_files/SO2/Output/SO2_daily_gridded/IASI-A/2019/IASI_SO2_IASI-A_20190604.nc’)

The daily files are run at the same time, with the same script, the timestamps prove this.

Any thoughts?

Many thanks as always,
MariLiza

2

What is strange to me is that the products don’t contain a variable called weight (for the overall weights). Did you perhaps explicitly remove this in your operations?

Note that the variable-specific xxx_weight variable only appears if the underlying variable xxx has NaN values in the spatial binning. This xxx_weight variable then contains the sum of the weights for which xxx is not NaN (which will be a lower sum than for the overall weight variable).

HARP supports merging products where one product contains a variable-specific weight variable and the other doesn’t, but then it requires the global weight variable to be available. It then automatically creates an xxx_weight variable equal to the weight variable for this.

So, there are two solutions:

Either pre-filter your data to remove NaN values before performing the spatial binning. This will prevent the generation of xxx_weight variables in the first place. And you probably don’t want to include NaN values in your binning anyway.

Or, make sure you keep the global weight variable and let HARP ‘do its thing’ automatically for the merge.

1 Like
3

Oh dear! Well, your first option cannot work for this dataset, I’m afraid, for various IASI-related issues that are of no interest to this forum. I’ll go with option #2. My mistake for excluding the weight from the original daily files, I did not think of it.
Thanks as ever!