# Curious harpmerge issue with gridded data
**URL:** https://forum.atmospherictoolbox.org/t/curious-harpmerge-issue-with-gridded-data/901
**Category:** HARP
**Tags:** python
**Created:** [April 4, 2025, 8:02am UTC](https://forum.atmospherictoolbox.org/t/curious-harpmerge-issue-with-gridded-data/901 "2025-04-04T08:02:31Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@mariliza](https://forum.atmospherictoolbox.org/u/mariliza)
#### Post date: [April 4, 2025, 8:02am UTC](https://forum.atmospherictoolbox.org/t/curious-harpmerge-issue-with-gridded-data/901/1 "2025-04-04T08:02:31Z")
Good morning Sander,
I am running python 3.11.7 with harp 1.23 on the operational EUMETSAT IASI data on our HPC system, linux. For the EUMETSAT ACSAF project.
I am creating daily gridded files [how original.]
However, some of them contain all the weights of all parameters, e.g.
[mariliza@aristotle6 SO2\_yearly\_gridded]$ ncdump -h /mnt/lapaero\_b/groups/lap\_aero/IASI\_files/SO2/Output/SO2\_daily\_gridded/IASI-A/2019/IASI\_SO2\_IASI-A\_20190604.nc
netcdf IASI\_SO2\_IASI-A\_20190604 {
dimensions:
time = 1 ;
latitude = 720 ;
longitude = 1440 ;
spectral = 5 ;
variables:
double sza(time, latitude, longitude) ;
sza:units = “degree” ;
sza:\_FillValue = NaN ;
double so2\_bt(time, latitude, longitude) ;
so2\_bt:units = “K” ;
so2\_bt:\_FillValue = NaN ;
double so2\_vcd\_alt(time, latitude, longitude, spectral) ;
so2\_vcd\_alt:units = “” ;
so2\_vcd\_alt:\_FillValue = NaN ;
double so2\_alt(time, latitude, longitude) ;
so2\_alt:units = “” ;
so2\_alt:\_FillValue = NaN ;
double so2\_vcd(time, latitude, longitude) ;
so2\_vcd:units = “” ;
so2\_vcd:\_FillValue = NaN ;
float so2\_alt\_weight(time, latitude, longitude) ;
so2\_alt\_weight:\_FillValue = NaNf ;
float so2\_vcd\_weight(time, latitude, longitude) ;
so2\_vcd\_weight:\_FillValue = NaNf ;
double longitude(longitude) ;
longitude:units = “degree\_east” ;
longitude:\_FillValue = NaN ;
double latitude(latitude) ;
latitude:units = “degree\_north” ;
latitude:\_FillValue = NaN ;
float **so2\_vcd\_alt\_weight** (time, latitude, longitude, spectral) ;
so2\_vcd\_alt\_weight:\_FillValue = NaNf ;
// global attributes:
:Conventions = “HARP-1.0” ;
While others do not
[mariliza@aristotle6 SO2\_yearly\_gridded]$ ncdump -h /mnt/lapaero\_b/groups/lap\_aero/IASI\_files/SO2/Output/SO2\_daily\_gridded/IASI-A/2019/IASI\_SO2\_IASI-A\_20190601.nc
netcdf IASI\_SO2\_IASI-A\_20190601 {
dimensions:
time = 1 ;
latitude = 720 ;
longitude = 1440 ;
spectral = 5 ;
variables:
double sza(time, latitude, longitude) ;
sza:units = “degree” ;
sza:\_FillValue = NaN ;
double so2\_bt(time, latitude, longitude) ;
so2\_bt:units = “K” ;
so2\_bt:\_FillValue = NaN ;
double so2\_vcd\_alt(time, latitude, longitude, spectral) ;
so2\_vcd\_alt:units = “” ;
so2\_vcd\_alt:\_FillValue = NaN ;
double so2\_alt(time, latitude, longitude) ;
so2\_alt:units = “” ;
so2\_alt:\_FillValue = NaN ;
double so2\_vcd(time, latitude, longitude) ;
so2\_vcd:units = “” ;
so2\_vcd:\_FillValue = NaN ;
float so2\_alt\_weight(time, latitude, longitude) ;
so2\_alt\_weight:\_FillValue = NaNf ;
float so2\_vcd\_weight(time, latitude, longitude) ;
so2\_vcd\_weight:\_FillValue = NaNf ;
double longitude(longitude) ;
longitude:units = “degree\_east” ;
longitude:\_FillValue = NaN ;
double latitude(latitude) ;
latitude:units = “degree\_north” ;
latitude:\_FillValue = NaN ;
// global attributes:
:Conventions = “HARP-1.0” ;
I.e. so2\_vcd\_alt\_weight is missing. As a result, I cannot merge them into monthly, seasonal. annual files using the command line harpmerge as it complains, for e.g.
(harp-env) [mariliza@aristotle4 NewPrograms]$ harpmerge /mnt/lapaero\_b/groups/lap\_aero/IASI\_files/SO2/Output/SO2\_daily\_gridded/IASI-A/2019/_201906_.nc output.nc
ERROR: products don’t both have variable ‘so2\_vcd\_alt\_weight’ (while merging ‘/mnt/lapaero\_b/groups/lap\_aero/IASI\_files/SO2/Output/SO2\_daily\_gridded/IASI-A/2019/IASI\_SO2\_IASI-A\_20190604.nc’)
The daily files are run at the same time, with the same script, the timestamps prove this.
Any thoughts?
Many thanks as always,
MariLiza
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [April 4, 2025, 10:19am UTC](https://forum.atmospherictoolbox.org/t/curious-harpmerge-issue-with-gridded-data/901/2 "2025-04-04T10:19:26Z")
What is strange to me is that the products don’t contain a variable called `weight` (for the overall weights). Did you perhaps explicitly remove this in your operations?
Note that the variable-specific `xxx_weight` variable only appears if the underlying variable `xxx` has NaN values in the spatial binning. This `xxx_weight` variable then contains the sum of the weights for which `xxx` is not NaN (which will be a lower sum than for the overall `weight` variable).
HARP supports merging products where one product contains a variable-specific weight variable and the other doesn’t, but then it requires the global `weight` variable to be available. It then automatically creates an `xxx_weight` variable equal to the `weight` variable for this.
So, there are two solutions:
Either pre-filter your data to remove NaN values before performing the spatial binning. This will prevent the generation of `xxx_weight` variables in the first place. And you probably don’t want to include NaN values in your binning anyway.
Or, make sure you keep the global `weight` variable and let HARP ‘do its thing’ automatically for the merge.
---
### Author:  [@mariliza](https://forum.atmospherictoolbox.org/u/mariliza)
#### Post date: [April 4, 2025, 1:29pm UTC](https://forum.atmospherictoolbox.org/t/curious-harpmerge-issue-with-gridded-data/901/3 "2025-04-04T13:29:09Z")
Oh dear! Well, your first option cannot work for this dataset, I’m afraid, for various IASI-related issues that are of no interest to this forum. I’ll go with option #2. My mistake for excluding the weight from the original daily files, I did not think of it.
Thanks as ever!