HARP command line tool harpmerge | averaging monthly gridded files

Dear Sander,

I would like to ask as to the following, using the HARP 1.10 command line tools via linux.

I am creating daily files for S5P SO2 based on orbital files, using the simple:

spawn,“harpmerge -ap 'bin_spatial(”+sc(nlat+1)+","+sc(latmin)+","+sc(grid)+","+sc(nlon+1)+","+sc(lonmin)+","+sc(grid)+");derive(latitude {latitude});derive(longitude {longitude})’ -a ‘keep(latitude_bounds,longitude_bounds,"+strjoin(measorbit(0).variablename,’,’)+",cloud_fraction)’ “+harpfiles+” “+harpdir+“S5P_RegriddedData_”+date+”.nc"

The daily plots look as one would expect for SO2, gaps and noise but also the expected hotspots.

[the red spot at the bottom left is the Mt Etna outgassing plume]

Then, I use harpmerge again to make the daily files into monthly files, using:

spawn, “harpmerge -ap ‘squash(time,latitude,longitude,latitude_bounds,longitude_bounds));bin();exclude(count,weight)’ “+harpfiles+” “+harpdir+“S5P_RegriddedData_”+ndate(i)+”.nc”

This also appears to work fine since the monthly plots look all normal, or so it seems of course.

Then I thought to merge the monthly files into seasonal files, simply using:

spawn, “harpmerge -ap ‘squash(time,(latitude,longitude,latitude_bounds,longitude_bounds))’ “+harpfiles+” “+harpdir+“S5P_RegriddedData_”+title+”.nc”

This doesn’t seem to work and the result is simply the first month I include in the variable “harpfiles” and not the average of all three.

This plot is exactly the same as the above, the monthly, believe me, I’ve created difference plots as well.

In case you ask, since for SO2 we do not have data for all days for each month and for each grid point it would not be statistically 100% accurate to create the seasonal from the daily since the months that are more complete would be more represented in the outcome hence skewing the seasonal towards one of the months.

Any ideas as to why the seasonal doesn’t seem to work?

Many thanks,

You need to use the bin() operation, otherwise you end up with a time series of grids.

1 Like

I did try bin() as well, apologies for not mentioning it, as per:

spawn, “harpmerge -ap ‘squash(time,(latitude,longitude,latitude_bounds,longitude_bounds));bin()’ “+harpfiles+” “+harpdir+“S5P_RegriddedData_”+title+”_bin.nc”

Got the same result, see the difference… :slight_smile:

Also, in my seasonal files there is only one array for SO2 and not three.

sulfurdioxide_total_vertical_column(time=1, latitude=52, longitude=48);


Sander, I found the issue, it lay in the way the input files where passed onto the command line, something which was not clear to me from the description in the text, http://stcorp.github.io/harp/doc/html/harpmerge.html

I was erroneously introducing an array of directories, not a single one with wildcards for e.g., expecting the command to read in simultaneously all the files found in all directories.

Thanks for your patience!