HARP command line tool harpmerge | averaging monthly gridded files

HARP
1

Dear Sander,

I would like to ask as to the following, using the HARP 1.10 command line tools via linux.

I am creating daily files for S5P SO2 based on orbital files, using the simple:

spawn,“harpmerge -ap 'bin_spatial(”+sc(nlat+1)+","+sc(latmin)+","+sc(grid)+","+sc(nlon+1)+","+sc(lonmin)+","+sc(grid)+");derive(latitude {latitude});derive(longitude {longitude})’ -a ‘keep(latitude_bounds,longitude_bounds,"+strjoin(measorbit(0).variablename,’,’)+",cloud_fraction)’ “+harpfiles+” “+harpdir+“S5P_RegriddedData_”+date+”.nc"

The daily plots look as one would expect for SO2, gaps and noise but also the expected hotspots.

sulfurdioxide_total_vertical_in_S5P_RegriddedData_20180803
sulfurdioxide_total_vertical_in_S5P_RegriddedData_20180803880×576 216 KB

[the red spot at the bottom left is the Mt Etna outgassing plume]

Then, I use harpmerge again to make the daily files into monthly files, using:

spawn, “harpmerge -ap ‘squash(time,latitude,longitude,latitude_bounds,longitude_bounds));bin();exclude(count,weight)’ “+harpfiles+” “+harpdir+“S5P_RegriddedData_”+ndate(i)+”.nc”

This also appears to work fine since the monthly plots look all normal, or so it seems of course.

sulfurdioxide_total_vertical_in_S5P_RegriddedData_201806
sulfurdioxide_total_vertical_in_S5P_RegriddedData_201806880×576 222 KB

Then I thought to merge the monthly files into seasonal files, simply using:

spawn, “harpmerge -ap ‘squash(time,(latitude,longitude,latitude_bounds,longitude_bounds))’ “+harpfiles+” “+harpdir+“S5P_RegriddedData_”+title+”.nc”

This doesn’t seem to work and the result is simply the first month I include in the variable “harpfiles” and not the average of all three.

sulfurdioxide_total_vertical_in_S5P_RegriddedData_2018_Summer
sulfurdioxide_total_vertical_in_S5P_RegriddedData_2018_Summer880×576 217 KB

This plot is exactly the same as the above, the monthly, believe me, I’ve created difference plots as well.

In case you ask, since for SO2 we do not have data for all days for each month and for each grid point it would not be statistically 100% accurate to create the seasonal from the daily since the months that are more complete would be more represented in the outcome hence skewing the seasonal towards one of the months.

Any ideas as to why the seasonal doesn’t seem to work?

Many thanks,
MariLiza

2

You need to use the bin() operation, otherwise you end up with a time series of grids.

1 Like
3

I did try bin() as well, apologies for not mentioning it, as per:

spawn, “harpmerge -ap ‘squash(time,(latitude,longitude,latitude_bounds,longitude_bounds));bin()’ “+harpfiles+” “+harpdir+“S5P_RegriddedData_”+title+”_bin.nc”

Got the same result, see the difference… :slight_smile:

Difference
Difference880×576 176 KB

Also, in my seasonal files there is only one array for SO2 and not three.

sulfurdioxide_total_vertical_column(time=1, latitude=52, longitude=48);

Thanks!
MariLiza

4

Sander, I found the issue, it lay in the way the input files where passed onto the command line, something which was not clear to me from the description in the text, http://stcorp.github.io/harp/doc/html/harpmerge.html

I was erroneously introducing an array of directories, not a single one with wildcards for e.g., expecting the command to read in simultaneously all the files found in all directories.

Thanks for your patience!
MariLiza