Harp error handling in IDL

HARP
1

Hi Sander and colleagues,

Apologies in advance for a, I am sure, simpleton question!

I am ingesting S5P OFFL CO data using:

product = harp_import(file,‘CO_column_number_density_validity>50;derive(CO_column_volume_mixing_ratio {time} [ppbv]);area_inside_area((39.6335, 39.6335
, 41.6335, 41.6335),(21.9572,23.9572,23.9572,21.9572));keep(latitude, longitude, solar_zenith_angle, CO_column_number_density_validity, orbit_index, su
rface_pressure, CO_column_number_density, sensor_zenith_angle, CO_column_number_density_uncertainty, scan_subindex, H2O_column_number_density, CO_colum
n_volume_mixing_ratio,datetime_start, datetime_length, pressure_bounds)’)

One of orbits does not pass the criteria and IDL replies with

% HARP-IDL ERROR -300: “invalid polygon (line segments overlap)”

The structure now contains the error

IDL> help, product,/str
** Structure HARP_ERROR, 2 tags, length=24, data length=18:
ERRNO INT -300
MESSAGE STRING ‘invalid polygon (line segments overlap)’

Can someone help me with catching the error and jumping through the rest of my analysis so that the code simply moves to the next file?

Thanks!
MariLiza

2

Hi Mariliza,

Welcome to the forum!

The issue you raised is actually an interesting one. A similar problem was also found within the S5P-MPC as part of the automated validation (which is based on the HARP tools).

In that case the issue was found in S5P_RPRO_L2__O3_____20180704T133322_20180704T151649_03748_01_010105_20190212T074951.nc, but there are potentially other orbits that have a similar problem.

The problem is that for the S5P data at the end of the orbit there can be a bit of curl. You can see this if you plot the sensor zenith angle of the full product and zoom in on Alaska. You will then see this:

l1_zenith_angle
l1_zenith_angle1039×573 38.8 KB

This back curling in the southern corner causes at least one ground pixel to flip around and it ends up with a bow-tie shape (which is invalid in most GIS applications as it is in HARP).

The problem is actually not just in the L2, but also in L1B (S5P_OFFL_L1B_RA_BD1_20180704T133421_20180704T151550_03748_01_010000_20180704T170457.nc).
It is actually an instrument sensing artefact. As we got clarified by the instrument team:

Once a day the solar irradiance is measured over the North Pole.
To improve the accuracy of this measurement, it is executed at a reference azimuth angle to remove seasonal effects.
In order to achieve this reference azimuth angle, the spacecraft performs a yaw maneuver prior to the solar measurement.
When doing this, the instrument is still in radiance measurement mode.
The combination of the speed of the yaw angle change, together with the instrument wide swath, and the spacecraft along track speed causes the observed radiance scanlines to cross at certain times.

In addition, some measurements around nadir will not have rectangular ground pixel corner points, but rather the coverage resembles a bow-tie.
This may look strange, but is physically correct.

So there you have it.

Now, for your HARP operations. You are hitting this error because you are trying an area_inside_area matching filter. This filter will match all satellite pixel areas against your area and therefore also try this weird pixel in Alaska. The area_inside_area operation is quite inefficient because of all these area matching operations. You should at least start of with a plain latitude/longitude filter (which is very efficient). And then, if this mid point filtering is not enough for you purposes, you could fine tune this by still adding the area_inside_area filter afterwards. And since you are then only filtering on satellite pixels that match the lat/lon range, you will not include this weird pixel at the end of the orbit anymore and thus your error should disappear.

So, use:

latitude>=39.6335;latitude<=41.6335;longitude>=21.9572;longitude<=23.9572

Or:

latitude>=39.6335;latitude<=41.6335;longitude>=21.9572;longitude<=23.9572;area_inside_area((39.6335, 39.6335, 41.6335, 41.6335),(21.9572,23.9572,23.9572,21.9572))
3

Very interesting!

Dear Sander,

May I ask what is the difference between bin_spatial() and area_inside_area() to filter data. I know that bin_spatial() also dictates the grid pixel size but I was also using it on S5P data to consider only an area of interest.

Thanks,
Ryan

4

The L2 data are considered a flat time series of pixels by HARP (so all data is only dependent on the time dimension). With area_inside_area you can filter this list to just those pixels that fall into a specific area (considering the area of each satellite pixel v.s. the area you define as parameter to the operation). But the result is still a time series of satellite pixels.

With bin_spatial you modify the data by turning it into grid with latitude and longitude dimensions (it is not a filter!). This will result in each target grid cell to have the weighted average of all satellite pixels that have an overlap.

5

Dear Sander,
Thank you so much for getting back to me so promptly!
I would never have guessed, of course. Which brings to mind another question: what happens if the ingest command does not find any data that full fill the requirements? what does the command reply then? an empty product structure or another error? is there a way to catch that one?
In case you are interested, the orbit that caused the original error is this:
S5P_OFFL_L2__CO_____20190706T103007_20190706T121137_08953_01_010302_20190712T094815.nc
Code works fine so far, after adding the min/max requirements.
Many thanks,
MariLiza

6

Ah, while I was writing the reply, indeed I got an error:

CO/S5P_OFFL_L2__CO_____20190731T091909_20190731T110039_09307_01_010302_20190806T084611.nc
% HARP-IDL ERROR -900: “no data left after operation”

:slight_smile:

MariLiza

7

The easiest way that I know of to check for an harp error in IDL is to look at the type of the return value.
typename(product) will be ANONYMOUS when a HARP product is returned and will be HARP_ERROR if an error occurred.
So just compare the type name against HARP_ERROR and if this succeeds then you can perform your error handling.

1 Like
8

Thank you! Is there an HARP-related IDL “manual” somewhere perhaps and I couldn’t find it on GitHub? I recall from the era of BEAT a link with IDL examples, relevant commands, etc? or maybe I recall wrong. In any case, thank you again!
MariLiza

9

We have the IDL interface documentation, but that is indeed a bit minimal.

However, most users we know are using (or moving to) Python, so examples will mostly be provided in that langue.

Nevertheless, we now have a forum, which we didn’t have before. So if you need some custom examples, this forum will be the likely place for us to put it.

1 Like