Cannot filter harp product data for qa_value > 0.75

Hello,

I have been trying to filter harp product NO2 data if the qa_value is > 0.75. Looking at multiple examples online this is what I passed as ‘operations’ to harp.import_product function:

operations = “bin_spatial(1975,38.025,0.01,2355,11.815,0.01);tropospheric_NO2_column_number_density_validity > 75;derive(longitude{longitude});derive(latitude{latitude})”

I am using python for development. When I run it, I keep getting this error: “CLibraryError: variable ‘tropospheric_NO2_column_number_density_validity’ does not exist”.

When I get rid of the tropospheric_NO2_column_number_density_validity from the operations an print the harp product to see the list of variables, this is what I see:

“double datetime_start {time=1} [seconds since 2010-01-01]
float datetime_length [s]
int orbit_index
double sensor_altitude {time=1, latitude=1973, longitude=2353} [m]
double solar_zenith_angle {time=1, latitude=1973, longitude=2353} [degree]
double solar_azimuth_angle {time=1, latitude=1973, longitude=2353} [degree]
double sensor_zenith_angle {time=1, latitude=1973, longitude=2353} [degree]
double sensor_azimuth_angle {time=1, latitude=1973, longitude=2353} [degree]
double pressure_bounds {time=1, latitude=1973, longitude=2353, vertical=34, 2} [Pa]
double tropospheric_NO2_column_number_density {time=1, latitude=1973, longitude=2353} [mol/m^2]
double tropospheric_NO2_column_number_density_amf {time=1, latitude=1973, longitude=2353} []
double NO2_column_number_density {time=1, latitude=1973, longitude=2353} [mol/m^2]
double NO2_column_number_density_amf {time=1, latitude=1973, longitude=2353} []
double stratospheric_NO2_column_number_density {time=1, latitude=1973, longitude=2353} [mol/m^2]
double stratospheric_NO2_column_number_density_amf {time=1, latitude=1973, longitude=2353} []
double NO2_slant_column_number_density {time=1, latitude=1973, longitude=2353} [mol/m^2]
double cloud_fraction {time=1, latitude=1973, longitude=2353} []
double absorbing_aerosol_index {time=1, latitude=1973, longitude=2353} []
double cloud_albedo {time=1, latitude=1973, longitude=2353} []
double cloud_pressure {time=1, latitude=1973, longitude=2353} [Pa]
double surface_albedo {time=1, latitude=1973, longitude=2353} []
double surface_altitude {time=1, latitude=1973, longitude=2353} [m]
double surface_pressure {time=1, latitude=1973, longitude=2353} [Pa]
double surface_meridional_wind_velocity {time=1, latitude=1973, longitude=2353} [m/s]
double surface_zonal_wind_velocity {time=1, latitude=1973, longitude=2353} [m/s]
double sea_ice_fraction {time=1, latitude=1973, longitude=2353} []
double tropopause_pressure {time=1, latitude=1973, longitude=2353} [Pa]
long count {time=1}
float weight {time=1, latitude=1973, longitude=2353}
float solar_zenith_angle_weight {time=1, latitude=1973, longitude=2353}
float solar_azimuth_angle_weight {time=1, latitude=1973, longitude=2353}
float sensor_zenith_angle_weight {time=1, latitude=1973, longitude=2353}
float sensor_azimuth_angle_weight {time=1, latitude=1973, longitude=2353}
float tropospheric_NO2_column_number_density_weight {time=1, latitude=1973, longitude=2353}
float tropospheric_NO2_column_number_density_amf_weight {time=1, latitude=1973, longitude=2353}
float NO2_column_number_density_weight {time=1, latitude=1973, longitude=2353}
float NO2_column_number_density_amf_weight {time=1, latitude=1973, longitude=2353}
float stratospheric_NO2_column_number_density_weight {time=1, latitude=1973, longitude=2353}
float stratospheric_NO2_column_number_density_amf_weight {time=1, latitude=1973, longitude=2353}
float NO2_slant_column_number_density_weight {time=1, latitude=1973, longitude=2353}
float cloud_fraction_weight {time=1, latitude=1973, longitude=2353}
float absorbing_aerosol_index_weight {time=1, latitude=1973, longitude=2353}
float cloud_albedo_weight {time=1, latitude=1973, longitude=2353}
float cloud_pressure_weight {time=1, latitude=1973, longitude=2353}
float surface_albedo_weight {time=1, latitude=1973, longitude=2353}
float tropopause_pressure_weight {time=1, latitude=1973, longitude=2353}
double latitude_bounds {latitude=1973, 2} [degree_north]
double longitude_bounds {longitude=2353, 2} [degree_east]
double longitude {longitude=2353} [degree_east]
double latitude {latitude=1973} [degree_north]”

‘tropospheric_NO2_column_number_density_validity’ is missing and I even tried editing the netCDF for convert all the NO2 values to 0 if the qa_value for that specific location is greater than 0.75. I created a visualisation of the netCDF file after this filtering and it looks like this:

I don’t understand why does the visualisation has boxes of NO2 pixel values. My questions regarding these issues are:

  1. Why is the harp product missing ‘tropospheric_NO2_column_number_density_validity’ variable?
  2. What is the workaround for this?
  3. If editing the netCDF file to filter qa_values before passing it into harp is a workaround, why does the visualisation look strange?

Can anyone please help me with this? I would really appreciate your response!

Thanks!

Be aware that the order of the operations in your operations argument is important. You will thus first have to perform the validity filter and then perform the spatial binning. Otherwise you are binning unfiltered data, which is not what you want.

1 Like

Thank you so much! That worked :slight_smile: I have a visualisation of the data now.

1 Like

The result is .nc extension.
I would be very happy if you give information about making this Geotif or the process that makes hdf

Hello
Congratulations.
Can you share your operation commands?
Thank you from now.