# Processing S-5p using Python (only)
**URL:** https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101
**Category:** Uncategorized
**Tags:** sentinel-5p, python
**Created:** [March 30, 2020, 8:34am UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101 "2020-03-30T08:34:39Z")
**Posts on this page:** 12
**Page:** 2
### Author:  [@ucesmcl](https://forum.atmospherictoolbox.org/u/ucesmcl)
#### Post date: [August 30, 2020, 2:16pm UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/23 "2020-08-30T14:16:25Z")
Hello. I am doing similar data processing as people above, however I want to also keep the time variable (in my case, this is just the date of the scan). I have added it in with the keep operations, however it says the time variable does not exist, is there a way I can get round this.
I know the scan date is also in the name of the file but I need it as a variable.
Will appreciate any help!
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [August 31, 2020, 7:31am UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/24 "2020-08-31T07:31:14Z")
Note that HARP does a format conversion when it reads S5P products. Several variables will end up with a different name. Have a look at the [HARP documentation for the S5P NO2 product](http://stcorp.github.io/harp/doc/html/ingestions/S5P_L2_NO2.html) to see which variable names to use. In your case, you should at least include `datetime_start` and probably also `datetime_length`.
---
### Author:  [@ucesmcl](https://forum.atmospherictoolbox.org/u/ucesmcl)
#### Post date: [August 31, 2020, 8:34am UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/25 "2020-08-31T08:34:22Z")
Oh I see! That makes alot of sense, thank you !
---
### Author:  [@OpioCornelius](https://forum.atmospherictoolbox.org/u/OpioCornelius)
#### Post date: [October 23, 2021, 11:53pm UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/26 "2021-10-23T23:53:28Z")
Hi Sander.
I am interested in applying the averaging kernel from TROPOMI onto WRF-chem data and in this post you mention that HARP can do it.
Kindly share with me a sample script on how HARP can do this.
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [October 25, 2021, 7:17am UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/27 "2021-10-25T07:17:30Z")
Unfortunately, I don’t have some example script at hand. It would take some effort to create an example for this as well. Not to mention that HARP currently does not support WRF-chem data natively, so the conversion of the model data to HARP would then also first have to be performed.
Roughly speaking, how it goes is that you interpolate the model to the satellite grid. Then you collocate the model data and satellite data by assigning a `collocation_index` variable (something that normally is done via `harpcollocate`, but you can shortcut it in this case). Then you apply the averaging kernel using either the `smooth()` HARP operation when you want profiles or `derive_smoothed_column()` when you want columns. And finally you can regrid both datasets back to a lat/lon grid for easier visualisation.
---
### Author:  [@jonasv](https://forum.atmospherictoolbox.org/u/jonasv)
#### Post date: [November 10, 2021, 6:24pm UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/28 "2021-11-10T18:24:42Z")
related to the derive(latitude {latitude}) operation I have the following question:
Why is “squash(time,(latitude,longitude));” needed as a post processing in the following commmand to produce lat/lon center
harpmerge -f hdf5 -a “bin\_spatial(360,-90,0.5,720,-180,0.5);derive(latitude{latitude});
derive(longitude{longitude});” -ap “squash(time,(latitude,longitude));” x.nc y.nc
(with x.nc , for example, a tropomi L2 file)
Whereas without the squash operation in the previous command, y.nc does not contain the lat/lon centers.
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [November 10, 2021, 10:30pm UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/29 "2021-11-10T22:30:49Z")
If you don’t perform the squash then you will still have latitude/longitude variables containing the center positions (a simple ncdump will show you this). _However_, these variables will have dimensions `time,latitude` and `time,longitude`, which most applications don’t accept as axis variables. The squash operation removes the time axis (which is allowed if your grid is the same for each time step). And the latitude/longitude variables will then be ‘proper’ axis variables to those applications.
---
### Author:  [@jonasv](https://forum.atmospherictoolbox.org/u/jonasv)
#### Post date: [November 15, 2021, 2:58pm UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/30 "2021-11-15T14:58:57Z")
I doubt that (but maybe I am doing something else wrong). So what I do (with harp version 1.14) is the following:
harpmerge -f hdf5 -a “bin\_spatial(360,-90,0.5,720,-180,0.5);derive(latitude{latitude});d
erive(longitude{longitude});” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc y.nc
then the output file y.nc does not contain lat or lon centers (it contains all 0.0 for lat/lon)
When I do
harpmerge -f hdf5 -a “bin\_spatial(360,-90,0.5,720,-180,0.5);derive(latitude{latitude});derive(longitude{longitude});” -ap “squash(time,(latitude,longitude));” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc y.nc
y.nc then contains the lat lon centers, apparently the squash operation causes the lat/lon centers to be printed out, whereas without the squash operation the lat/lon centers are not printed, but are all 0.0.
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [November 15, 2021, 4:15pm UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/31 "2021-11-15T16:15:34Z")
There is indeed something strange going on. This looks like a bug in harp. I will investigate.
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [November 16, 2021, 8:46am UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/32 "2021-11-16T08:46:35Z")
I looked into it. Things are actually as they are intended.
The point is that you are using hdf5(/netcdf4). Everything is actually explained already in [our documentation](http://stcorp.github.io/harp/doc/html/conventions/hdf5.html).
The actual latitude/longitude values are stored in variables called `_nc4_non_coord_latitude` and `_nc4_non_coord_longitude` if they are not one-dimensional.
When you read the data with either the netcdf4 library or harp, you will not see those 0 values for lat/lon. As they are picked up from these `_nc4_non_coord_` variables.
---
### Author:  [@eveliseb](https://forum.atmospherictoolbox.org/u/eveliseb)
#### Post date: [November 26, 2021, 12:43pm UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/33 "2021-11-26T12:43:07Z")
Hi Sorry to pop in that discussion. I have a related question and maybe some of you have the answer. About the averaging kernel. I’m trying to use the averaging kernel to compare modelled mixing ratio with tropomi retrievals. From a netcdf file, I am able to get:
“DETAILED\_RESULTS/column\_averaging\_kernel”
“INPUT\_DATA/altitude\_levels”
but the averaging kernel has 12 levels (array 12xlonxlat) while the altitude has 13 levels (array 13xlonxlat).
I have been browsing online document for days to find out how to match altitude with averaging kernel values. I have not find if (1) the altitude in the netcdf is the one to use and (2) how to reconciliate the dimension (is it intervals of altitudes?).
Thanks!
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [December 5, 2021, 7:27pm UTC](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101/34 "2021-12-05T19:27:37Z")
@eveliseb, first, please make sure to mention what product you are trying to read. I assume you are trying to use a CH4 product?
If you read the data with HARP, then all this complication gets resolved for you. You will then get lower/upper bounds for each layer in an `altitude_bounds` variable. And you can get the layer mid points if you provide a `derive(altitude {time,vertical} [km])` step in your HARP operations.
You can find all the details on how HARP maps the S5P data in the [documentation for the CH4 product](http://stcorp.github.io/harp/doc/html/ingestions/S5P_L2_CH4.html). The mapping table at the bottom describes where HARP gets all the data from. This table is also a good reference for how to interpret the data if you don’t want to use HARP.
[Previous page](https://forum.atmospherictoolbox.org/t/processing-s-5p-using-python-only/101.md?page=1)