# How is the average of a HARP "bin" calculated? Role of weight and count variables?
**URL:** https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112
**Category:** HARP
**Created:** [April 17, 2020, 12:23pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112 "2020-04-17T12:23:36Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@Steven\_Compernolle](https://forum.atmospherictoolbox.org/u/Steven_Compernolle)
#### Post date: [April 17, 2020, 12:23pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/1 "2020-04-17T12:23:36Z")
I understood that since HARP 1.9 or so the HARP bin operation is no longer an arithmetic average but a weighted average.
Suppose for example a “point\_distance(lat,lon,50 [km]);bin();” operations on satellite orbit files, where the satellite pixel dimensions are present (i.e., there is latitude\_bounds and longitude\_bounds)
The result is a weighted average. But how is it exactly calculated? By weighting according to the pixel area?
In the resulting file there are the new “weight” and “count” variables.
“weight” will be used in subsequent bin operations, correct?
Does “count” play a role in subsequent bin operations? Perhaps when the weight variable is excluded before?
For a limited number of variables, there is now a corresponding “\_weight” variable:
latitude\_weight, longitude\_weight, solar\_zenith\_angle\_weight, solar\_azimuth\_angle\_weight, sensor\_zenith\_angle\_weight, sensor\_azimuth\_angle\_weight
What is the role of these new \_weight variables? Why only for these variables?
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [April 17, 2020, 12:46pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/2 "2020-04-17T12:46:02Z")
The best documentation on this is, is in the documentation for the C library functions for [harp\_product\_bin\_spatial](http://stcorp.github.io/harp/doc/html/libharp_product.html#harp_product_bin_spatial__harp_productP.l.l.lP.l.doubleP.l.doubleP) and [harp\_product\_bin](http://stcorp.github.io/harp/doc/html/libharp_product.html#harp_product_bin__harp_productP.l.l.lP).
We know that this is something we need to [document better](https://github.com/stcorp/harp/issues/172).
Let me know if you have any additional questions after reading those links.
---
### Author:  [@Steven\_Compernolle](https://forum.atmospherictoolbox.org/u/Steven_Compernolle)
#### Post date: [April 17, 2020, 1:01pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/3 "2020-04-17T13:01:36Z")
Thanks. As I use the “bin” operation, I guess “harp\_product\_bin” (and not harp\_product\_bin\_spatial) is the relevant C function.
The explanation for harp\_product\_bin\_spatial seems a bit more complete than for harp\_product\_bin.
harp\_product\_bin\_spatial:
“In addition, a ‘weight’ variable will be added that will contain the sum of weights for the contribution to each cell. If a variable contained NaN values then a variable specific weight variable will be created with only the sum of weights for the non-NaN entries.”
-\> I guess the same holds for harp\_product\_bin, but replacing “to each cell” by “to each bin” ??
In harp\_product\_bin\_spatial, area binning means that "each sample will be allocated to each lat/lon grid cell based on the amount of overlap. "
-\> For harp\_product\_bin, there are no grid cells and so no overlap can be calculated. So I guess weighting uses surface area, not amount of overlap?
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [April 17, 2020, 1:29pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/4 "2020-04-17T13:29:39Z")
harp\_product\_bin\_spatial actually does _both_ temporal binning and spatial binning (although in practice you would mostly use it with a single temporal target bin).
harp\_product\_bin only does the temporal binning.
The documentation is pretty accurate. There is no ‘spatial overlap’ in the temporal binning case (harp\_product\_bin). Weight variables are only created for angular variables (to store the length of the combined unit vectors).
---
### Author:  [@Steven\_Compernolle](https://forum.atmospherictoolbox.org/u/Steven_Compernolle)
#### Post date: [April 17, 2020, 1:49pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/5 "2020-04-17T13:49:43Z")
Ok.
Does this mean that in the above example (bin() per orbit file), an arithmetic average of the individual pixels will be taken?
And would, in a subsequent bin() operation (combining several “binned-per-overpass” instances), the average be weighted using the “count” variable?
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [April 17, 2020, 2:02pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/6 "2020-04-17T14:02:27Z")
That is indeed how it will happen.
---
### Author:  [@jonasv](https://forum.atmospherictoolbox.org/u/jonasv)
#### Post date: [November 22, 2021, 8:16pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/7 "2021-11-22T20:16:24Z")
I don’t observe that _both_ temporal and spatial binning is done when the bin\_spatial operation is applied. The example I have is:
harpmerge -f hdf5 -a “bin\_spatial(360,-90,0.5,720,-180,0.5);derive(latitude{latitude});derive(longitude{longitude});exclude(_angle_)” -ap “squash(time, (latitude,longitude));” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T231652\_20211002T005821\_20566\_02\_020201\_20211003T144620.nc output.nc
this gives an output.nc file with two bins in the time dimension, to average the temporal binning, I need to do a bin() operation. So, I dont understand what you mean by " harp\_product\_bin\_spatial actually does _both_ temporal binning and spatial binning" ?
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [November 22, 2021, 8:46pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/8 "2021-11-22T20:46:16Z")
You have to dissect what is happening.
First of all, you are calling harpmerge. This merges the result of operations done on multiple products. In your case 2 products. _Per product_ you are performing a `bin_spatial` operation. So your `bin_spatial` does not work on the products combined. The final `bin()` that you want to do (but are not showing in your example) is because you want to flatten the concatenation that the merge is doing (i.e. you want to unbin, what got binned per product).
The `bin_spatial` that you perform on a _single_ product, is, in fact, binning your product in time: it is putting everything in a single bin that has start time / stop time equal to that of the product. In other words, your time resolution changes (the time resolution is no longer a scanline, but an orbit).
If you would use the C-library you can actually, when gridding a single product, provide a binning grid in the time dimension when performing a `harp_product_bin_spatial`. See the `num_time_bin` and `num_time_elements` arguments, which behave similar to that of `harp_product_bin`. You could actually provide the `validity` (quality flag) variable for this, and then you would get separate grids per quality flag value.
---
### Author:  [@jonasv](https://forum.atmospherictoolbox.org/u/jonasv)
#### Post date: [November 22, 2021, 9:24pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/9 "2021-11-22T21:24:29Z")
thanks a lot, I think it is clear now. So I can do the following:
harpmerge -f hdf5 -ap “bin\_spatial(360,-90,0.5,720,-180,0.5);” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T231652\_20211002T005821\_20566\_02\_020201\_20211003T144620.nc output.nc
harpmerge -f hdf5 -a “derive(latitude{latitude});derive(longitude{longitude});exclude(_angle_);squash(ti
me, (latitude,longitude));” output.nc output2.nc
from output.nc it is clear that a temporal average (over all scanlines of both S5P files) is done and the end results is a single bin in time dimension. The second command can then perform some additional operations on output.nc
How can I get separate grids per quality flag value without using the C interface (maybe this is not possible? ). I see I can do a temporal average per quality flag as follows:
harpmerge -f hdf5 -ap “bin(validity);” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T231652\_20211002T005821\_20566\_02\_020201\_20211003T144620.nc output.nc
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [November 22, 2021, 9:41pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/10 "2021-11-22T21:41:08Z")
Doing a `bin_spatial` as an `-ap` means that you first merge the datasets and _then_ perform the spatial binning. This may require too much memory to keep all the L2 data in memory.
The whole idea of doing `bin_spatial` as `-a` and then perform ‘additional operations’ on the merged data is what the `-ap` is actually for. There should be no need to split things in two commands.
Also, if you are not using multiple products as input, just use harpconvert.
Doing a spatial bin with binning on a temporal variable is indeed not possible as a harp operation. This would make the `bin_spatial` function too complex (it already has quite a few parameters).
---
### Author:  [@jonasv](https://forum.atmospherictoolbox.org/u/jonasv)
#### Post date: [November 23, 2021, 10:29am UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/11 "2021-11-23T10:29:22Z")
OK, thanks for the clear explanation. I still have some question on the -ar option. So when I do the following:
harpmerge -f hdf5 -a “bin()” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T231652\_20211002T005821\_20566\_02\_020201\_20211003T144620.nc output.nc
output.nc will contain two bins in time dimension. (since -a does the temporal binning opeation on each input S5P file)
harpmerge -f hdf5 -ap “bin()” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T231652\_20211002T005821\_20566\_02\_020201\_20211003T144620.nc output.nc
output.nc will contain one bin in time dimension. (since -ap does the temporal binning opeation on the merged input S5P files)
But when I try the -ar option to take a temporal average, I get the following error:
harpmerge -f hdf5 -ar “bin()” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T231652\_20211002T005821\_20566\_02\_020201\_20211003T144620.nc output.nc
ERROR: products don’t both have variable ‘latitude\_weight’
I would say output.nc should be the same as in the previous command (with -ap). But apparently I still need to do something here.
---
### Author:  [@jonasv](https://forum.atmospherictoolbox.org/u/jonasv)
#### Post date: [December 6, 2021, 12:56pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/12 "2021-12-06T12:56:03Z")
I found the answer of my own previous question. To apply the bin() operation with harpmerge in combination with the -ar option, the following can be done: (I only keep the HCHO\_slant\_column\_number\_density variable here)
harpmerge -f hdf5 -a “keep(HCHO\_slant\_column\_number\_density);bin();” -ar “bin();derive(HCHO\_slant\_column\_number\_density float {time})” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T231652\_20211002T005821\_20566\_02\_020201\_20211003T144620.nc output.nc
this is because the reduce operations (after -ar) acts on the merged output file that is created after each append. Therefore this -ar is memory efficient (as explained in [harpmerge — HARP 1.14 documentation](http://stcorp.github.io/harp/doc/html/harpmerge.html)) .
However I dont understand why I needed to do the change from double to float (derive(HCHO\_slant\_column\_number\_density float {time}) ). This causes the result in output.nc to be of another datatype than when one should have used the -ap option.
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [December 6, 2021, 1:48pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/13 "2021-12-06T13:48:33Z")
I don’t understand your example. If I execute your command I actually get a `ERROR: variables don't have the same datatype (HCHO_slant_column_number_density)`. So I have to _leave out_ the type conversion of the variable.
---
### Author:  [@jonasv](https://forum.atmospherictoolbox.org/u/jonasv)
#### Post date: [December 6, 2021, 3:15pm UTC](https://forum.atmospherictoolbox.org/t/how-is-the-average-of-a-harp-bin-calculated-role-of-weight-and-count-variables/112/14 "2021-12-06T15:15:19Z")
Indeed, you 're right, sorry, I ve made some mistake here. The correct command is indeed:
harpmerge -f hdf5 -a “keep(HCHO\_slant\_column\_number\_density);bin();” -ar “bin();” S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T011729\_20211001T025859\_20553\_02\_020201\_20211002T165738.nc S5P\_OFFL\_L2\_\_HCHO\_\_\_20211001T231652\_20211002T005821\_20566\_02\_020201\_20211003T144620.nc output.nc