# File size problem after harpmerge
**URL:** https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239
**Category:** HARP
**Tags:** sentinel-5p, python
**Created:** [November 21, 2020, 8:17am UTC](https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239 "2020-11-21T08:17:28Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@hmadadi](https://forum.atmospherictoolbox.org/u/hmadadi)
#### Post date: [November 21, 2020, 8:17am UTC](https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239/1 "2020-11-21T08:17:28Z")
I am trying to merge two netcdf files with harpmerge in python in two method. Each netcdf file size is 127 MB. It is important for me to clip the netcdf dataset based on my study area.
# First method:
I merged these tow files with harp in python:
> myfiles = ‘…/\*.nc’
> i = 0
> file\_names =
> products = # array con los productos a juntar
>
> for filename in os.listdir(os.path.dirname(os.path.abspath(myfiles))):
> base\_file, ext = os.path.splitext(filename)
>
> ```
> if ext == ".nc" and base_file.split('_')[0] == 'S5P':
> product_name = base_file + "_" + str(i)
>
> try:
> product_name = harp.import_product( base_file + ext,
> operations="latitude > 24.8 [degree_north]; latitude < 39.9 [degree_north];\
> longitude > 43.8 [degree_east]; longitude < 63.4 [degree_east];\
> bin_spatial(1511,24.8,0.01,1961,43.8,0.01); derive(longitude {longitude});\
> derive(latitude {latitude})")
> print("Product " + base_file + ext + " imported")
>
> products.append(product_name)
> except:
> print ("Product not imported")
>
> i = i + 1
>
> ```
> product\_bin = harp.execute\_operations(products, post\_operations=“bin();squash(time, (latitude,longitude))”)
> harp.export\_product(product\_bin, “merged.nc”)
Merged netcdf file created has 296 MB size, It is very large because when I want to use this method for many files, it will need huge capacity of hdd.
# Second method:
In this method I tried to use xarray capability, for do that first I clip each file based on my study area:
> ds = xr.open\_dataset(file, group = ‘PRODUCT’)
> ds\_ir = ds.where((44 \< ds.longitude) & (ds.longitude \< 65)
> & (24 \< ds.latitude) & (ds.latitude \< 41), drop=True)
>
> src\_fname, ext = os.path.splitext(file) # split filename and extension
> save\_fname = os.path.join(outpath, os.path.basename(src\_fname)+‘.nc’)
> ds\_ir.to\_netcdf(save\_fname)
File size of each clipped netcdf file with xarray is 3.5 MB (it’s good size).
But when I want merge these two clipped netcdf files in first step of import product:
> product\_name = harp.import\_product(‘…/clipped\_nc\_in xarray.nc’)
It return:
> CLibraryError: /clipped\_nc\_in xarray.nc: unsupported product
What is my problem? Why netcdf file created with xarray does not supported with harp?
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [November 21, 2020, 12:19pm UTC](https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239/2 "2020-11-21T12:19:55Z")
Regarding the size difference, be aware that with xarray you are only reading the variables from the ‘PRODUCT’ netcdf group. HARP will read a lot of other variables as well. If you are interested in only a e.g. single variable (together with the time/lat/lon axis) then you should use the `keep()` operation as part of your HARP operations to limit the number of ingested variables. This will greatly reduce the size of the output product (and speed up the bin\_spatial() operation).
In addition, you are using a target grid resolution that is slightly higher than the native S5P resolution. This is not necessarily a bad thing, but you will then of course end up with a higher file size than if you would just filter the original satellite pixels using a lat/lon filter (as you are doing with xarray).
Regarding reading of data saved by xarray in HARP, note that the file that xarray writes is neither a valid S5P product, nor is it a valid HARP product. This means that HARP will not be able to import it.
---
### Author:  [@hmadadi](https://forum.atmospherictoolbox.org/u/hmadadi)
#### Post date: [November 21, 2020, 1:06pm UTC](https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239/3 "2020-11-21T13:06:20Z")
> [@svniemeijer](#):
>
> native S5P resolution
Hi @svniemeijer, thanks for your complete explanation, but what is native S5P resolution? I could not find it in decimal degree.
I clipped original (downloaded) S5P file in xarray and then mereged in harp. If I want to merge original (downloaded) S5P file (without clip) in harp, what is correct operations with native S5P resolution?
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [November 21, 2020, 5:02pm UTC](https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239/4 "2020-11-21T17:02:11Z")
The pixel resolution of S5P is about 3.5km x 5km. It is not defined in decimal degrees (since the length of a longitudinal degree depends on the latitude).
If you want to keep the data in the original satellite grid resolution, then don’t perform the `bin_spatial()` operation.
---
### Author:  [@hmadadi](https://forum.atmospherictoolbox.org/u/hmadadi)
#### Post date: [November 21, 2020, 6:15pm UTC](https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239/5 "2020-11-21T18:15:59Z")
Without `bin_spatial`, the `longitude` and `latitude` dimensions removed in output nc file.
with these cods:
`product_name = harp.import_product( base_file + ext, operations="latitude > 24.8 [degree_north]; latitude < 39.9 [degree_north]; longitude > 43.8 [degree_east]; longitude < 63.4 [degree_east]; derive(longitude {longitude}); derive(latitude {latitude})") `
OR
`product_name = harp.import_product( base_file + ext, operations="latitude > 24.8 [degree_north]; latitude < 39.9 [degree_north]; longitude > 43.8 [degree_east]; longitude < 63.4 [degree_east]") `
OR
`product_name = harp.import_product( base_file + ext) `
What is my problem?
---
### Author:  [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [November 21, 2020, 6:33pm UTC](https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239/6 "2020-11-21T18:33:47Z")
This is because the S5P data does not have a `latitude` and `longitude` dimension (it has `latitude` and `longitude` _variables_). The dimensions in the S5P product are `scanline` and `groundpixel`. But those are more virtual raster dimensions (since the S5P pixels actually have a small overlap, they don’t form an actual grid). So HARP flattens all pixels into a single `time` dimension when reading the data.
Your xarray data also does not have `latitude` and `longitude` _dimensions_.
---
### Author:  [@hmadadi](https://forum.atmospherictoolbox.org/u/hmadadi)
#### Post date: [November 21, 2020, 7:58pm UTC](https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239/7 "2020-11-21T19:58:19Z")
Thanks for your clear explanation. Based on your comment, with above code, have I `latitude` and `longitude` variables in output nc file? If not, how can I to have them?
When I open output nc file with xarray, I have only one number for `latitude` and `longitude` as follow:
---
### Author:  [@hmadadi](https://forum.atmospherictoolbox.org/u/hmadadi)
#### Post date: [November 22, 2020, 7:32am UTC](https://forum.atmospherictoolbox.org/t/file-size-problem-after-harpmerge/239/8 "2020-11-22T07:32:39Z")
Without `bin_spatial` operation, output netcdf file is 14.2 KB, and lost `longitude` and `latitude`!