# Opening S-5p with harp.import\_product()

**URL:** https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115
**Category:** HARP
**Tags:** python
**Created:** [April 20, 2020, 1:48pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115 "2020-04-20T13:48:37Z")
**Posts on this page:** 18
**Page:** 1

<div class="post-metadata">

### Author: ![user231093](https://forum.atmospherictoolbox.org/letter_avatar_proxy/v4/letter/u/b4bc9f/32.png) [@user231093](https://forum.atmospherictoolbox.org/u/user231093)
#### Post date: [April 20, 2020, 1:48pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/1 "2020-04-20T13:48:37Z")

</div>

Hi,

I am trying to read a Sentinel-5p Level-2 NO2 product using the HARP python interface:

```
import harp

product = .../S5P_OFFL_L2 __NO2____ 20200401T110350_20200401T124521_12784_01_010302_20200403T040102
test_harp = harp.import_product(product)

```

I am getting an error from time to time claiming:

```
---------------------------------------------------------------------------
CLibraryError Traceback (most recent call last)
~/Training/Sentinel.py in <module>
----> 1 test_harp = harp.import_product(product)

~/anaconda3/envs/rus/lib/python3.7/site-packages/harp/_harppy.py in import_product(filename, operations, options, reduce_operations, post_operations)
   1185 if _lib.harp_import(_encode_path(filename), _encode_string(operations), _encode_string(options),
   1186 c_product_ptr) != 0:
-> 1187 raise CLibraryError()
   1188 
   1189 try:

CLibraryError: [HDF5] H5F_open(): file close degree doesn't match (major="File accessibilty", minor="Unable to initialize object") (H5Fint.c:1738)

```

When acessing the file with `xarray` - `xarray.import_dataset(product)` any issue in the process happens and I can process my data properly.

Any clue on this matter?

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [April 20, 2020, 2:04pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/2 "2020-04-20T14:04:42Z")

</div>

This might be because you already (or still) have the file open somewhere else (also using the HDF5 library underneath). Is this something you can confirm?

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [April 20, 2020, 2:09pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/3 "2020-04-20T14:09:06Z")

</div>

I think it may be linked to [this issue](https://github.com/h5py/h5py/issues/218).

---

<div class="post-metadata">

### Author: ![user231093](https://forum.atmospherictoolbox.org/letter_avatar_proxy/v4/letter/u/b4bc9f/32.png) [@user231093](https://forum.atmospherictoolbox.org/u/user231093)
#### Post date: [April 21, 2020, 9:50am UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/4 "2020-04-21T09:50:14Z")

</div>

That could explain the issue as prior to running `harp.import_product()` I am opening that file with `xarray`. However, there must be still something wrong since I use a `with` statmenent , which in theory closes the file automatically.

> with xr.open\_dataset(product, group=‘PRODUCT’) as file:

Anyways, If Ido not run `xarray` in advacned, `harp.import_product()` does not generate that error anymore. However, now I am facing a new situation (error):

In some files (I am processing 25 that cover my study area) when I use `harp.import_product()` the following is happening:

1- If I use `harp.import_product(product)`, it reads the file properly  
2- If I ise `harp.import_product(product, operations=convert_operations)`, then the following error takes place:

```
---------------------------------------------------------------------------
NoDataError Traceback (most recent call last)
~/Training/Sentinel.py in <module>
      8 
----> 9 test_harp = harp.import_product(product, operations=convert_operations)
     11 

~/anaconda3/envs/rus/lib/python3.7/site-packages/harp/_harppy.py in import_product(filename, operations, options, reduce_operations, post_operations)
   1190 # Raise an exception if the imported C product contains no variables, or variables without data.
   1191 if _lib.harp_product_is_empty(c_product_ptr[0]) == 1:
-> 1192 raise NoDataError()
   1193 
   1194 # Convert the C product into its Python representation.

NoDataError: product contains no variables, or variables without data

```

Where `convert_operations` is a list containing `derive`, `bin_spatial`, and `keep` operations

From previous posts in this forum, I have seen this may be caused by a corrupted file and a redownload should solved. However, since opening the product without any operatoin works fine, I assume the issue is related to my operations. However, why is it working for some files (out of the 25 I have) and not some one in particular?

Any previous experience on this?

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [April 21, 2020, 10:05am UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/5 "2020-04-21T10:05:15Z")

</div>

> [@user231093](#):
>
> However, since opening the product without any operatoin works fine, I assume the issue is related to my operations. However, why is it working for some files (out of the 25 I have) and not some one in particular?

This is probably because you are using a filter that for some products filters out all satellite ground pixels in the product. It just means that there a no good quality measurements in the product. This is something that can happen.  
You can explicitly catch the harp.NoDataError exception in your python code to handle this case.

---

<div class="post-metadata">

### Author: ![ryan](https://forum.atmospherictoolbox.org/letter_avatar_proxy/v4/letter/r/a8b319/32.png) [@ryan](https://forum.atmospherictoolbox.org/u/ryan)
#### Post date: [May 13, 2020, 9:11am UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/7 "2020-05-13T09:11:22Z")

</div>

I am also getting this error for about 20% of the files downloaded for January 2020:

CLibraryError: [HDF5] H5O\_\_prefix\_deserialize(): bad object header version number (major=“Object header”, minor=“Wrong version number”) (C:\ci\hdf5\_1545244154871\work\src\H5Ocache.c:1231)  
OR  
CLibraryError: [HDF5] H5B\_\_cache\_deserialize(): wrong B-tree signature (major=“B-Tree node”, minor=“Bad value”) (C:\ci\hdf5\_1545244154871\work\src\H5Bcache.c:181)

and sometimes other CLibraryErrors

I am not processing the files using xarray before but simply using “harp.import\_product”.

It is strange that most of the files are executed normally and only around 20% of the files produce such errors. Could be that the files are corrupted during download?

Thanks.

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [May 13, 2020, 9:28am UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/8 "2020-05-13T09:28:54Z")

</div>

It is very likely that the products got corrupted during download indeed.

Please try to redownload to see if that resolves it.

---

<div class="post-metadata">

### Author: ![pravashtiwari](https://forum.atmospherictoolbox.org/letter_avatar_proxy/v4/letter/p/f04885/32.png) [@pravashtiwari](https://forum.atmospherictoolbox.org/u/pravashtiwari)
#### Post date: [June 8, 2020, 10:25am UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/9 "2020-06-08T10:25:55Z")

</div>

Has this issue got resolved already,  
I encountered a similar error  
CLibraryError: [HDF5] H5F\_open(): file close degree doesn’t match (major=“File accessibilty”, minor=“Unable to initialize object”) (C:\ci\hdf5\_1545244154871\work\src\H5Fint.c:1738)

And my Level 3 files get partially made ???

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [June 8, 2020, 10:38am UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/10 "2020-06-08T10:38:16Z")

</div>

The resolution is mentioned right above. Just redownload the files as suggested.

---

<div class="post-metadata">

### Author: ![falahfakhri](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/falahfakhri/32/62_2.png) [@falahfakhri](https://forum.atmospherictoolbox.org/u/falahfakhri)
#### Post date: [June 18, 2020, 12:23pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/11 "2020-06-18T12:23:27Z")

</div>

Dear Sandar,

I have as test two s5p NO2 OFF\_L2, both has been opened and processed properly using netcdf4, but now with the following piece of script,

export\_path = “D:\TESTS\S-5P-SCRIPTS\DATA\process”

for i in input\_files\_OFFL:  
harp\_L2\_L3 = harp.import\_product(i, operations="   
tropospheric\_NO2\_column\_number\_density\_validity\>75;   
derive(tropospheric\_NO2\_column\_number\_density [Pmolec/m2]);   
derive(datetime\_stop {time});   
latitude \> 28. [degree\_north] ; latitude \< 31.6 [degree\_north]; longitude \> 50. [degree\_east]; longitude \< 56.1 [degree\_east];   
bin\_spatial(360, 28., 0.01, 610, 50., 0.01);   
derive(latitude {latitude}); derive(longitude {longitude});   
keep(NO2\_column\_number\_density, tropospheric\_NO2\_column\_number\_density,   
stratospheric\_NO2\_column\_number\_density, NO2\_slant\_column\_number\_density,   
tropopause\_pressure, absorbing\_aerosol\_index, cloud\_fraction, sensor\_altitude,   
sensor\_azimuth\_angle, sensor\_zenith\_angle, solar\_azimuth\_angle, solar\_zenith\_angle)")

```
export_folder="{export_path}/{name}".format(export_path=export_path, name=i.split('/')[-1].replace('L2', 'L3'))

harp.export_product(harp_L2_L3, export_folder, file_format='netcdf')

```

print(colored(‘All L2 products covnertod to L3’, ‘green’))

I got the following error,

 ![image](https://forum.atmospherictoolbox.org/uploads/default/original/1X/7fbe11be1adc9c7c9b9b7e426449fb9100a4ef57.png)

Would you please to help!

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [June 18, 2020, 12:40pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/12 "2020-06-18T12:40:47Z")

</div>

This issue has already been discussed [here](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115)

---

<div class="post-metadata">

### Author: ![falahfakhri](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/falahfakhri/32/62_2.png) [@falahfakhri](https://forum.atmospherictoolbox.org/u/falahfakhri)
#### Post date: [June 18, 2020, 12:46pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/13 "2020-06-18T12:46:22Z")

</div>

Thanks a lot, so, Is it not solved bug?,

Or is my filter is the problem, \>75

Because, I don’t think, it’s download issue, I tested each single product separately.

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [June 18, 2020, 12:58pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/14 "2020-06-18T12:58:27Z")

</div>

Please explain what else you are doing in your script. Are you also using other libraries to access the products besides harp? e.g. h5py? Because if so, then that is your problem.

This is not a bug in HARP.

---

<div class="post-metadata">

### Author: ![falahfakhri](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/falahfakhri/32/62_2.png) [@falahfakhri](https://forum.atmospherictoolbox.org/u/falahfakhri)
#### Post date: [June 18, 2020, 1:02pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/15 "2020-06-18T13:02:41Z")

</div>

Dear Sandar,

These are all the libraries I’m using to access the products, but not h5py,

```
from netCDF4 import Dataset
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm
from mpl_toolkits.basemap import Basemap
from glob import iglob
from os.path import join
from termcolor import colored
import sys
import os
import harp
```

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [June 18, 2020, 1:18pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/16 "2020-06-18T13:18:31Z")

</div>

Then it is your use of the netCDF4 library that is causing the problems. You cannot have the same file open with HDF5 (which both netCDF4 and HARP use underneath) at the same time when they use different fclose\_degrees.  
You have to make sure that your file is closed properly with netCDF4 before performing the harp import

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [October 23, 2020, 8:40pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/18 "2020-10-23T20:40:21Z")

</div>

The error is right there at the bottom. You are not providing correct arguments. Please review error messages first before posting on this forum

---

<div class="post-metadata">

### Author: ![sergeantpol](https://forum.atmospherictoolbox.org/letter_avatar_proxy/v4/letter/s/54ee81/32.png) [@sergeantpol](https://forum.atmospherictoolbox.org/u/sergeantpol)
#### Post date: [October 26, 2020, 5:50pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/19 "2020-10-26T17:50:24Z")

</div>

ok I think I solved it but now I have a new error.  
[HDF5] H5F\_\_super\_read(): truncated file: eof = 25682432, sblock-\>base\_addr = 0, stored\_eof = 458939235 (major=“File accessibilty”, minor=“File has been truncated”) (C:\ci\hdf5\_1545244154871\work\src\H5Fsuper.c:623)  
Do you know what should I do?

---

<div class="post-metadata">

### Author: ![sander.niemeijer](https://forum.atmospherictoolbox.org/user_avatar/forum.atmospherictoolbox.org/sander.niemeijer/32/5_2.png) [@sander.niemeijer](https://forum.atmospherictoolbox.org/u/sander.niemeijer)
#### Post date: [October 26, 2020, 6:58pm UTC](https://forum.atmospherictoolbox.org/t/opening-s-5p-with-harp-import-product/115/20 "2020-10-26T18:58:28Z")

</div>

This means that your file is corrupted. Please try to download it again.
