Hi,
I am trying to pre-process S-5P files to clip the files to ROI so they can be merged. The code I have is not causing an error but it is not processing any files either. After running it just shows the statement: Pre-processing time: 0.0 seconds. I have 89 S-5p files to pre-process. Any Idea what I am missing or doing wrong? Could it be a problem with my harp package or environment?
This is my code:
input_path_aer=‘/S5_Data/Sentinel_FireEmissions_CA2020/Original/SSP_AER_AI/’
export_path_aer=‘/S5_Data/Sentinel_FireEmissions_CA2020/Processing/SSP_AER_AI/’
list_files = sorted(os.listdir(input_path_aer))
files_input = sorted(glob.glob(join(input_path_aer, ‘SSP_OFFL_".nc’)))
for file in list_files:
if file.startswith(“SSP_OFFL_”)==False:
list_files.remove(file)
t0 = time.time()
for i in range(len(files_input)):
Converted_AER=harp.import_product(files_input[i],
operations= “latitude <= 40[degree_north]; latitude >= 35[degree_north];
longitude <= -124 [degree_east]; longitude >= -120 [degree_east];
keep(latitude,latitude_bounds,longitude,longitude_bounds,
absorbing_aerosol_index,absorbing_aerosol_index_validity,datetime_start)”)
harp.export_product(Converted_AER, join(export_path_aer+'pre_processing_files', list_files[i]),file_format="netCDF")
print("product", os.path.basename(files_input[i]), "pre-processed")
t1 = time.time()
pre_processing_time = t1-t0
print(“Pre-processing time: {} seconds”.format(pre_processing_time))
I think this is because you should be using longitude >= -124 [degree_east]; longitude <= -120 [degree_east]
Hi Sander,
Thank you for your quick response. I made those adjustments, and it still didn’t process files.
These are the packages I have installed using conda install -c conda-forge just to rule a package problem out.
_r-mutex 1.0.1 anacondar_1 conda-forge
aiohttp 3.8.3 py38h91455d4_1 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
anyio 3.5.0 py38haa95532_0
aom 3.5.0 h63175ca_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 21.3.0 pyhd3eb1b0_0
argon2-cffi-bindings 21.2.0 py38h2bbff1b_0
asttokens 2.0.5 pyhd3eb1b0_0
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
attrs 22.1.0 py38haa95532_0
backcall 0.2.0 pyhd3eb1b0_0
beautifulsoup4 4.11.1 py38haa95532_0
bleach 4.1.0 pyhd3eb1b0_0
blosc 1.21.3 hdccc3a2_0 conda-forge
boost-cpp 1.78.0 h9f4b32c_1 conda-forge
branca 0.6.0 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 h8ffe710_7 conda-forge
brotli-bin 1.0.9 h8ffe710_7 conda-forge
brotlipy 0.7.0 py38h91455d4_1005 conda-forge
bzip2 1.0.8 h8ffe710_4 conda-forge
ca-certificates 2022.12.7 h5b45459_0 conda-forge
cairo 1.16.0 hd694305_1014 conda-forge
cartopy 0.21.1 py38h84fee79_0 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py38h2bbff1b_3
cfitsio 4.2.0 h9ebe7e4_0 conda-forge
cftime 1.6.2 py38hbaf524b_1 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 win_pyhd8ed1ab_2 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cmcrameri 1.4 pyhd8ed1ab_0 conda-forge
coda 2.24.1 py38h2706d8f_0 conda-forge
colorama 0.4.6 py38haa95532_0
comm 0.1.2 py38haa95532_0
console_shortcut 0.1.1 4
contourpy 1.0.5 py38h59b6b97_0
cryptography 39.0.0 py38h95f5157_0 conda-forge
curl 7.87.0 h68f0423_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
debugpy 1.5.1 py38hd77b12b_0
decorator 5.1.1 pyhd3eb1b0_0
defusedxml 0.7.1 pyhd3eb1b0_0
double-conversion 3.2.0 h63175ca_1 conda-forge
eigen 3.4.0 h2d74725_0 conda-forge
entrypoints 0.4 py38haa95532_0
executing 0.8.3 pyhd3eb1b0_0
expat 2.5.0 h1537add_0 conda-forge
ffmpeg 4.4.2 gpl_h02edd2a_109 conda-forge
fiona 1.8.22 py38h6f5d430_2 conda-forge
flit-core 3.6.0 pyhd3eb1b0_0
folium 0.14.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.1 hbde0cde_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.12.1 h546665d_1 conda-forge
freexl 1.0.6 h67ca5e6_1 conda-forge
frozenlist 1.3.3 py38h91455d4_0 conda-forge
gdal 3.5.3 py38h5a6f081_10 conda-forge
geojson 2.5.0 py_0 conda-forge
geomet 1.0.0 pyhd8ed1ab_0 conda-forge
geopandas 0.12.2 pyhd8ed1ab_0 conda-forge
geopandas-base 0.12.2 pyha770c72_0 conda-forge
geos 3.11.1 h1537add_0 conda-forge
geotiff 1.7.1 h4ffd875_4 conda-forge
gettext 0.21.1 h5728263_0 conda-forge
gl2ps 1.4.2 h0597ee9_0 conda-forge
glew 2.1.0 h39d44d4_2 conda-forge
glib 2.74.1 h12be248_1 conda-forge
glib-tools 2.74.1 h12be248_1 conda-forge
gst-plugins-base 1.21.3 h001b923_1 conda-forge
gstreamer 1.21.3 h6b5321d_1 conda-forge
harp 1.17 py38r41h2706d8f_0 conda-forge
hdf4 4.2.15 h1b1b6ef_5 conda-forge
hdf5 1.12.2 nompi_h57737ce_101 conda-forge
html2text 2020.1.16 py_0 conda-forge
icu 70.1 h0e60522_0 conda-forge
idna 3.4 py38haa95532_0
imageio 2.24.0 pyh24c5eb1_0 conda-forge
importlib_resources 5.2.0 pyhd3eb1b0_1
intel-openmp 2023.0.0 h57928b3_25922 conda-forge
ipykernel 6.19.2 py38hd4e2768_0
ipython 8.7.0 py38haa95532_0
ipython_genutils 0.2.0 pyhd3eb1b0_1
jedi 0.18.1 py38haa95532_1
jinja2 3.1.2 py38haa95532_0
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h8ffe710_2 conda-forge
jsoncpp 1.9.5 h2d74725_1 conda-forge
jsonschema 4.16.0 py38haa95532_0
jupyter_client 7.4.8 py38haa95532_0
jupyter_core 5.1.1 py38haa95532_0
jupyter_server 1.23.4 py38haa95532_0
jupyterlab_pygments 0.1.2 py_0
kealib 1.5.0 h61be68b_0 conda-forge
kiwisolver 1.4.4 py38hbd9d945_0 conda-forge
krb5 1.20.1 heb0366b_0 conda-forge
lcms2 2.14 h90d422f_0 conda-forge
lerc 4.0.0 h63175ca_0 conda-forge
libaec 1.0.6 h39d44d4_0 conda-forge
libblas 3.9.0 8_mkl conda-forge
libbrotlicommon 1.0.9 h8ffe710_7 conda-forge
libbrotlidec 1.0.9 h8ffe710_7 conda-forge
libbrotlienc 1.0.9 h8ffe710_7 conda-forge
libcblas 3.9.0 8_mkl conda-forge
libclang 15.0.7 default_h77d9078_0 conda-forge
libclang13 15.0.7 default_h77d9078_0 conda-forge
libcurl 7.87.0 h68f0423_0 conda-forge
libdeflate 1.14 hcfcfb64_0 conda-forge
libffi 3.4.2 hd77b12b_6
libgdal 3.5.3 hefe12b1_10 conda-forge
libglib 2.74.1 he8f3873_1 conda-forge
libhwloc 2.8.0 h039e092_1 conda-forge
libiconv 1.17 h8ffe710_0 conda-forge
libkml 1.3.0 hf2ab4e4_1015 conda-forge
liblapack 3.9.0 8_mkl conda-forge
libnetcdf 4.8.1 nompi_h85765be_104 conda-forge
libogg 1.3.4 h8ffe710_1 conda-forge
libpng 1.6.39 h19919ed_0 conda-forge
libpq 15.1 ha9684e8_3 conda-forge
librttopo 1.1.0 he22b5cd_12 conda-forge
libsodium 1.0.18 h62dcd97_0
libspatialindex 1.9.3 h39d44d4_4 conda-forge
libspatialite 5.0.1 h07bf483_22 conda-forge
libsqlite 3.40.0 hcfcfb64_0 conda-forge
libssh2 1.10.0 h9a1e1f7_3 conda-forge
libtheora 1.1.1 h8d14728_1005 conda-forge
libtiff 4.4.0 hc4f729c_5 conda-forge
libvorbis 1.3.7 h0e60522_0 conda-forge
libwebp 1.2.4 h8ffe710_0 conda-forge
libwebp-base 1.2.4 h8ffe710_0 conda-forge
libxml2 2.10.3 hc3477c8_0 conda-forge
libxslt 1.1.37 h0192164_0 conda-forge
libzip 1.9.2 h519de47_1 conda-forge
libzlib 1.2.13 hcfcfb64_4 conda-forge
loguru 0.6.0 py38haa244fe_2 conda-forge
lxml 4.9.2 py38h8e95c58_0 conda-forge
lz4-c 1.9.3 h8ffe710_1 conda-forge
m2w64-bwidget 1.9.10 2 conda-forge
m2w64-bzip2 1.0.6 6 conda-forge
m2w64-expat 2.1.1 2 conda-forge
m2w64-fftw 3.3.4 6 conda-forge
m2w64-flac 1.3.1 3 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gettext 0.19.7 2 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-gsl 2.1 2 conda-forge
m2w64-libiconv 1.14 6 conda-forge
m2w64-libjpeg-turbo 1.4.2 3 conda-forge
m2w64-libogg 1.3.2 3 conda-forge
m2w64-libpng 1.6.21 2 conda-forge
m2w64-libsndfile 1.0.26 2 conda-forge
m2w64-libtiff 4.0.6 2 conda-forge
m2w64-libvorbis 1.3.5 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
m2w64-libxml2 2.9.3 3 conda-forge
m2w64-mpfr 3.1.4 4 conda-forge
m2w64-pcre2 10.34 0 conda-forge
m2w64-speex 1.2rc2 3 conda-forge
m2w64-speexdsp 1.2rc3 3 conda-forge
m2w64-tcl 8.6.5 3 conda-forge
m2w64-tk 8.6.5 3 conda-forge
m2w64-tktable 2.10 5 conda-forge
m2w64-wineditline 2.101 5 conda-forge
m2w64-xz 5.2.2 2 conda-forge
m2w64-zlib 1.2.8 10 conda-forge
mapclassify 2.5.0 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.1 py38h2bbff1b_0
matplotlib 3.6.2 py38haa244fe_0 conda-forge
matplotlib-base 3.6.2 py38h1094b8e_0
matplotlib-inline 0.1.6 py38haa95532_0
mistune 0.8.4 py38he774522_1000
mkl 2020.4 hb70f87d_311 conda-forge
msys2-conda-epoch 20160418 1 conda-forge
multidict 6.0.4 py38h91455d4_0 conda-forge
munch 2.5.0 py_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
nbclassic 0.4.8 py38haa95532_0
nbclient 0.5.13 py38haa95532_0
nbconvert 6.5.4 py38haa95532_0
nbformat 5.7.0 py38haa95532_0
nest-asyncio 1.5.6 py38haa95532_0
netcdf4 1.6.2 nompi_py38h78680c8_100 conda-forge
networkx 3.0 pyhd8ed1ab_0 conda-forge
nlohmann_json 3.11.2 h39d44d4_0 conda-forge
notebook 6.5.2 py38haa95532_0
notebook-shim 0.2.2 py38haa95532_0
numpy 1.22.3 py38h1d2777f_2 conda-forge
openh264 2.3.1 h63175ca_1 conda-forge
openjpeg 2.5.0 hc9384bd_1 conda-forge
openssl 3.0.7 hcfcfb64_1 conda-forge
packaging 22.0 py38haa95532_0
pandas 1.5.2 py38h5846ac1_2 conda-forge
pandocfilters 1.5.0 pyhd3eb1b0_0
parso 0.8.3 pyhd3eb1b0_0
pathlib2 2.3.7.post1 py38haa244fe_2 conda-forge
pcre2 10.40 h17e33f8_0 conda-forge
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 9.3.0 py38hdc2b20a_1
pip 22.3.1 py38haa95532_0
pixman 0.40.0 h8ffe710_0 conda-forge
pkgutil-resolve-name 1.3.10 py38haa95532_0
platformdirs 2.5.2 py38haa95532_0
ply 3.11 py_1 conda-forge
pooch 1.6.0 pyhd8ed1ab_0 conda-forge
poppler 22.12.0 ha6c1112_0 conda-forge
poppler-data 0.4.11 hd8ed1ab_0 conda-forge
postgresql 15.1 hd87cd2b_3 conda-forge
proj 9.1.0 h3863b3b_0 conda-forge
prometheus_client 0.14.1 py38haa95532_0
prompt-toolkit 3.0.36 py38haa95532_0
psutil 5.9.0 py38h2bbff1b_0
pthread-stubs 0.4 hcd874cb_1001 conda-forge
pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge
pugixml 1.11.4 h0e60522_0 conda-forge
pure_eval 0.2.2 pyhd3eb1b0_0
pycparser 2.21 pyhd3eb1b0_0
pygments 2.11.2 pyhd3eb1b0_0
pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyproj 3.4.1 py38hac5b721_0 conda-forge
pypubsub 4.0.3 py_0 conda-forge
pyqt 5.15.7 py38hd6c051e_2 conda-forge
pyqt5-sip 12.11.0 py38hd3f51b4_2 conda-forge
pyrsistent 0.18.0 py38h196d8e1_0
pyshp 2.3.1 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyh0701188_6 conda-forge
python 3.8.15 h4de0772_0_cpython conda-forge
python-dateutil 2.8.2 pyhd3eb1b0_0
python-fastjsonschema 2.16.2 py38haa95532_0
python_abi 3.8 2_cp38 conda-forge
pytz 2022.7 pyhd8ed1ab_0 conda-forge
pywin32 305 py38h2bbff1b_0
pywinpty 2.0.2 py38h5da7b33_0
pyzmq 23.2.0 py38hd77b12b_0
qt 5.15.6 h91493d7_0 conda-forge
qt-main 5.15.6 h9580fe5_5 conda-forge
qt-webengine 5.15.4 h0bdee70_3 conda-forge
r-base 4.1.3 hdca333a_5 conda-forge
requests 2.28.1 pyhd8ed1ab_1 conda-forge
rtree 1.0.1 py38h8b54edf_1 conda-forge
scikit-learn 1.2.0 py38h69724d7_0 conda-forge
scipy 1.10.0 py38h0f6ee2a_0 conda-forge
send2trash 1.8.0 pyhd3eb1b0_1
sentinelsat 1.1.1 pyhd8ed1ab_0 conda-forge
setuptools 65.6.3 py38haa95532_0
shapely 2.0.0 py38h9c0aba1_0 conda-forge
sip 6.7.5 py38hd3f51b4_0 conda-forge
six 1.16.0 pyhd3eb1b0_1
snappy 1.1.9 hfb803bf_2 conda-forge
sniffio 1.2.0 py38haa95532_1
soupsieve 2.3.2.post1 py38haa95532_0
sqlite 3.40.1 h2bbff1b_0
stack_data 0.2.0 pyhd3eb1b0_0
svt-av1 1.3.0 h63175ca_0 conda-forge
tbb 2021.7.0 h91493d7_1 conda-forge
tbb-devel 2021.7.0 h91493d7_1 conda-forge
terminado 0.17.1 py38haa95532_0
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tiledb 2.13.2 h3132609_0 conda-forge
tinycss2 1.2.1 py38haa95532_0
tk 8.6.12 h8ffe710_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tornado 6.2 py38h2bbff1b_0
tqdm 4.64.1 pyhd8ed1ab_0 conda-forge
traitlets 5.7.1 py38haa95532_0
typing-extensions 4.4.0 py38haa95532_0
typing_extensions 4.4.0 py38haa95532_0
ucrt 10.0.22621.0 h57928b3_0 conda-forge
urllib3 1.26.14 pyhd8ed1ab_0 conda-forge
utfcpp 3.2.3 h57928b3_0 conda-forge
vc 14.2 h21ff451_1
visan 4.3.1 py38hd3f51b4_3 conda-forge
vs2015_runtime 14.34.31931 h4c5c07a_10 conda-forge
vtk 9.2.2 qt_py38hc1024be_201 conda-forge
wcwidth 0.2.5 pyhd3eb1b0_0
webencodings 0.5.1 py38_1
websocket-client 0.58.0 py38haa95532_4
wheel 0.37.1 pyhd3eb1b0_0
win32_setctime 1.1.0 pyhd8ed1ab_0 conda-forge
win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge
wincertstore 0.2 py38haa95532_2
winpty 0.4.3 4
wslink 1.10.0 pyhd8ed1ab_0 conda-forge
wxpython 4.2.0 py38h5e21985_3 conda-forge
x264 1!164.3095 h8ffe710_2 conda-forge
x265 3.5 h2d74725_3 conda-forge
xarray 2022.12.0 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.4 h63175ca_1 conda-forge
xyzservices 2022.9.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h8d14728_0 conda-forge
yarl 1.8.2 py38h91455d4_0 conda-forge
zeromq 4.3.4 hd77b12b_0
zipp 3.11.0 py38haa95532_0
zlib 1.2.13 hcfcfb64_4 conda-forge
zstd 1.5.2 h7755175_4 conda-forge
I tried it and with those operations (with proper longitude filter) I am getting results. Of course, you should be using L2 products of orbits that actually have an overlap with your specified region.
Are you sure your 89 input products match your ROI?
P.S. The package listing was unnecessary. It should have no impact on the results.
I wonder if it’s due to a firewall issue. It looks like Jupyter notebook is not importing my files to process for some reason.
I also wonder if my working directory has something to do with it. There are times when I get a FileNotFoundError: [Errno 2] No such file or directory: This happens randomly and my code hasn’t been changed.
I figured out my problem. I had " instead of * before .nc
(input_path_aer, ‘S5P_OFFL_*.nc’)))
My files processed!! 
I unfortunately have another error.
Converted_AER = harp.import_product(input_files,
operations= “latitude <= 41[degree_north]; latitude >= 29[degree_north];
longitude >= -124 [degree_east]; longitude <= -117 [degree_east];
asorbing_aerosol_index_validity>50;
bin_spatial(550, 29, 0.02, 800, -117, 0.02);
derive(latitude {latitude}); derive(longitude {longitude});
keep(latitude,longitude,absorbing_aerosol_index,weight)”,
post_operations=“bin(); squash(time, (latitude,longitude,absorbing_aerosol_index))”
CLibraryError: variable ‘asorbing_aerosol_index_validity’ does not exist
Any idea what I am missing?
1 Like
Please read the variable name again. You just made a typo.
You are right.
Thank you Sander
Hi Sander,
I have been able to get my S-5P files to process correctly and all of my aerosol png files are completed. How do you get a time series visual gif file to play in Jupyter notebook?
Thank you in advance.
Sorry I forgot to add this.
This is my code
with open(figure_path_aer+‘animated_aerosol.gif’,‘rb’) as f:
display(Image(data=f.read(), format=‘png’))
It is showing a picture of a broken file below the code