77.8 MB daily average file vs 51.9 MB daily average file (Tropomi SO2)

Hi Sander,
Thanks for your help earlier. When I went to map November’s SO2 monthly average initially everythign was fine. The daily average files mapped correctly but i did notice that around 20 of the 30 daily average files were 77.8 mb files while the remaining 10 were 51.9 mb files. Turns out when I went to average the daily averages to create a monthly average I wasn’t able to do so. I could do a 20 day average for all the 77.8 mb files or a 10 day average fore the 51.9 mb files. I thought this was odd so I analyzed a 51.9 mb file and 77.8 mb file in panoply. Turns out the 77.8 mb files had the column number density weight variable in addition to column number density. I don’t understand why this would occur given I run the same code to create the daily averages and the monthly average and I also exclude so2 column number density in the code for daily average:
SO2 Daily average CODE: .product = harp.import_product(r"/Users/jpoetzscher/Downloads/NOV 30 SO2/*.nc", operations='SO2_column_number_density_validity>50;keep(latitude_bounds,longitude_bounds,SO2_column_number_density);bin_spatial(1801,-90,0.1,3601,-180,0.1);derive(SO2_column_number_density [DU]);exclude(SO2_column_number_density_weight)', post_operations="bin();squash(time, (latitude_bounds,longitude_bounds));derive(latitude {latitude});derive(longitude {longitude});exclude(latitude_bounds,longitude_bounds,latitude_bounds_weight,longitude_bounds_weight,count,weight)")

Monthly average
average = harp.import_product("/Users/jpoetzscher/Downloads/November SO2/*.nc", post_operations=“bin();squash(time, (latitude,longitude))”)

Are you using HARP 1.9.2?
There were some known issues with the creation of the weight variables.

Note that the best way to exclude the weight variable is to exclude it after merging all orbits into the daily product:

product = harp.import_product(r"/Users/jpoetzscher/Downloads/NOV 30 SO2/*.nc", operations='SO2_column_number_density_validity>50;keep(latitude_bounds,longitude_bounds,SO2_column_number_density);bin_spatial(1801,-90,0.1,3601,-180,0.1);derive(SO2_column_number_density [DU])', post_operations="bin();squash(time, (latitude_bounds,longitude_bounds));derive(latitude {latitude});derive(longitude {longitude});exclude(latitude_bounds,longitude_bounds,latitude_bounds_weight,longitude_bounds_weight,count,weight,SO2_column_number_density_weight)")

Yes I am using harp 1.9.2. Ok so you would suggest for the SO2 daily average files with the weight files I redo the merging with this new code, excluding the weight variable after merging.

That would indeed be a valid approach which would keep your daily files small and mergeable on a monthly basis.