How to handle and organize multi-site ensemble runs?

[dlebauer]

Overview

@infotroph and I were discussing how to put together multi-site ensemble forecast outputs. Current behavior is that

• PEcAn standard netcdf files are organized thus

|- out/
  - ENS-<ens number>-<site id>/
    - <YYYY>.nc

And PEcAn generates summary Rdata files including

- ensemble.<ensemble_id>.<varname>.<startYYYY-endYYYY>.Rdata
# note that ensemble_id != ens number

But for these Rdata files (and associated PDF summary plots), each site overwrites the previous site's output.

So we were looking for a way to organize these outputs by site.

CMS project approach

For the CMS project, outputs are organized thus¹:

• sda.outputN.Rdata: this series of files contains the general SDA outputs for the Nth time step. Within each file, the obs.mean contains the mean observations for each state variable of each site, and the obs.cov contains the observation covariance matrix for each state variable of each site. The forecast includes a matrix of forecast results with the dimensions of the ensemble number (rows) and the state variable number times site number (columns) (e.g., if we have 25 ensembles for each site, four state variables, and 6400 sites, then the matrix dimensions will be 25*(46400) = 2525600). Every four columns correspond to the ensemble forecasts of every state variable for each site. ... The [[6]] object contains the relative ensemble weights calculated by comparing analysis and forecast results. The param.list contains the parameters used for each ensemble member of each site.
• sda.all.forecast.analysis.Rdata: This file summarizes the forecast and analysis matrix across all runs.

sda.all.forecast.analysis.Rdata contains analysis.all and forecast.all, which are lists of matrices (data.frames, one per time step) that can be used for downscaling.

These data.frames look like:

> head(forecast.all[[1]][,1:12], n = 2)
  AbvGrndWood LAI SoilMoistFrac TotSoilCarb AbvGrndWood.1 LAI.1 SoilMoistFrac.1
1      5.0004   0          50.6    64.18524        1.6277     0            52.4
2      5.4908   0          48.4     0.98392        2.0679     0            52.4
  TotSoilCarb.1 AbvGrndWood.2 LAI.2 SoilMoistFrac.2 TotSoilCarb.2
1      30.81041        1.4220     0            52.0      11.70966
2      25.42046        1.7641     0            48.4      60.87190

Questions:

@dongchen Zhang

• What scripts creates these files?
• How do you align sites with the columns in the matrix?

@PecanProject (community):

• Generally, what would be a good way (or are there current implementations) to handle ensemble.*.Rdata and associated files from individual sites that get clobbered during multisite runs?
• Any suggested refinements to the approach above?

Footnotes

1. from geo.bu.edu:/projectnb/dietzelab/dongchen/anchorSites/NA_runs/SDA_25ens_2024_11_25/ReadMe.txt ↩

[mdietze]

• The SDA workflow generates the analysis and forecast files, they aren't generated by separate scripts that post-process the raw output
• For the alignment of sites with columns, that info is in a separate file. I've asked @DongchenZ many times to embed this info in the file itself to reduce errors/confusion. Also, that output would really be better stored as a 3-D array (site, variable, ensemble member) but SDA itself requires unwrapping a site x variable matrix into a vector, and no one ever took the extra step of reorganizing that confusing format into an array as part of the SDA output, which admittedly would take extra processing time that is arguably better spent during a postprocessing phase
• The ensemble.*.Rdata file format was never updated for multisite because no one ever needed it and no one was running the workflow step that generates it as part of their workflow. Before deciding HOW to modify that step, I think the top priority would be to discuss IF we even need that step in the first place since in the 9 years that MultiSettings has existed no one has ever needed this functionality.
• I think the original ensemble post processing functions never really served their intended purpose because they would only process a pre-specified variable as part of a monolithic workflow. I think functions that read PEcAn outputs on the fly, such as read.outputs, are much more useful.
• My strong prediction is that if you try to merge all the outputs into one file that file will be too big to work with effectively. You'll then shift to cloud-native file formats that tile the date, which is basically where we are already but with a few bells and whistles that help make the looping over files invisible. One alternative would be to tweak all of pecans outputs to be cloud native to begin with, which is a great/noble end goal but a lot of work in the short term. The other would be to have some helper functions that would allow you to more easily read ensemble outputs on the fly
• If you think it's worth pre-processsing ensemble outputs to a lower dimensional analysis-ready file (which is what Dongchen inadvertently ended up with on the SDA outputs) then (1) I'd recommend using the EFI standard and (2) you'd have to decide ahead of time what variables and what timestep you want in that summary file.