How to handle no rain in pysteps?

[ladc]

Dear developers,

As mentioned in #282 and #295, the STEPS nowcasting component of pysteps (with/without blending with NWP) cannot really handle the case where there is no rain in the radar image / QPE. The problems are not only technical but also scientific: how to determine optical flow and AR/scaling parameters from a field of zeroes? And subsequently, how to extrapolate/blend?
We see 2 possible problem scenarios:

• QPE is zero and NWP is nonzero
  Here, as a workaround, one could use the "past" NWP forecasts as if they were observed QPE fields, and compute optical flow and other parameters on these fields. One could do a blending, but it's not clear how, as the AR parameters of the QPE are undefined.
• QPE is zero and NWP is zero:
  This is a bigger problem since there's simply no rain to calculate any optical flow or noise parameters. One could immediately use the NWP instead of blending, but that does not yet give us a way to generate variability in the ensemble forecast. A possible solution is to use climatological values (or values of the past X days) to determine the noise parameters. That way, you can generate some noise in the ensemble members (to generate variability when it actually starts to rain in the NWP forecast).

What are your thoughts on this?

Cheers,

Michiel and Lesley

[dnerini]

Dear Michiel and Lesley

Many thanks for starting this important discussion.
(let's ping @pulkkins @RubenImhoff @loforest directly here too)

For radar only nowcasts, the problem is somewhat simpler. Some routines will have no problems in dealing with all zeros: e.g. the LK optical flow will simply return a field of zero displacement. Others, the steps nowcasts for example, will fail (as noted in #282).

I guess that in the past we have mostly dealt with this before calling the main nowcasting routine: if no precipitation is observed by radar in the recent past, do not run the nowcast as it will be all zeros anyway. I acknowledge that it would be better to deal with such cases within the nowcasting method itself (if input is all zeros, return a forecast with all zeros), or at least raise a clear error message. Clearly, this starts by adding tests cases for nowcasting with input without precipitation.

For blending this gets more complicated, as you nicely summarized above. Maybe we should ask directly Alan? Surely he had to find a solution for Australia ...

[RubenImhoff]

Hi Michiel, Lesley and Daniele,

Very relevant points that we come across often in operational systems..

I think that the proposed options by you and Daniele (also in his PR) are worth the try. Regarding having no AR parameters because of no rain in the rainfall estimates for the nowcasting part (in STEPS nowcasting or blending), would it be an option to have a "base set" of parameters in case the model returns nothing (when there is no rain)? In any case, let's see what Alan and colleagues have done at BoM. Perhaps they have come up with a similarish solution.

I like your idea for when the QPE of the observation and the NWP forecasts are zero. This may be worth the try, although I find it hard to judge how much implementation time this will take - i.e., is it an easy try out or a lot of work to test the idea?

[RubenImhoff]

@ladc and @michiel, any updates about this issue? Or is this something we should try to pick up together soon?

[ladc]

Hello Ruben,

Not yet, indeed. This could be one of the issues to tackle during the pysteps workshop :-)
I'll check this with Alan, I had not yet done so.

[ladc]

Hello everyone,

I summarize Alan's reply below

• Set hreshold at around 3% raining fraction: default auto-correlations [code from STEPS1] below this fraction. Default (lag2) auto-correlations use a power law for the correlation length as a function of scale + black magic
• If QPE zero and NWP zero: resulting forecast = all zeros. With NWP ensembles this is not so likely - EPS may quantify errors in NWP forecast (or not). NB: checking for this case saves a lot of computer time since it is likely to be the case around 80-90% of the time.
• If QPE zero and NWP nonzero, NWP provides the advection. Set radar mean and variance and advection to zero, use default power spectrum slopes, and the default auto-correlations set the rate of blending from zero rain to NWP forecasts through the calculations of nowcasting skill as a function of lead time and scale.

He also noted that STEPS1 kept running mean (30d) for the parameters; transition from observed to climatological parameters over 6h (this is implemented in our pysteps blending method, but Ruben found that 3d gave a better performance).

[ladc]

I think we can now consider this issue solved, thanks everyone!

[dnerini]

Hi @ladc great to hear that you managed to implement a solution! Are you planning to merge those contributions back to pysteps soon? Till then, I'd rather keep this issue open

[ladc]

Yes, you're right, closing the issue was a bit premature :-)
I think these changes can be merged back after review, but it does change some default behaviour of the blending, so a new minor version number should be assigned.