Investment appraisal fails when demand is equal in 2 consecutive years in (a slight adaptation of) the simple model

[ahawkes]

Describe the bug

I modified the "simple" model so there are more milestone years. If two MSY in a row have the same service demand input (RSHEAT in this case), investment appraisal fails. It appears to calculate NaN for the metric.

A clear and concise description of what the bug is, including error messages.

[09:39:10 INFO muse2::simulation] Milestone year: 2034
[09:39:10 INFO muse2::simulation] Running agent investment...
[09:39:10 ERROR muse2] Agent investment failed

Caused by:
No feasible investment options for commodity 'GASNAT' after appraisal

To Reproduce

Run attached model

Expected behavior

I can't see a reason why investment appraisal should fail - it should be fine.

simple.zip

[alexdewar]

Sorry @ahawkes, I didn't get a notification about this... I've just taken a look at it now.

The main problem here seems to be that in the place where it's failing, demand is very nearly met but there is one timeslice with 7.105427357601002e-14 worth of unmet demand. I hacked the code to use a threshold of 1e-10 and it ran to completion (results attached, if you're interested). The problem is that I don't want to hardcode this threshold, because the user could be working with a system where the flows have very small values -- it doesn't seem quite right to use the same threshold for every model. Can you think of a principled way of calculating the threshold? If not, we can always make it another model parameter. It's not ideal to put this decision on the user either though.

What do you think? We could also consider telling the user how much unmet demand there is in the error message, which might help them fix the model.

There is a secondary problem: no asset should get a value of NaN for a metric, so that's a bug. The reason it's happening in this case is because the asset in question had no activity in any time slice, so the denominator is zero for the LCOX equation and we get NaN. The correct thing to do would be to just skip this junk asset. This bug isn't related to the problem you're seeing though (MUSE2 is correctly ignoring it because it has zero capacity, so isn't a feasible investment anyway).

[alexdewar]

Forgot to attach results. Here they are: results_for_adam.zip

[ahawkes]

Thanks. 1e-10 probably fine. I thought an alternative might be 1e-6 of demand in the time slice, but I guess that also doesn't work when demand numbers are really small. Maybe just throw a helpful error when demand is really really small AND there's unmet demand?

[alexdewar]

So, to be clear, are you saying you think we should use this threshold by default? Or should we make it opt-in (defaulting to no threshold)?

We should probably just include the total unmet demand in the error message when investment fails, whether it's large or small.

[ahawkes]

Yes, by default. Can it be an optional input parameter? And default to 1e-12? We could see how that goes.
Maybe we can raise a warning if demand in the time slice is close to this number e.g. if demand is less than 100x the number?
Also yes if it fails then a message that highlights the unmet demand point would be useful, so the user knows what to change.
There are probably Highs tolerance parameters that will influence how large the number can be.

[alexdewar]

Yes, by default. Can it be an optional input parameter? And default to 1e-12? We could see how that goes. Maybe we can raise a warning if demand in the time slice is close to this number e.g. if demand is less than 100x the number? Also yes if it fails then a message that highlights the unmet demand point would be useful, so the user knows what to change. There are probably Highs tolerance parameters that will influence how large the number can be.

Sure, I'll make it a param.

I'm not sure about warning if we're close to the tolerance though. The problem is that as we're subtracting demand tranche by tranche we'll probably often end up close to the tolerance anyway, so we may end up with a lot of false positives.

I checked the HiGHS docs and it seems they use 1e-7 as a default for various tolerance params (e.g. see here) -- which is higher than I'd expect. (Of course this won't necessarily relate to our threshold in an obvious way.)

[Aurashk]

Should we have a relative tolerance with a sensible default here too? I.e in each time slice we consider that there is remaining demand if remaining_demand > rtol * previous_unmet_demand + atol where atol = rtol = 1e-12 by default? It seems like the degree of error from highs will be proportional to the current unmet demand. rtol should only matter if previous_unmet_demand can be large though.