Monte Carlo Revisited

Monte Carlo, from the Mary-Mare collection. Photo credit: Mary Katrantzou

Our previous visit to Monte Carlo (‘That white tuxedo … at last!’, 26 November 2020) furnished an example of how Monte Carlo simulation can be used to investigate the behaviour of models whose behaviour cannot be determined analytically. The model used in the example was the bushfire model (BUSHF1), and the object of the investigation was to see how variation in fire ignition capability affects mean instantaneous fire size and mean fire lifetime. (Yes, of course I’m going to be precise here: ‘mean’ means ‘arithmetic mean’.) The conclusion drawn from the simulation was that there seems to be some sort of tipping point in the model’s behaviour at an ignition capability of around 0.35. Fires with ignition capabilities less than 0.35 burn out naturally, some much quicker than others; fires with ignition capabilities greater than 0.35 rarely burn out naturally. The fires in the first group are not really dangerous; they may be inconvenient and therefore sensible to fight, but if all else fails they can be left to themselves. The fires in the second group are the dangerous ones; if they are not fought – by whatever means are possible – then they will consume much of what lies in their path.

Let’s assume now that we’ve been approached by Dodgy Brothers GmbH, a company that is seeking to become established in the field of bushfire prediction and control. They’ve come across the BUSHF1 program – perhaps by downloading it from theendtoendblog – and are interested in having us use it to estimate mean fire sizes for fires with known ignition capabilities. Yes, we can do that, and we accordingly design a suitably comprehensive programme of work. Naturally we quote our usual consultancy rates. We are rebuffed – unsurprisingly given Dodgy Brothers’ cheapskate reputation. They don’t want the work we’ve proposed; instead they want just two numbers – an estimate of the mean fire size for fires with ignition capabilities in the range 0.30 to 0.35, and an estimate of the mean fire size for fires with ignition capabilities in the range 0.35 to 0.40. They tell us we should get these estimates by running BUSHF1 for two ignition capabilities – 0.325 and 0.375. Their logic in saying this is that 0.325 – the arithmetic mean of 0.30 and 0.35 – is a best guess for the ignition capability of fires in the first group; similarly 0.375 – the arithmetic mean of 0.35 and 0.40 – is a best guess for the ignition capability of fires in the second group. “Always use the best guess for a model’s input,” they say, “then you automatically get the best guess for the model’s output. Best guess in, best guess out – that’s the rule!”

Oh dear!

Monte Carlo simulations illustrating Jensen’s Inequality

This diagram summarises the results of four further Monte Carlo simulations run using BUSHF1. The histograms are for the fire size at a timestep. The left-hand histograms (green annotation) are for fires in the first group of ignition capabilities (0.30 to 0.35); the right-hand histograms (red annotation) are for fires in the second group (0.35 to 0.40). The upper histograms are for fires having the best guess ignition capabilities (0.325 and 0.375); the lower histograms are for fires having ignition capabilities spread evenly across the ranges 0.30 to 0.35 and 0.35 to 0.40. The calculated mean fire size for fires having the best guess ignition capability is not a good estimate of the true mean fire size. For fires in the first group it is an underestimate (16.56 against 19.40); for fires in the second group it is an overestimate (58.50 against 54.84).

This underestimation and overestimation is not specific to this example. It follows automatically from what mathematicians call Jensen’s Inequality: https://en.wikipedia.org/wiki/Jensen’s_inequality#:~:text=In%20mathematics%2C%20Jensen%27s%20inequality%2C%20named%20after%20the%20Danish,equal%20to%20the%20mean%20applied%20after%20convex%20. Savage termed this ‘The Flaw of Averages’ – a name that has stuck; see ‘Two Questions’, 3 December 2020.

To appreciate what’s involved in this, look at the inset plot at the top of the diagram. The relationship between mean fire size and ignition capability is convex-down to the left of the tipping point (green annotation) and concave-down to the right (red annotation). The mean value of a function that is convex-down is necessarily greater than the value of the function evaluated at the mean input (therefore 19.40 > 16.56); the mean value of a function that is concave-down is necessarily less than the value evaluated at the mean input (therefore 54.84 < 58.50). The only situation in which the mean value of a function can be guaranteed to be equal to the value of the function evaluated at the mean input is when the function is strictly linear over the range involved – a highly unlikely situation for any even halfway realistic model of a natural system.

The take-home message? Firstly, from theendtoendblog’s ‘Ten Point Guide to Safe Modelling’:

Beware of averaging.

Secondly, from an article on rock property estimation by two Stanford University petroleum geophysicists (Mukerji, T. and Mavko, G., 2008. The flaw of averages and the pitfalls of ignoring variability in attribute interpretations. The Leading Edge 27, 382-384 – once again I’m sorry that for copyright reasons I can’t provide you with direct access to the paper):

‘When there is variability and nonlinearity, calculations using single-point values are almost worthless. …Calculations based on a single “best guess” input may not give the “best guess” output. …One should not expect to get even correct average results using average values of inputs. …Simple simulations to account for the variability can be easily performed. Monte Carlo simulations…help to avoid the flaw of averages.’

Thirdly, enjoy Monte Carlo.

Fourthly, steer well clear of Dodgy Brothers.

Comments are closed.