Add built-in function for local sensitivity analysis

[EwoutH]

Add a function that automatically configures the batchrunner to perform local sensitivity analysis. The only required input is a model.

Extensive docstring is included, see that for details.

Feedback, both conceptually and implementation wise, is very welcome!

Usage example:

# Define a relative range (e.g., 90% and 110% of the default value)
common_range = [0.9, 1.1]

# Perform the sensitivity analysis
results = local_sensitivity_analysis(
    model_cls=MyModel,
    common_range=common_range,
    ignore_parameters = ['migration_rate']
    iterations=10,  # Number of iterations for each parameter set
    max_steps=500   # Maximum number of steps per model run

[codecov]

Codecov Report

Attention: 23 lines in your changes are missing coverage. Please review.

Comparison is base (ca5e8f9) 77.35% compared to head (ab7bea4) 75.58%.
Report is 54 commits behind head on main.

Files	Patch %	Lines
mesa/batchrunner.py	8.00%	23 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1908      +/-   ##
==========================================
- Coverage   77.35%   75.58%   -1.78%     
==========================================
  Files          15       15              
  Lines        1007     1032      +25     
  Branches      220      211       -9     
==========================================
+ Hits          779      780       +1     
- Misses        197      223      +26     
+ Partials       31       29       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[EwoutH]

@jackiekazil @tpike3 @Corvince @quaquel curious what you think!

[rht]

Concept SGTM. Having more canned helper function like this would encourage fast exploration from a Jupyter notebook. This reminds me of the toolkit in the fastai library.

[quaquel]

You know my opinion on One At a Time Sensitivity Analysis: thou shall not use it.

The broader question is what is within scope for MESA? Having some simple helper functions for model exploration seems sensible. However, with libraries like SALib and the EMA workbench, I would point people to those for more sophisticated analyses.

[EwoutH]

You know my opinion on One At a Time Sensitivity Analysis: thou shall not use it.

Just curious: Assume you you have compute budget for about a thousand model runs. Your model is stochastically quite noisy, you need about 10 runs to get your metrics within a satisfyingly small confidence interval. That means you can test about 100 configurations. You have around 10 uncertanties (and maybe 3 policy levers with 3 magnitudes each, if relevant).

How would you approach sensitivity / extreme value analysis?

(this is an actual scenario I encountered in a project. Runtime 5 to 7 min, 12 simultaneous runs on my PC, so ~100 to ~150 runs per hour. One night of simulation was about a thousand runs)

[quaquel]

Use common random numbers to reduce the noise. Use system knowledge and the question at hand (i.e., the purpose of the model) to carefully select the scenarios to run. The point with one-at-a-time sensitivity analysis is that it is bound to produce misleading results because of interaction effects. Another more sophisticated direction is to use adaptive sparse grids (see this thesis), which is what was used to do Sobol on the CovidSim.

[EwoutH]

Another more sophisticated direction is to use adaptive sparse grids (see this thesis)

Looks very interesting. Do we have this in the workbench? Do we want it?

[rht]

I'm new to the sensitivity analysis libraries in the ABM context. The sensitivity analysis of my past work, on economic models, is mainly guided by intuition. I am eager to learn the systematic feature learning under uncertainty. In the context of Mesa, it would be great if there is a representative example model that showcases this.

The point with one-at-a-time sensitivity analysis is that it is bound to produce misleading results because of interaction effects.

I can't quite exactly unpack this sentence. If you look at variation in only 1 parameter, with other parameters ceteris paribus, they shouldn't affect how I should interpret the effect of a particular parameter?

[EwoutH]

@jackiekazil @tpike3 @Corvince @wang-boyu, from your modelling and teaching experience, what do you think of a local (single variable) sensitivity analysis method in Mesa?

[quaquel]

Another more sophisticated direction is to use adaptive sparse grids (see this thesis)

Looks very interesting. Do we have this in the workbench? Do we want it?

yes, but it is not trivial algorithmically.

[quaquel]

I'm new to the sensitivity analysis libraries in the ABM context. The sensitivity analysis of my past work, on economic models, is mainly guided by intuition. I am eager to learn the systematic feature learning under uncertainty. In the context of Mesa, it would be great if there is a representative example model that showcases this.

The point with one-at-a-time sensitivity analysis is that it is bound to produce misleading results because of interaction effects.

I can't quite exactly unpack this sentence. If you look at variation in only 1 parameter, with other parameters ceteris paribus, they shouldn't affect how I should interpret the effect of a particular parameter?

Ceteris paribus is exactly the problem here. Take the Ishigami function below, a classic sensitivity analysis test function, as an example. If you vary $x_1$, $x_2$, and $x_3$ in isolation, this will work fine for $x_1$ and $x_2$. However, the results for $x_3$ will critically depend on the assumed value for $x_1$. If $x_1=0$, $x_3$ will not show any impact. If $x_1=\pi$, $x_3$ will show very high sensitivity. This is also known as an interaction effect. Complex system models are complex in part because they have such interaction effects. And thus when analyzing complex systems models, you should use global sensitivity analysis. For more details see this paper.

$$ f(x) = sin(x_1) + a sin^2(x_2)+ b x^4_3 sin(x_1) $$

[rht]

That makes sense. It's, in a way, similar to surveying the relevant local extrema (and additionally identifying the global optimum) of a function. It's both of SciPy's global optimization and local minima finding.

@jackiekazil @tpike3 @Corvince @wang-boyu, from your modelling and teaching experience, what do you think of a local (single variable) sensitivity analysis method in Mesa?

What I had in mind was actually to efficiently map the Schelling model

using the adaptive sparse grid, if the method can be packaged in a nice API call in Mesa. The bulk area doesn't require much traversal, but OTOH the boundary between the 2 phases of the system would require a detailed mapping.

For more details see this paper.

It's somewhat a tangential point, but a simulation in Python being slow sometimes discourages people from doing sensitivity analysis.

[jackiekazil]

This is interesting to me, but part of me wonders -- Is this "core" functionality or is this the start of something bigger that is an analyst tool kit?

[rht]

This is interesting to me, but part of me wonders -- Is this "core" functionality or is this the start of something bigger that is an analyst tool kit?

I think given that there are already existing SA tools (SALib and the EMA workbench), it would make more sense to document their usage in the examples, instead of reinventing the wheels.

[quaquel]

I agree with @rht that showing how other tools can be used in conjunction with MESA is the way to go.

However, if you try to use numpy.qmc or SALib with MESA you do run into the issue that the current BatchRunner does not really support this. So, as a next step, why not make an example with numpy.qmc or SALib and see what would be needed in the BatchRunner to keep the code nice and concise?

The EMA workbench has a MESA example: https://emaworkbench.readthedocs.io/en/latest/examples/example_mesa.html so it would be easy to point to that rather than repeat it.

[rht]

The EMA workbench has a MESA example: https://emaworkbench.readthedocs.io/en/latest/examples/example_mesa.html so it would be easy to point to that rather than repeat it.

From my reading of the example,

    # Define model parameters and their ranges to be sampled
    model.uncertainties = [
        IntegerParameter("num_nodes", 10, 100),
        IntegerParameter("avg_node_degree", 2, 8),
        RealParameter("virus_spread_chance", 0.1, 1),
        RealParameter("virus_check_frequency", 0.1, 1),
        RealParameter("recovery_chance", 0.1, 1),
        RealParameter("gain_resistance_chance", 0.1, 1),
    ]

provides a range of the parameters to be sampled. Is a rigid grid scan (num_nodes x avg_node_degree x virus_spread_chance x ...) of the parameters performed for the sensitivity analysis, or is it a grid with adaptive resolution?

[quaquel]

The EMA workbench has a MESA example: https://emaworkbench.readthedocs.io/en/latest/examples/example_mesa.html so it would be easy to point to that rather than repeat it.

From my reading of the example,

    # Define model parameters and their ranges to be sampled
    model.uncertainties = [
        IntegerParameter("num_nodes", 10, 100),
        IntegerParameter("avg_node_degree", 2, 8),
        RealParameter("virus_spread_chance", 0.1, 1),
        RealParameter("virus_check_frequency", 0.1, 1),
        RealParameter("recovery_chance", 0.1, 1),
        RealParameter("gain_resistance_chance", 0.1, 1),
    ]

provides a range of the parameters to be sampled. Is a rigid grid scan (num_nodes x avg_node_degree x virus_spread_chance x ...) of the parameters performed for the sensitivity analysis, or is it a grid with adaptive resolution?

when calling perform_experiments, you specify the sampling technique. This code defines the space.