Configuration and one-factor sweeps¶

Two settings files¶

settings_training.py decides what is trained; settings_analysis.py decides what is computed from the trained models. Each defines SETTINGS, PROFILES, and make_settings(profile), which deep-copies SETTINGS and applies explicit preset adjustments. Values set by a preset override the corresponding values in SETTINGS. settings_training.py also defines DEFAULT_PROFILE, the profile used for both files when none is given.

Settings file	Class it names	Other sections
`settings_training.py`	`"experiment"`: an `Experiment`	`output`, `seed`, `problem`, `cohort`, `priors`, `data`, `model`, `training`, `runtime`
`settings_analysis.py`	`"analysis"`: an `Analysis`	`inference`, `verification`, `metrics`, `plots`, `plotting`

The settings reference displays both complete files directly, so names and default values are not maintained in a separate copy. Application and settings modules are trusted Python, not sandboxed settings.

Baseline plus one changed knob¶

Sweeps are for training settings. Use Choice to mark alternatives:

from nnpd import Choice, expand_sweep

variants = expand_sweep({
    "data": {"train_size": Choice([50_000, 100_000, 150_000])},
    "training": {"learning_rate": Choice([0.1, 0.01, 0.001])},
})
assert len(variants) == 5

Training size	Learning rate	Changed setting
50,000	0.1	Baseline
100,000	0.1	`data.train_size`
150,000	0.1	`data.train_size`
50,000	0.01	`training.learning_rate`
50,000	0.001	`training.learning_rate`

The first option is always the baseline. Every other option changes just its own knob against that baseline. Repeated resolved configurations are removed. Dictionary insertion order determines presentation order, not configuration identity. There is no hidden Cartesian-product mode.

Literal lists and atomic choices¶

An ordinary list is one value. For example, [64, 64] specifies two hidden layers. To compare complete architectures, wrap the alternatives:

Choice([[64, 64, 64, 64], [128, 128]])

Options may also be complete dictionaries. Use an atomic dictionary option when several dependent settings must change together; do not nest a Choice inside another option or inside a literal list. That nesting is rejected.

Choice is rejected in settings_analysis.py. An analysis is one setting: to compare, for example, two candidate counts, analyze with one, then change it and analyze again.

Change the model or the inferred parameters¶

The following are edits in settings_training.py, not additional command-line flags:

# Values for problem.dimension, problem.infer, and model.kind, respectively:
2
Choice([["mean:*"], ["mean:*", "std:1"]])
Choice(["mlp", "residual"])

mean:* means all observation means. std:1 adds the standard deviation of the second observation dimension; indices start at zero. At observation dimension two, these parameter sets have two and three inferred coordinates, giving model input sizes four and five. NNPD derives the input size.

Each alternative is validated against the baseline; a std:1 alternative is not valid with a one-dimensional baseline.

Unselected likelihood parameters stay fixed at fixed_mean, fixed_std, or named entries in problem.fixed. They are not marginalized nuisance parameters. Parameter-family bounds live in problem.parameters; selected coordinate overrides live in problem.overrides.

Priors and three different kinds of repetition¶

cohort.priors is normally a literal list repeated within every configuration. To make the prior itself an OFAT alternative, use:

Choice([["uniform"], ["normal"], ["exponential"], ["grid"]])

as the value of cohort.priors.

Setting	What repeats
`cohort.replicas` (training)	Independent training data/models for a prior.
`inference.repeats` (analysis)	Independent observation ensembles at each truth.
`inference.observations` (analysis)	Observations sharing one truth within each ensemble.

These settings are not interchangeable.

Fixed runtime and compatible analysis choices¶

runtime and output must be fixed within one execution launch. Compare CPU/CUDA or jobs/ddp through separate launches. runtime is in the training settings, and analyze uses it too. Inference targets, selected metrics, and plots must be compatible: for example, bias and coverage require the ratio target. Planning checks every training alternative and the analysis settings before anything runs; backend availability is checked when the runtime is created.

A useful workflow is: edit the settings, print the plan, inspect every workload’s warnings/errors, and only then execute the desired stage.

Truth design¶

The truth points are the true parameter values at which inference is evaluated. inference.truth_design in settings_analysis.py chooses how they are placed; p is the number of inferred parameters:

Value	Truths
`"sobol"` (default)	`sobol_truths × 2**(p-1)` Sobol points: 1,024 at `p = 1`, doubling per additional parameter.
`"grid"`	The exhaustive `truth_points_per_axis**p` grid. It grows exponentially with `p`.
`"size_adaptive"`	The grid while it has at most `max_grid_truths` points, otherwise the Sobol design.

The paper preset uses size_adaptive: the exact 25-point-per-axis grid for p ≤ 4 and Sobol truths above. The plan reports the design actually used.

Analysis must be feasible before anything is computed¶

The analysis cost of every model is estimated from both settings before anything is computed. max_model_evaluations, max_saved_bytes and max_candidate_points in the analysis inference settings set the limits. If any model exceeds them:

run refuses the whole launch before training, so no model is trained that could not also be analyzed;
analyze refuses before computing anything;
plan prints the plan and then exits with an error.

The error names each infeasible configuration and the settings to reduce. Nothing is reduced automatically. train alone does not check: the analysis settings can still be changed afterwards.