Settings reference

settings_training.py and settings_analysis.py are the source of truth. This page groups their responsibilities and then includes both files verbatim. Website builds read the files; they do not import them or execute make_settings.

Training settings: what is trained

Section

Responsibility and important conventions

experiment, output, seed

Trusted module:class experiment, output folder, base random seed.

problem

Observation dimension, ordered inferred parameters, fixed likelihood coordinates, bounds and prior-shape parameters.

cohort

Priors and independent training replicas repeated within a configuration.

priors

Named prior definitions; by_parameter may choose a family for an inferred coordinate.

data

Training, validation, and test row counts. Each must be even for exactly balanced classes.

model

mlp or residual; hidden widths or residual width/blocks; activation and dropout.

training

Epochs, global batch size, optimizer, learning rate, precision, clipping, checkpoint policy, evaluation batch size.

runtime

CPU/CUDA selection, independent jobs versus DDP, CPU threads, determinism, process-group timeout. Also used by analyze.

Choice marks sweep alternatives here. training.batch_size is global even under DDP. training.evaluation_batch_size also sets the model-evaluation batch during analysis.

Analysis settings: what is computed from the trained models

Section

Responsibility and important conventions

analysis

Trusted module:class analysis: the registered products, metrics, and plots.

inference

Truth design, observation ensembles, candidate measure/resolution, HLD levels, targets, score retention, and analysis limits.

verification

Shared pairwise-ratio, reweighting, normalization, and showcase data.

metrics, plots

Literal lists of registered hook names to run.

plotting

Figure resolution, histogram sizes, prior samples, and displayed inference cases.

Choice is rejected here: an analysis is one setting, and the latest one wins. inference.truth_design is sobol (default), grid or size_adaptive; see Truth design. The max_* inference settings are the analysis limits checked before anything is computed. HLD levels are enclosed masses, not tail probabilities. inference.retain="diagnostic" keeps score rows for a subset while preserving per-case summaries; "full" retains all score rows. The limits estimate selected inference arrays, not complete peak RAM, disk use, or wall time.

Each successful launch also copies the settings files it used into outputs/<profile>/; see Change settings and run again.

Presets

Both files define the same presets; --profile selects one in both.

Preset

Training

Analysis

default

The full study exactly as defined in SETTINGS, dimensions one through six.

The full analysis exactly as defined in SETTINGS.

paper

The paper’s exponential sign.

Exhaustive truths while feasible, no truth margin, continuous ratio-candidate measure, and ratio/exact targets.

smoke

Small four-prior end-to-end run.

Small analysis of the smoke models.

The scientific notes map these settings to the paper and explain the default/paper differences in detail.

Complete settings and preset definitions

settings_training.py
 1"""Training settings: what is trained. Analysis choices live in settings_analysis.py.
 2
 3Choice([...]) means baseline + one-factor alternatives, NEVER a Cartesian product.
 4An ordinary list is one value (hidden layer widths, parameter names, cohorts, etc.).
 5The four training priors are an explicitly repeated comparison cohort in every run.
 6"""
 7from copy import deepcopy
 8from nnpd import Choice
 9
10DEFAULT_PROFILE = "default"
11PROFILES = ("default", "paper", "smoke")  # settings_analysis.py defines the same profiles
12
13SETTINGS = {
14    "experiment": "gaussian:GaussianExperiment",
15    "output": "outputs/default",
16    "seed": 0,
17    "problem": {
18        "dimension": Choice([1, 2, 3, 4, 5, 6]),
19        "infer": ["mean:*"],       # e.g. Choice([["mean:*"], ["mean:*", "std:1"]])
20        "fixed_mean": 5.0,
21        "fixed_std": 2.0,
22        "fixed": {},                # e.g. {"mean:0": 4.0}, only for uninferred parameters
23        "parameters": {
24            "mean": {"low": 0.0, "high": 10.0, "normal_mean": 5.0, "normal_std": 4.0,
25                     "exponential_rate": 0.1, "grid_step": 0.2},
26            "std": {"low": 0.2, "high": 4.0, "normal_mean": 2.0, "normal_std": 0.7,
27                    "exponential_rate": 0.1, "grid_step": 0.2},
28        },
29        "overrides": {},            # e.g. {"mean:1": {"low": 1.0, "high": 9.0}}
30    },
31    "cohort": {
32        "priors": ["uniform", "normal", "exponential", "grid"],
33        "replicas": 1,              # independent training replicas; not the 15 inference observations
34    },
35    "priors": {
36        "uniform": {"family": "uniform"},
37        "normal": {"family": "normal"},
38        "exponential": {"family": "exponential"},
39        "grid": {"family": "grid"},
40        # A family may be overridden by inferred name:
41        # "mixed": {"family": "uniform", "by_parameter": {"mean:0": "grid"}},
42    },
43    "data": {"train_size": 70_000, "validation_size": 15_000, "test_size": 15_000},
44    "model": {
45        "kind": "mlp",             # Choice(["mlp", "residual"])
46        "hidden": [64, 64, 64, 64],
47        "width": 64, "blocks": 3,  # residual model only
48        "activation": "relu", "dropout": 0.0,
49    },
50    "training": {
51        "epochs": 5, "batch_size": 128,  # GLOBAL batch size in DDP too
52        "optimizer": "adam", "learning_rate": 0.001,
53        "betas": [0.9, 0.999], "epsilon": 1e-8, "weight_decay": 0.0,
54        "precision": "off",         # off, float16 (CUDA), bfloat16
55        "gradient_clip": None, "checkpoint": "last", "verbose": True,
56        "evaluation_batch_size": 65_536,  # rows per model evaluation, also during analysis
57    },
58    "runtime": {                   # also used by analyze; cannot vary within one launch
59        "device": "auto",          # auto, cpu, cuda, cuda:0; torchrun picks LOCAL_RANK
60        "parallel": "jobs",        # jobs: parallel full runs; ddp: train each model together
61        "cpu_threads": 1, "deterministic": True, "timeout_minutes": 60,
62    },
63}
64
65
66def make_settings(profile: str = DEFAULT_PROFILE) -> dict:
67    if profile not in PROFILES:
68        raise ValueError(f"Unknown profile {profile!r}; choose from {PROFILES}.")
69    settings = deepcopy(SETTINGS)
70    settings["output"] = f"outputs/{profile}"
71    if profile == "paper":
72        settings["problem"]["parameters"]["mean"]["exponential_rate"] = -0.1
73    if profile == "smoke":
74        settings["problem"]["dimension"] = 1
75        settings["data"] = {"train_size": 1024, "validation_size": 256, "test_size": 256}
76        settings["model"]["hidden"] = [32, 32]
77        settings["training"].update(epochs=2, evaluation_batch_size=2048)
78    return settings
settings_analysis.py
 1"""Analysis settings: what is computed from the trained models. Analysis never trains.
 2
 3A profile here analyzes the models trained with the same profile in settings_training.py.
 4Change anything and run `analyze` again: the saved models are reused, and the latest
 5analysis replaces the previous one. Choice is not allowed here; sweeps are for training.
 6"""
 7from copy import deepcopy
 8
 9PROFILES = ("default", "paper", "smoke")  # the same profiles as settings_training.py
10
11SETTINGS = {
12    "analysis": "gaussian:GaussianAnalysis",
13    "inference": {
14        "seed": 2026,
15        "observations": 15,         # one common theta for these independent observations
16        "repeats": 1,               # independently repeated observation ensembles per truth
17        "truth_design": "sobol",   # sobol; grid: truth_points_per_axis**p; size_adaptive: grid up to max_grid_truths, else sobol
18        "sobol_truths": 1_024,      # Sobol truths at p=1 inferred parameter; doubles with each additional one
19        "truth_points_per_axis": 25, "max_grid_truths": 1_000_000,  # grid and size_adaptive only
20        "truth_margin_fraction": 0.1,
21        "align_truths_to_grid": True,
22        "candidate_design": "sobol",  # sobol or grid (grid can become enormous)
23        "candidate_count": 16_384,
24        "candidate_points_per_axis": 200,  # used only for candidate_design='grid'
25        "native_grid_prior": True,  # discrete prior uses counting measure on its support
26        "levels": [0.68, 0.95],     # ENCLOSED mass, not tail probabilities
27        "targets": ["ratio", "posterior", "exact"],
28        "retain": "diagnostic",    # diagnostic: a few cases; full: all candidate scores
29        "diagnostic_cases": 3,
30        # Analysis limits: run/analyze refuse up front if any trained model exceeds them.
31        "max_model_evaluations": 2_000_000_000_000,
32        "max_saved_bytes": 8_000_000_000,
33        "max_candidate_points": 2_000_000,
34    },
35    "verification": {
36        "seed": 2027, "pairs": 256, "pair_design": "grid", "samples_per_endpoint": 512,
37        "source_samples": 2048, "target_samples": 2048, "directions": 32,
38        "marginal_samples": 1000, "normalization_thetas": 5,
39        "normalization_design": "diagonal", "normalization_on_grid": True,
40        "showcase": {"source_fraction": 0.3, "target_fraction": 0.7,
41                     "samples": 1_000_000, "bins": 1000},
42    },
43    "metrics": ["classification", "bias", "coverage", "width", "resolution",
44                "posterior", "exact_reference", "pairwise", "reweighting", "normalization"],
45    "plots": ["training", "prior", "inference", "pairwise", "reweighting", "showcase"],
46    "plotting": {"dpi": 140, "bins": 100, "prior_points": 200_000, "max_inference_cases": 200},
47}
48
49
50def make_settings(profile: str) -> dict:
51    if profile not in PROFILES:
52        raise ValueError(f"Unknown profile {profile!r}; choose from {PROFILES}.")
53    settings = deepcopy(SETTINGS)
54    if profile == "paper":
55        settings["inference"]["truth_margin_fraction"] = 0.0
56        settings["inference"]["align_truths_to_grid"] = False
57        settings["inference"]["truth_design"] = "size_adaptive"  # exact 25**p grid while feasible (p <= 4)
58        settings["inference"]["targets"] = ["ratio", "exact"]
59        settings["inference"]["native_grid_prior"] = False  # continuous domain in paper Eq. 4
60    if profile == "smoke":
61        settings["inference"].update(truth_design="grid", truth_points_per_axis=7,
62                                     candidate_count=256, diagnostic_cases=2)
63        settings["verification"].update(pairs=4, samples_per_endpoint=32, source_samples=64,
64                                        target_samples=64, directions=4, marginal_samples=128,
65                                        normalization_thetas=4)
66        settings["verification"]["showcase"].update(samples=256, bins=30)
67        settings["plotting"].update(prior_points=1000, bins=30)
68    return settings