CPU, CUDA, and multiple GPUs

These are the execution modes implemented by nnpd/core/runtime.py and the bundled trainer. Custom trainers must implement their own distributed contract. The test suite runs CPU and two-process CPU (Gloo) execution everywhere; its CUDA tests run only on machines with suitable GPUs, and multi-node launches are not tested.

Choose a device

Edit runtime.device in settings_training.py; analyze uses the same runtime:

Value	Behavior
"auto"	CUDA when available, otherwise CPU.
"cpu"	Explicit CPU execution.
"cuda"	Require CUDA; fail clearly if unavailable.
"cuda:1"	Select a single indexed device outside a distributed launch.

Use a compatible CUDA-enabled PyTorch installation for GPU runs. On Linux, the PyTorch that uv sync installs from PyPI is built for CUDA 13 and needs an NVIDIA driver that supports it. For another CUDA version, ROCm, or a CPU-only build, point uv at the matching PyTorch index as described in uv’s PyTorch guide; the PyTorch installation selector lists the available builds. Hardware and package availability do not follow automatically from selecting a configuration string.

Independent jobs on two GPUs

Set runtime.parallel="jobs" and runtime.device="cuda" (or "auto") in settings_training.py, then launch from a shell:

CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc_per_node=2 run.py run --profile smoke

Workers handle disjoint complete jobs, including their analysis. train and analyze can be launched the same way. Each model fits on one GPU. This is not tensor/model parallelism and does not combine GPU memory for one network. Common compatible products can be shared through the output store.

Collective DDP training

Set runtime.parallel="ddp" in settings_training.py, then launch:

CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc_per_node=2 run.py run --profile smoke

All workers train each model together. Its analysis runs on rank zero. training.batch_size is the global batch size, not the batch size per GPU. The bundled trainer handles unequal or empty final shards without intentionally dropping or duplicating real examples.

Use device="cuda", not "cuda:0", under a multi-worker launch; local rank selects the GPU. Use one worker per visible GPU. The CUDA collective backend is NCCL; CPU DDP uses Gloo. Setting parallel="ddp" with only one process does not itself launch additional processes.

Precision, reproducibility, and restoration

training.precision accepts "off", "float16", or "bfloat16". The bundled float16 path requires CUDA. bfloat16 also depends on device/operation support. Deterministic settings do not promise identical weights across all devices, world sizes, or dependency versions.

Backend identity participates in the model cache key. Changing CPU/CUDA or DDP world size is not treated as an identical training artifact. To inspect an already trained model on CPU, use restore(..., device="cpu"); do not expect a new CPU analyze launch to match a GPU training recipe automatically.

Interrupted training restarts at stage level: complete model weights are saved, but mid-epoch optimizer, RNG position, and scaler state are not checkpointed.

Distributed limits

Runtime settings cannot be swept within one launch. Multi-node launches need correct torchrun rendezvous configuration and a shared filesystem whose locks and atomic operations work across workers. No site-specific scheduler settings are supplied. Do not run two independent analyses that write the same result directory concurrently.

See Storage and Testing and validation for the precise guarantees and validation boundaries.