[Hendrik-code]

Why

A model trained on a narrow field of view has never been shown what lies outside it. On whole-body axial NAKO VIBE, the SpineGAN-trained models lose most of their Dice to false positives far from the spine. The contrast augmentations are not failing here: the network has never been asked to reject that context.

Diagnosis, measured on Ours (fp32) (paper config c69b9872 on 01bf0b3), NAKO VIBE test split:

vertebra instance precision, all four training sets, in-phase and water	0.01, median 76-125 false-positive components per subject (recall ~0.90)
FP voxels more than 50 mm from the reference spine (803 in-phase, n=10)	60%
FP voxels outside the body	1%
FP voxels by VibeSeg-100 class (803 in-phase / 804 water, n=15 each)	unlabelled 31/41%, muscle 14/21%, gluteals 9/6%, subcutaneous fat 8/4%, intestine 5/11%, femur 4/4%, sacrum 2/1%
Dice if only the largest connected component is kept (803 in-phase, n=15)	vertebra 0.729 → 0.804, IVD 0.616 → 0.696, canal 0.697 → 0.808

The cause is the training geometry. Preprocessed 803 has a median of 85 (LR) × 297 × 187 voxels at 1 mm, with a 128³ patch. The foreground leaves only 22 mm of LR free, so nnU-Net centres the slab and pads the rest:

• data is padded with 0;
• seg is padded with -1, which the tail then turns into 0;
• so every training window contains spine, and beyond ±42 mm laterally the network has only seen a flat slab labelled background.

What

RandomFOVExtensionGPU is a new GEO transform in smauglab/transforms/gpu/context.py. It runs directly after RandomAffineGPU and before every intensity transform. For each selected sample:

1. Shift. Image and mask move by the same whole-voxel offset, up to shift of the patch per axis. At shift: 1.0 the spine can leave the window entirely.
   • With spare_label_faces (default true), the shift never vacates a window face that a label comes within margin voxels of. A structure can still leave through that face, but synthetic tissue is never placed against it.
   • The spine touches both faces along its length, so it is not shifted along that axis. Which axis that is gets read off each window's labels; it is not configured.
2. Fill region. Everything that is not scan: the padding (every channel exactly pad_value) plus whatever the shift vacated.
3. Synthesise it SynthSeg-style, from the patch as it arrived, before the shift.
   • EM splits the patch's background into a random 3-10 regions. Only voxels labelled 0, beyond margin of any label and off the padding seam are used.
   • A 3×3×3 majority filter and a random diffeomorphic field (SynthSeg's random_svf_field) post-process the region map.
   • layers randomly rotated, scaled, mirrored and translated copies of that fixed map are composited into the fill.
   • Each region is rendered from its own mean and std. The texture is spatially correlated (σ = 1 voxel), then blurred as SynthSeg blurs.
   • The fill is labelled 0.

Everything outside the fill region is the shifted input, bit for bit, and with shift: 0 only the padding changes.

Commits:

• 6d889b9 adds the transform.
• 85b971b fixes two defects that the first screen exposed (see Evaluation):
  • the fill is now built from the pre-shift patch;
  • spare_label_faces is added;
  • _shift now gathers on the device, so sparing adds no host sync.
• 5a8b044 sets the shipped config to the best arm.

Other details:

• Per-sample selection is pinned. Without a pipeline section the trainer forces same_on_batch=True onto every child, which would make p one coin per batch and could empty both samples of a step of spine.
• The EM is batched and weighted. It is a form of _em_gmm_1d with no boolean indexing and no host read.
• The EM moments use a broadcast product instead of resp.mT @ design: 0.04 ms against ~7.5 ms on this shape. Without it the transform cost 190 ms per batch.
  • synthseg.functional._em_gmm_1d has the same pattern and probably the same cost. It is untouched here and left for a separate PR.
• New config: transform_params_one-sequence-to-segment-them-all-fov.json (d35b9d91). It is c69b9872 plus the transform with shift 1.0, regions [3, 10], margin 4, layers 6, spare_label_faces true and p 0.5, and nothing else. It is byte-identical to the screen's fov2_spare arm. FOV stays opt-in, and the paper config is unchanged.
• The README gets a "Field of view beyond the scan" section, and the matrix and all_augmentations.json are regenerated.

Checks

All on 5a8b044.

• pytest unit_tests, including the slow tests, with one A40 visible: 497 passed, 13 skipped, 1570 subtests. main (7169464) in the same venv: 469 passed, 13 skipped, 1519 subtests.

• The 28 new tests in test_fov_extension.py cover:

  • image and mask moving together, checked with a sentinel intensity under the label;
  • no foreground intensity reaching the fill;
  • bit-identity outside the fill;
  • the no-background case;
  • per-sample selection under same_on_batch=True;
  • seed determinism and bf16 autocast;
  • EM against two known populations;
  • the fill using background the shift removed;
  • no single region dominating a large-shift fill;
  • a fully vacated window getting a varied fill;
  • a through-label not being cut, and a one-face label shifting only away from its face;
  • the touch tolerance equalling margin;
  • sparing per row, with no RNG consumed.

  The fill and sparing tests fail on 6d889b9.

• The registry sweeps pick the transform up automatically.

• Pre-commit hooks (ruff) pass over 7169464..5a8b044. mypy smauglab/ reports no issues, run locally with mypy 2.1.0 (CI pins 2.4.0).

• Fill check on 8 real 803 patches shifted so that about 11% or 0% of the scan stays in the window:

  	6d889b9	85b971b
  share of the fill covered by the layers	8-39%	97.6% (median)
  largest region's share of the fill	75-99%	54% (median)
  fully vacated windows	flat 0	none flat

  The SI axis was spared on both faces in 8/8 patches, the other two axes in 0/8.

• Cost on an A40, from the training logs: median epoch 62 s, against 51-52 s for four paper-config runs on the same card model (about +20%).

Evaluation

Dataset803 (SpineGAN Dixon in-phase), fold 0, one run per arm. The references are four runs of the paper config c69b9872:

• ms0_base, the paired reference, on 7169464;
• Ours (fp32);
• two more replicates on 7169464.

Their spread is the noise floor. The class-mean Dice is panoptica's global Dice for the paper cohorts. For NAKO VIBE it is recomputed against VibeSeg-100, four parts pooled (436 subject-parts on test, 62 on val). Arms are chosen on val and reported on test. TOST is at ±0.01, alpha 0.05.

First screen (6d889b9, L40S)

Dice delta vs ms0_base, test:

arm	NAKO VIBE fat / in / out / water	SpineGAN Dixon in-phase (in-domain)	paper cohorts TOST-equivalent
fov_fill (shift 0)	-0.003 / +0.032 / +0.015 / -0.013	-0.0001	all
fov_shift050 (shift 0.5)	+0.014 / +0.043 / +0.059 / +0.058	-0.0060	all but SpineGAN CT
fov_shift100 (shift 1.0)	+0.066 / +0.089 / +0.083 / +0.081	-0.0092	all but Dixon in-phase

• fov_shift100 removed 94% of the far FPs: voxels more than 20 mm from the reference spine fell from 37.9k to 2.30k per subject-part.
• It failed two of the three pre-stated criteria:
  • recall fell, with FN up 43%, concentrated in the posterior elements and the superior fifth of the spine;
  • ms0_base with largest-connected-component (LCC) post-processing still beat it on every class;
  • in-domain Dixon in-phase fell by 0.0092, so TOST failed.
• Two causes in the code:
  1. The fill was built from what the shift left. After a large shift only a sliver of scan remains, so one EM region filled 75-99% of the window, and a fully vacated window stayed flat 0. Under U[-1, 1]³ about 60% of shifted samples keep under 10% of the scan.
  2. Every SI shift cut the spine. The spine runs through the window along SI, so the shift gave it an artificial end with background-labelled tissue beyond.

85b971b fixes both. The criteria for the second screen were written down before it launched.

Second screen (85b971b, A40)

Both arms use fov_shift100's settings. fov2_nospare turns sparing off, so fov2_spare - fov2_nospare isolates it. NAKO VIBE test, means per subject-part:

	4 replicates	v1 fov_shift100	fov2_nospare	fov2_spare (shipped)
class-mean Dice, raw	0.683-0.715	0.795	0.789	0.795
class-mean Dice, after LCC	0.810-0.815	0.805	0.791	0.807
FP > 20 mm from the spine	30.1k-43.3k	2.30k	0.64k	2.32k
FN, all	12.0k-12.7k	17.4k	20.7k	15.5k
FN > 5 mm from any prediction	305-408	1487	3812	632
FN, superior fifth of the spine	2.49k-2.69k	4.45k	5.97k	3.51k
FN, posterior elements	5.97k-6.53k	9.06k	9.86k	7.94k
SpineGAN Dixon in-phase, Dice vs ms0_base	-0.0001 to +0.0004	-0.0092	-0.0079	-0.0056

pre-stated criterion	fov2_spare	fov2_nospare
1. vs v1: total, superior-fifth and posterior FN lower; far FP at most 2x	pass (val and test)	fail: FN above v1
2. raw Dice ≥ ms0_base + LCC on vertebra, IVD and canal	fail: -0.003 / -0.012 / -0.030	fail
3. every paper cohort TOST-equivalent to ms0_base	fail, only on SpineGAN CT	fail, only on SpineGAN CT

• Sparing is the change that recovers recall.

  • fov2_spare - fov2_nospare, all FN: -5259 [-5816, -4701].
  • FN more than 5 mm from any prediction: -3180 [-3599, -2760].
  • Every paper cohort gains +0.0006 to +0.0035 Dice.
  • Far FP rises by 1686 voxels.

• The fill fix alone makes the model more conservative than v1: FN +3283 and far FP -1665.

• Against v1, fov2_spare recovers part of the recall it lost, measured as the share of v1's excess over the replicate mean:

  • total FN: 39%;
  • FN more than 5 mm from any prediction: 75%;
  • superior-fifth FN: 51%;
  • posterior-element FN: 41%.

  Far FP stays at v1's level (+21 [-327, 368] voxels), and the in-domain cost is cut from -0.0092 to -0.0056.

• Criterion 3 fails on the upper side. On SpineGAN CT, fov2_spare is +0.0053 with interval [-0.0015, +0.0122] (n=27), so equivalence cannot be shown. Every other paper cohort is equivalent with p < 0.001.

• Criterion 2 fails on IVD. With LCC applied to every model, fov2_spare is:

  • above all four replicates on spinal canal (0.830 vs 0.824-0.828);
  • 0.002-0.005 below them on vertebra;
  • 0.011-0.020 below them on IVD.

  About two thirds of its lost IVD recall is IVD predicted as vertebra (17.6% of reference IVD voxels, vs 14.1-14.6% for ms0_base and one replicate). That is a class-boundary error, not missed tissue.

What this transform is, on this evidence. It is in-model removal of far false positives on wide-field-of-view data. On NAKO VIBE it takes raw class-mean Dice from ms0_base's 0.715 to 0.795, with no post-processing. It does not beat the paper config plus LCC post-processing on class-mean Dice. Its in-domain cost (-0.0056 on Dixon in-phase) is inside the ±0.01 margin, but outside replicate noise.

Known limits

• One run per arm. v1 ran on an L40S and v2 on an A40, so v1-vs-v2 deltas also carry a card term and a different random stream. The replicate spread above bounds them only loosely.
• The 2x2 has no "v1 fill + sparing" arm, so it is open whether the fill fix still helps once sparing is on. The v1 fill was degenerate in any case, as shown in Checks.
• Only Dataset803 was screened on v2.
• Background that the scanner itself zeroed normalises to a constant other than 0, so it counts as scan and stays. An example is the masked air of a Dixon reconstruction.
  • It forms constant bands along some SpineGAN slab edges, and on some 803 patches it is 30-41% of the fill sources, where it shows as flat straight bands in the fill.
  • nnU-Net's -1 marks only part of it, about 17% in the case checked, so -1 is not used.
• There is no femur or pelvis texture among the sources, so the ~5% of FPs on bone may persist.
• A label-to-image generator later in the pipeline (SynthSeg, PALETTE synthesis) overwrites the fill.

🤖 Generated with Claude Code

[NathanMolinier]

Without the transform	With the transform