Skip to content

BMD: decode every section from the assembly kit, open battle-map .bin BMDs - #391

Open
robert-d-schultz wants to merge 23 commits into
donkeyProgramming:masterfrom
robert-d-schultz:bmd-prop-v30
Open

robert-d-schultz wants to merge 23 commits into
donkeyProgramming:masterfrom
robert-d-schultz:bmd-prop-v30

Conversation

@robert-d-schultz

@robert-d-schultz robert-d-schultz commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The BMD parser and writer now follow the WH3 assembly kit's own serialisers (empireutility.modder.x64.dll) instead of guessed layouts, so every section is decoded, including the ones that used to throw NotImplementedException. That also covers the fields a recent WH3 update added to props (up to version 30) and VFX.

Checked against vanilla WH3: all 15,872 battle-map BMDs and 5,863 of the 6,008 .bmd files parse and write back byte for byte. The other 145 are FASTBIN < 11, which the writer still refuses. That check was run before the last three commits; since then only the unit tests have been re-run.

Changes outside the BMD editor

Nothing in Shared.Core, Shared.Ui or GameWorld is changed. The parts that reach beyond the editor:

  • The BMD editor now registers .bin. .bin is shared with other formats, so the registration adds folder rules (bmd_data, bmd_nogo_data, bmd_objects_sound, _bmd.bin, plus .bmd), using the existing ValidForFoldersContaining. EditorDatabase itself is untouched. Checked against every vanilla WH3 path: it matches the battle-map BMDs and nothing else, and does not overlap the Campaign Animation Set editor's .bin rule (animations\campaign\database).
  • Shared/GameFiles/Bmd model changes. BmdFile properties were renamed, retyped or removed to match the kit (e.g. CaptureLocations is now CaptureLocationLists, building properties moved to a BuildingProperties class, the Something1-style placeholders are gone). The only users are the BMD editor and the BMD report, both updated.
  • BmdParser fails loudly. A parse that errors, or that doesn't end exactly at the end of the file, throws InvalidDataException naming the section and item.
  • New BmdFileNames in Shared/GameFiles/Bmd decides which file names are BMDs.
  • Tests are in Testing/FileTypesTests/FileTypes/Bmd/BmdRoundTripTests.cs. That project isn't in the solution and doesn't currently build (TestCommon), so they don't run in CI; I ran them through a scratch project. I left the project as it is.

BMD editor, export and report

  • Opens battle-map BMDs stored as .bin (view and save).
  • Terry export: writes the fields it used to drop or hardcode (decal settings, blend edges, shroud and tactical visibility, autoplay, cylinder probes, scene script ids), and takes an entity's culture from the BMD it sits in. Offered for .bmd prefabs only.
  • Elements of the editable types can be deleted (toolbar button, or Del in the component list).
  • Fixes: new elements from the Add menu defaulted to shroud-only and, apart from props, to no season; version 1 VFX were misread.
  • The BMD report also covers the .bin BMDs.
  • Trimmed comments across the BMD code.
  • New user documentation page: docs/Documentation/BmdEditor.html.

🤖 Generated with Claude Code

robert-d-schultz and others added 23 commits November 10, 2026 14:30
Prop version 30 adds blend_edges, decal_fade_start, decal_angle_threshold,
decal_mirror_x, decal_mirror_z, decal_layer and decal_tiling_alpha after
uses_terrain_vertex_offset; particle emitter version 11 adds no_culling.
Without them the parser fell 19 bytes behind per prop and ran off the end of
the file. Field order checked against Terry's .bmd.xml for test files with
each value set. Versions 26-29 have not been seen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The byte read as VisibleInShroudOnly is Terry's visible_without_shroud: a
test compile turned visible_in_shroud_only="true" in the layer into
visible_without_shroud='false', and vice versa, for every emitter. Rename
it, invert it on Terry export as props already do, and fix the editor
checkbox, which was showing ordinary emitters as shroud-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decals were always exported with parallax_scale and tiling of 0. Write the
prop's real values, and for version 30 props also its fade_start,
angle_threshold, mirror_x, mirror_z, layer and tiling_alpha_enabled, plus
blend_edges as the mesh's blend_edges_with_terrain. Attribute names are the
ones Terry writes in its own .layer files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This came from a global_props.bin script, where BMDs nest inside BMDs and
each carries a culture mask. A prefab .bmd has nothing like that: the mask on
a prefab instance is campaign_type_mask, and test compiles showed Terry
stores neither its use_culture_mask toggle nor a prop's culture_mask in the
.bmd. Each entity now keeps only its own mask.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fields that were read correctly but had placeholder names:
light probe SomeZero is is_cylinder, river node Something1/2 are width and
flow_speed, playable area Flag1-4 are valid_north/south/east/west, the poly
mesh Booleans/MoreBooleans bytes are its snow/destruction region flags and
visible_without_shroud, and a prefab instance's culture mask is
campaign_type_mask. The editor gains checkboxes for the cylinder and
without-shroud flags. New poly meshes now default to visible everywhere, as
Terry's own do, instead of all flags off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… export"

This reverts commit b279d93. Its reasoning was wrong: in global_props.bin,
where BMDs nest inside BMDs, the u64 Terry's .bmd.xml calls
campaign_type_mask is where a reference carries its child's culture, as
prefab_types bits, while the objects inside get a placeholder mask of 1
(this is how AtlasWH3 reproduces BOB's output byte for byte). Terry-compiled
prefabs leave it 0, so for them the inheritance changes nothing.

Adjusted for the later rename of BmdInfo.CultureMask to CampaignTypeMask.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Culture bit i is the prefab_types_tables row with value i + 1 (NONE, value
0, is bit 63). Checked against vanilla WH3's db/prefab_types_tables/data__
(version 3, 40 rows) through RPFM: every culture already on the table was on
its bit, but Undead Legions (bit 35) was missing, and CultureMask decoded
Beastmen from bit 1 (BUILDING_BASE) instead of bit 6. Formatting a parsed
CultureMask now uses its raw bytes, as props and VFX already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Attribute names from Terry's own layers (AtlasWH3's scan of 1.1 million
entities, and the test layers compiled for this branch):
- props: cast_shadow and render_into_skydome_fog
- VFX: the real autoplay, and no_culling
- light probes: ECDoubleCylinder instead of ECDoubleSphere for a cylinder
- terrain holes, point lights and spot lights: tactical-view visibility from
  the record's flags instead of always false
- poly meshes: the snow/destruction region flags and visible_in_shroud_only
- composite scenes: script_id, autoplay, hold_first_frame, hold_last_frame
  and tactical-view visibility
- sounds: culture from the first mask (campaign_type_mask), where
  AtlasWH3's BOB-parity global_props_sound.bin puts it, not the pdlc mask

Each is written only for record versions that store it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checked by parsing all 6,008 .bmd files in WH3's vanilla packs (through
RPFM's server) and writing each back:
- A parse must now end exactly at the end of the file, and lists no longer
  stop quietly when the data runs out. Before, a record read a few bytes
  short shifted everything after it and could still look like a success.
- Prop version 29 has every version 30 field except decal_tiling_alpha,
  which v30 added (1,273 v29 props in 65 campaign prefabs, which failed).
- Building properties version 12 adds a float and 13 a byte after the
  flags the parser knew (2,569 battlefield buildings in 122 prefabs, which
  failed; the float's meaning is unknown).
- A prefab instance's property overrides are that same building-properties
  block, one per overridden building, as Terry's <override> entries show.
  They were skipped with a v4-era size, at most one, and could not be
  saved; they are now read, kept and written. Only two old vanilla files
  have any.

All 6,008 now parse to the end of the file, and the 5,848 the writer
supports come back byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
foobar.bmd, a test prefab with seven property overrides, all of which now
parse and round-trip, has Terry's .bmd.xml name the building-properties
fields after include_in_fog: blend_edges (version 12) and receive_decals
(13). Prop versions 26-28 are still unseen, so the fields v29 has are
spread over them in file order, grouped as they're related: blend_edges
in 26, the decal fade and angle floats in 27, the mirror flags in 28, the
layer in 29.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Culture is a property of a whole BMD: Terry's per-entity culture_mask only
picks the bucket an entity is compiled into for global_props.bin, and a
prefab .bmd is a single bucket. So an entity's own masks say nothing about
its culture, and the export now gives every entity the mask of the
reference its BMD was reached through (empty for the root BMD), instead of
mixing in props', VFX' and sounds' own masks.

Also matches the decal export to the guessed prop versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
empireutility.modder.x64.dll has one versioned serialiser per
BATTLE_MAP_DEFINITION record that names every field and gates it on the
record version (found and read with AtlasWH3's research/bob_re scripts).
Where the parser disagreed:
- the list gates: building projectile emitters from FASTBIN 15 (not 11),
  catchment areas from 23 (not with deployments at 22); the guessed 18-20
  gates were right
- props: decal_angle_threshold from 28 (the guess said 27); the three decal
  flags after visible_in_shroud from 6, not 5; version 9's extra byte is
  decal_apply_to_rigids, and versions 21-22's are visible_in_tactical_view
  and _only
- building properties: building_id from 2 and the flags always (the other
  way round), a clamp_to_surface byte in 3-7, and most later flags one
  version later than assumed; legacy inline properties drop toggleable
  before 2 and lite before 3; battlefield buildings have an i16 parent id
  up to 10 and a uid from 10
- prefab instances: what was skipped as filler in versions 4-7 is the
  campaign_type_mask (4 bytes before 7), campaign_region_key from 4 and
  clamp_to_surface in 5-7, so those versions can now be written
- VFX pdlc_mask is 8 bytes from 7, not 6; light probe height_mode from 2;
  spot volumetric from 2, height_mode from 3, 8-byte pdlc from 7; sound
  river nodes from 2, clamp_to_surface 3, height_mode 4, campaign mask 5,
  a 4-byte pdlc mask in 6; composite scene height_mode from 2, a 4-byte
  pdlc mask before 7, autoplay always, visible_in_shroud 5, no_culling 6,
  parent_script_id 9

All 6,008 vanilla .bmd files still parse to the end; 5,853 now round-trip
(five more, old prefab-instance versions the writer used to refuse).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prop serialiser in the WH3 assembly kit names the bytes the parser was
skipping as "mystery bytes": clamp_to_surface, decal, logic_decal, the v1-2
decal_parallax/decal_scale/decal_tiled, allow_in_outfield, is_fauna, the
snow/destruction visibility bools, the four seasons, animated (v2+) and the v3
decal_parallax_scale/decal_tiling/decal_override_gbuffer_normal. The sizes
match the old skip counts (30, 31 and 23 bytes).

With the fields stored, the writer no longer refuses these props. Vanilla v3
props store junk such as 0x8D in decal_override_gbuffer_normal, so the raw
byte is kept until the bool is edited, and those files round-trip exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every section that threw NotImplementedException, or could only be written
empty, now has a reader and writer that follow the WH3 kit's serialisers
(empireutility.modder.x64.dll): far battlefield buildings, EF lines, terrain,
lite-building and water outlines, BMD outlines, camera zones, civilian
deployments and shelters, AI hint separators and directed points, terrain
stencil blend triangles, catchment areas, toggleable building slots, terrain
decals, and tree and grass list references. Zones-template lines are read
instead of skipped.

Capture locations are rewritten: the old reader's layout (Something1-10) was
wrong. Their records, like toggleable slots, have no version of their own and
gate on the list version, so those models drop Version. The AI hint polyline
list reads district only from version 2, as the kit does.

The pre-FASTBIN-22 deployment manager is now refused when non-empty, instead
of having its count ignored. Its item layout can't be read reliably from the
kit.

No vanilla file has data in any of these sections except one zones template
with no lines, so they rest on the kit alone. Round-trip tests in
FileTypesTests fill every one at old and new versions. Vanilla corpus: all
6,008 files parse to EOF, and 5,863 round-trip byte for byte. The remaining
145 are FASTBIN < 11, which the writer still refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vanilla battle maps keep their BMDs in terrain/tiles/battle as
bmd_data.bin, catchment_NN_layer_bmd_data.bin and similar, and these
have data in the sections the .bmd prefabs leave empty. Of 15,872 such
files, 1,215 failed to parse. Following each body list's element reader
in the kit fixed them:

- A capture location set is a vector of CAPTURE_LOCATION_LISTs, each
  its own vector of locations. One nesting level was missing.
- BMD_CATCHMENT_AREA's record is 1fef10 (name, float aabb, v6
  battle_type and defending_faction_restriction, v5
  valid_location_flags), not the similar-looking 1fdac0.
- Terrain holes have height_mode only from version 2. The gated reader
  existed but wasn't the one called.
- The pre-FASTBIN-22 BATTLE_DEPLOYMENT_AREA_MANAGER is decoded from
  four vanilla files that use it: items keyed by two ints, each holding
  collections of ALLIANCE_0/ALLIANCE_1 zone vectors.

Parse errors now name the section and item that failed.

All 15,872 battle BMDs now parse to EOF and round-trip byte for byte.
The 547 battle_locations_map.bin and tile_list.bin files are FASTBIN0
but not BMDs. The .bmd corpus is unchanged: 5,863 identical, 145
FASTBIN < 11 refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Battle maps keep their BMDs as .bin files (bmd_data.bin,
catchment_NN_layer_bmd_data.bin, *_procedural_bmd_data.bin,
bmd_nogo_data.bin, bmd_objects_sound*.bin, catchment_catchment_N_bmd.bin).
.bin is shared with other formats, including other FASTBIN0 files such
as battle_locations_map.bin and tile_list.bin, so BmdFileNames only
accepts a .bin whose name contains one of four BMD name parts.

The editor registers .bin with those parts as folder rules, plus ".bmd"
so prefabs still match, because the rules apply to every extension the
editor lists. The Terry export command uses the same check.

Checked against every vanilla path through the real EditorDatabase: all
6,008 .bmd and 15,871 BMD .bin files are offered the BMD editor, no other
file is, and no path matches more than one editor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New props, VFX and composite scenes from the Add menu were shroud-only,
and VFX and scenes had autoplay off; props also had every region flag off.
Version 1 VFX store autoplay, scale and enable_in_outfield where later
versions have the flags block (per the kit's serialiser); they were read
as two throwaway integers and written back as constants.
The Terry export forced composite scene autoplay on below version 3,
though the parser reads it for every version.
Also drops a dead rethrow and a dead length check in the parser.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Terry export writes a campaign prefab project, so it is no longer
offered for battle-map .bin BMDs, from the pack tree or the editor.
The BMD report now includes the .bin files BmdFileNames recognises.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Removes comments that restate the code and shortens the long doc blocks.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Add menu's VFX, lights, poly meshes and terrain holes had all four
season flags off; every flagged element in the vanilla files checked has
them on, as new props already did.
Adds a Delete button (Del in the component list) for the editable types.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The kit's serialisers name the 8 bytes after a prop's and a VFX' height
mode, and the second mask on a sound, pdlc_mask: 1 plus the bits of the
DLCs the entity's ECDLCMask names (all bits in a Terry-compiled prefab).
Point and spot lights and composite scenes already read it as PdlcMask.
These are now a ulong PdlcMask too, instead of a byte[8] (props, VFX) or
a CultureMask decoded into culture bits (sounds). Culture is a property of
a whole BMD, carried by a reference's campaign_type_mask. The bytes on
disk are unchanged: the 5,459 vanilla prefab .bmd files round-trip
exactly as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The BMD editor docs said new elements were "not restricted to any
culture", which described pdlc_mask as a culture mask. They now explain
the DLC mask: Terry sets it from an entity's DLC mask, and vanilla doesn't
use it (every vanilla prefab entity has every bit, BOB's global_props.bin
writes 1). Culture belongs to the whole BMD.

New point lights, spot lights and composite scenes got a mask of 0. They
now get every DLC, like new props, VFX and sounds, and like Terry's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant