Repository navigation
BMD: decode every section from the assembly kit, open battle-map .bin BMDs - #391
Open
robert-d-schultz wants to merge 23 commits into
Open
robert-d-schultz wants to merge 23 commits into
robert-d-schultz wants to merge 23 commits into
Conversation
Prop version 30 adds blend_edges, decal_fade_start, decal_angle_threshold, decal_mirror_x, decal_mirror_z, decal_layer and decal_tiling_alpha after uses_terrain_vertex_offset; particle emitter version 11 adds no_culling. Without them the parser fell 19 bytes behind per prop and ran off the end of the file. Field order checked against Terry's .bmd.xml for test files with each value set. Versions 26-29 have not been seen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The byte read as VisibleInShroudOnly is Terry's visible_without_shroud: a test compile turned visible_in_shroud_only="true" in the layer into visible_without_shroud='false', and vice versa, for every emitter. Rename it, invert it on Terry export as props already do, and fix the editor checkbox, which was showing ordinary emitters as shroud-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decals were always exported with parallax_scale and tiling of 0. Write the prop's real values, and for version 30 props also its fade_start, angle_threshold, mirror_x, mirror_z, layer and tiling_alpha_enabled, plus blend_edges as the mesh's blend_edges_with_terrain. Attribute names are the ones Terry writes in its own .layer files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This came from a global_props.bin script, where BMDs nest inside BMDs and each carries a culture mask. A prefab .bmd has nothing like that: the mask on a prefab instance is campaign_type_mask, and test compiles showed Terry stores neither its use_culture_mask toggle nor a prop's culture_mask in the .bmd. Each entity now keeps only its own mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fields that were read correctly but had placeholder names: light probe SomeZero is is_cylinder, river node Something1/2 are width and flow_speed, playable area Flag1-4 are valid_north/south/east/west, the poly mesh Booleans/MoreBooleans bytes are its snow/destruction region flags and visible_without_shroud, and a prefab instance's culture mask is campaign_type_mask. The editor gains checkboxes for the cylinder and without-shroud flags. New poly meshes now default to visible everywhere, as Terry's own do, instead of all flags off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… export" This reverts commit b279d93. Its reasoning was wrong: in global_props.bin, where BMDs nest inside BMDs, the u64 Terry's .bmd.xml calls campaign_type_mask is where a reference carries its child's culture, as prefab_types bits, while the objects inside get a placeholder mask of 1 (this is how AtlasWH3 reproduces BOB's output byte for byte). Terry-compiled prefabs leave it 0, so for them the inheritance changes nothing. Adjusted for the later rename of BmdInfo.CultureMask to CampaignTypeMask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Culture bit i is the prefab_types_tables row with value i + 1 (NONE, value 0, is bit 63). Checked against vanilla WH3's db/prefab_types_tables/data__ (version 3, 40 rows) through RPFM: every culture already on the table was on its bit, but Undead Legions (bit 35) was missing, and CultureMask decoded Beastmen from bit 1 (BUILDING_BASE) instead of bit 6. Formatting a parsed CultureMask now uses its raw bytes, as props and VFX already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Attribute names from Terry's own layers (AtlasWH3's scan of 1.1 million entities, and the test layers compiled for this branch): - props: cast_shadow and render_into_skydome_fog - VFX: the real autoplay, and no_culling - light probes: ECDoubleCylinder instead of ECDoubleSphere for a cylinder - terrain holes, point lights and spot lights: tactical-view visibility from the record's flags instead of always false - poly meshes: the snow/destruction region flags and visible_in_shroud_only - composite scenes: script_id, autoplay, hold_first_frame, hold_last_frame and tactical-view visibility - sounds: culture from the first mask (campaign_type_mask), where AtlasWH3's BOB-parity global_props_sound.bin puts it, not the pdlc mask Each is written only for record versions that store it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checked by parsing all 6,008 .bmd files in WH3's vanilla packs (through RPFM's server) and writing each back: - A parse must now end exactly at the end of the file, and lists no longer stop quietly when the data runs out. Before, a record read a few bytes short shifted everything after it and could still look like a success. - Prop version 29 has every version 30 field except decal_tiling_alpha, which v30 added (1,273 v29 props in 65 campaign prefabs, which failed). - Building properties version 12 adds a float and 13 a byte after the flags the parser knew (2,569 battlefield buildings in 122 prefabs, which failed; the float's meaning is unknown). - A prefab instance's property overrides are that same building-properties block, one per overridden building, as Terry's <override> entries show. They were skipped with a v4-era size, at most one, and could not be saved; they are now read, kept and written. Only two old vanilla files have any. All 6,008 now parse to the end of the file, and the 5,848 the writer supports come back byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
foobar.bmd, a test prefab with seven property overrides, all of which now parse and round-trip, has Terry's .bmd.xml name the building-properties fields after include_in_fog: blend_edges (version 12) and receive_decals (13). Prop versions 26-28 are still unseen, so the fields v29 has are spread over them in file order, grouped as they're related: blend_edges in 26, the decal fade and angle floats in 27, the mirror flags in 28, the layer in 29. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Culture is a property of a whole BMD: Terry's per-entity culture_mask only picks the bucket an entity is compiled into for global_props.bin, and a prefab .bmd is a single bucket. So an entity's own masks say nothing about its culture, and the export now gives every entity the mask of the reference its BMD was reached through (empty for the root BMD), instead of mixing in props', VFX' and sounds' own masks. Also matches the decal export to the guessed prop versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
empireutility.modder.x64.dll has one versioned serialiser per BATTLE_MAP_DEFINITION record that names every field and gates it on the record version (found and read with AtlasWH3's research/bob_re scripts). Where the parser disagreed: - the list gates: building projectile emitters from FASTBIN 15 (not 11), catchment areas from 23 (not with deployments at 22); the guessed 18-20 gates were right - props: decal_angle_threshold from 28 (the guess said 27); the three decal flags after visible_in_shroud from 6, not 5; version 9's extra byte is decal_apply_to_rigids, and versions 21-22's are visible_in_tactical_view and _only - building properties: building_id from 2 and the flags always (the other way round), a clamp_to_surface byte in 3-7, and most later flags one version later than assumed; legacy inline properties drop toggleable before 2 and lite before 3; battlefield buildings have an i16 parent id up to 10 and a uid from 10 - prefab instances: what was skipped as filler in versions 4-7 is the campaign_type_mask (4 bytes before 7), campaign_region_key from 4 and clamp_to_surface in 5-7, so those versions can now be written - VFX pdlc_mask is 8 bytes from 7, not 6; light probe height_mode from 2; spot volumetric from 2, height_mode from 3, 8-byte pdlc from 7; sound river nodes from 2, clamp_to_surface 3, height_mode 4, campaign mask 5, a 4-byte pdlc mask in 6; composite scene height_mode from 2, a 4-byte pdlc mask before 7, autoplay always, visible_in_shroud 5, no_culling 6, parent_script_id 9 All 6,008 vanilla .bmd files still parse to the end; 5,853 now round-trip (five more, old prefab-instance versions the writer used to refuse). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prop serialiser in the WH3 assembly kit names the bytes the parser was skipping as "mystery bytes": clamp_to_surface, decal, logic_decal, the v1-2 decal_parallax/decal_scale/decal_tiled, allow_in_outfield, is_fauna, the snow/destruction visibility bools, the four seasons, animated (v2+) and the v3 decal_parallax_scale/decal_tiling/decal_override_gbuffer_normal. The sizes match the old skip counts (30, 31 and 23 bytes). With the fields stored, the writer no longer refuses these props. Vanilla v3 props store junk such as 0x8D in decal_override_gbuffer_normal, so the raw byte is kept until the bool is edited, and those files round-trip exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every section that threw NotImplementedException, or could only be written empty, now has a reader and writer that follow the WH3 kit's serialisers (empireutility.modder.x64.dll): far battlefield buildings, EF lines, terrain, lite-building and water outlines, BMD outlines, camera zones, civilian deployments and shelters, AI hint separators and directed points, terrain stencil blend triangles, catchment areas, toggleable building slots, terrain decals, and tree and grass list references. Zones-template lines are read instead of skipped. Capture locations are rewritten: the old reader's layout (Something1-10) was wrong. Their records, like toggleable slots, have no version of their own and gate on the list version, so those models drop Version. The AI hint polyline list reads district only from version 2, as the kit does. The pre-FASTBIN-22 deployment manager is now refused when non-empty, instead of having its count ignored. Its item layout can't be read reliably from the kit. No vanilla file has data in any of these sections except one zones template with no lines, so they rest on the kit alone. Round-trip tests in FileTypesTests fill every one at old and new versions. Vanilla corpus: all 6,008 files parse to EOF, and 5,863 round-trip byte for byte. The remaining 145 are FASTBIN < 11, which the writer still refuses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vanilla battle maps keep their BMDs in terrain/tiles/battle as bmd_data.bin, catchment_NN_layer_bmd_data.bin and similar, and these have data in the sections the .bmd prefabs leave empty. Of 15,872 such files, 1,215 failed to parse. Following each body list's element reader in the kit fixed them: - A capture location set is a vector of CAPTURE_LOCATION_LISTs, each its own vector of locations. One nesting level was missing. - BMD_CATCHMENT_AREA's record is 1fef10 (name, float aabb, v6 battle_type and defending_faction_restriction, v5 valid_location_flags), not the similar-looking 1fdac0. - Terrain holes have height_mode only from version 2. The gated reader existed but wasn't the one called. - The pre-FASTBIN-22 BATTLE_DEPLOYMENT_AREA_MANAGER is decoded from four vanilla files that use it: items keyed by two ints, each holding collections of ALLIANCE_0/ALLIANCE_1 zone vectors. Parse errors now name the section and item that failed. All 15,872 battle BMDs now parse to EOF and round-trip byte for byte. The 547 battle_locations_map.bin and tile_list.bin files are FASTBIN0 but not BMDs. The .bmd corpus is unchanged: 5,863 identical, 145 FASTBIN < 11 refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Battle maps keep their BMDs as .bin files (bmd_data.bin, catchment_NN_layer_bmd_data.bin, *_procedural_bmd_data.bin, bmd_nogo_data.bin, bmd_objects_sound*.bin, catchment_catchment_N_bmd.bin). .bin is shared with other formats, including other FASTBIN0 files such as battle_locations_map.bin and tile_list.bin, so BmdFileNames only accepts a .bin whose name contains one of four BMD name parts. The editor registers .bin with those parts as folder rules, plus ".bmd" so prefabs still match, because the rules apply to every extension the editor lists. The Terry export command uses the same check. Checked against every vanilla path through the real EditorDatabase: all 6,008 .bmd and 15,871 BMD .bin files are offered the BMD editor, no other file is, and no path matches more than one editor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New props, VFX and composite scenes from the Add menu were shroud-only, and VFX and scenes had autoplay off; props also had every region flag off. Version 1 VFX store autoplay, scale and enable_in_outfield where later versions have the flags block (per the kit's serialiser); they were read as two throwaway integers and written back as constants. The Terry export forced composite scene autoplay on below version 3, though the parser reads it for every version. Also drops a dead rethrow and a dead length check in the parser. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Terry export writes a campaign prefab project, so it is no longer offered for battle-map .bin BMDs, from the pack tree or the editor. The BMD report now includes the .bin files BmdFileNames recognises. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Removes comments that restate the code and shortens the long doc blocks. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Add menu's VFX, lights, poly meshes and terrain holes had all four season flags off; every flagged element in the vanilla files checked has them on, as new props already did. Adds a Delete button (Del in the component list) for the editable types. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The kit's serialisers name the 8 bytes after a prop's and a VFX' height mode, and the second mask on a sound, pdlc_mask: 1 plus the bits of the DLCs the entity's ECDLCMask names (all bits in a Terry-compiled prefab). Point and spot lights and composite scenes already read it as PdlcMask. These are now a ulong PdlcMask too, instead of a byte[8] (props, VFX) or a CultureMask decoded into culture bits (sounds). Culture is a property of a whole BMD, carried by a reference's campaign_type_mask. The bytes on disk are unchanged: the 5,459 vanilla prefab .bmd files round-trip exactly as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The BMD editor docs said new elements were "not restricted to any culture", which described pdlc_mask as a culture mask. They now explain the DLC mask: Terry sets it from an entity's DLC mask, and vanilla doesn't use it (every vanilla prefab entity has every bit, BOB's global_props.bin writes 1). Culture belongs to the whole BMD. New point lights, spot lights and composite scenes got a mask of 0. They now get every DLC, like new props, VFX and sounds, and like Terry's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The BMD parser and writer now follow the WH3 assembly kit's own serialisers (
empireutility.modder.x64.dll) instead of guessed layouts, so every section is decoded, including the ones that used to throwNotImplementedException. That also covers the fields a recent WH3 update added to props (up to version 30) and VFX.Checked against vanilla WH3: all 15,872 battle-map BMDs and 5,863 of the 6,008
.bmdfiles parse and write back byte for byte. The other 145 are FASTBIN < 11, which the writer still refuses. That check was run before the last three commits; since then only the unit tests have been re-run.Changes outside the BMD editor
Nothing in
Shared.Core,Shared.UiorGameWorldis changed. The parts that reach beyond the editor:.bin..binis shared with other formats, so the registration adds folder rules (bmd_data,bmd_nogo_data,bmd_objects_sound,_bmd.bin, plus.bmd), using the existingValidForFoldersContaining.EditorDatabaseitself is untouched. Checked against every vanilla WH3 path: it matches the battle-map BMDs and nothing else, and does not overlap the Campaign Animation Set editor's.binrule (animations\campaign\database).Shared/GameFiles/Bmdmodel changes.BmdFileproperties were renamed, retyped or removed to match the kit (e.g.CaptureLocationsis nowCaptureLocationLists, building properties moved to aBuildingPropertiesclass, theSomething1-style placeholders are gone). The only users are the BMD editor and the BMD report, both updated.BmdParserfails loudly. A parse that errors, or that doesn't end exactly at the end of the file, throwsInvalidDataExceptionnaming the section and item.BmdFileNamesinShared/GameFiles/Bmddecides which file names are BMDs.Testing/FileTypesTests/FileTypes/Bmd/BmdRoundTripTests.cs. That project isn't in the solution and doesn't currently build (TestCommon), so they don't run in CI; I ran them through a scratch project. I left the project as it is.BMD editor, export and report
.bin(view and save)..bmdprefabs only..binBMDs.docs/Documentation/BmdEditor.html.🤖 Generated with Claude Code