Skip to content

Startup: build the audio cache in the background and cut its build time from ~3 min to ~25 s - #397

Open
robert-d-schultz wants to merge 8 commits into
donkeyProgramming:masterfrom
robert-d-schultz:startup-changes
Open

robert-d-schultz wants to merge 8 commits into
donkeyProgramming:masterfrom
robert-d-schultz:startup-changes

Conversation

@robert-d-schultz

Copy link
Copy Markdown
Collaborator

Problem

After a game update, Asset Editor can take minutes to open. A user reported 5 minutes. App.OnStartup builds the game files audio cache on the UI thread before showing the main window (added with the audio data cache in #386), so for every user, whether or not they use the audio tools, the screen stays blank until it finishes. On my WH3 install the build took about 3 minutes:

  • DAT data: ~98 s. SearchFiles matches extensions as substrings, so the .dat lookup also returns the 256 terrain hf_height_map.data files (31 MB compressed). Each one was decompressed and parsed as a sound dat, failed, and the failure was thrown away. DatLoader then looked up every file's path again through GetFullPath, which on the cached game container is a full table scan (~0.2 s × 270).
  • Bank indexing: ~76 s. WH3 has 3.57M HIRC rows, inserted while SQLite keeps three secondary indexes up to date, one of them on the effectively random HircId.

Startup also logged 9 Errors with stack traces ("File empty") from the skeleton/animation lookup. They come from empty .anim files that aren't on its hardcoded "known broken" list.

Changes (one commit each)

  1. Audio cache only reads real .dat/.bnk/.wwiseids files. It re-checks the extension after SearchFiles and filters on the paths it already has instead of calling GetFullPath. Dat files that fail to parse are now logged instead of dropped. This also stops terrain files from changing the audio fingerprint, and stops a terrain-only mod from counting as having audio files.
  2. HIRC/DIDX indexes are built after the bulk insert. They're dropped and recreated from their original SQL inside the same transaction. Replaying the real WH3 rows: 59.5 s with indexes in place, 10.7 s when created after.
  3. The game files audio cache is built in the background. AudioRepository already builds a missing cache on first use; loading or building now goes through a lock, so an audio tool opened while the prebuild runs waits for it instead of recreating the same file. The prebuild also used to resolve the scoped IAudioRepository from the root provider, which throws when scope validation is on (debugger attached), so it never ran there. It now creates its own scope.
  4. Stale game pack caches are deleted. Each game update writes a new ~300 MB WH3 cache under a new fingerprint and left the old one behind (I had five). It now cleans up like the audio cache does, removing only files with the same name and a different 64-hex fingerprint.
  5. Pack cache build dialog reworded. "Failed to load from cache - Generating new cache" appears on every first run and after every game update, when nothing has failed.
  6. Empty .anim files are skipped generically. Six of the seven "known broken" anims are 0-byte entries in WH3's anim2.pack, and WH3 has 15 such files. All 15 are now skipped with one Information line. The list keeps only the pre-WH3 roman_aqueduct entry, which isn't in WH3's packs, so I couldn't check it.

Not changed: SearchFiles itself

The root cause of (1) is that SearchFiles matches extensions with Contains. I didn't change that, because SearchFiles_ExtensionFilter_MatchesFilenameSubstrings_ForLegacyWemVariants asserts that behaviour (.wem matching .wem_temp / .wem.{sdf}). Vanilla WH3 has no files like that, and no code searches for .wem. The pack browser has the same issue: filtering for .dat lists the heightmaps, and filtering for .anim lists .animpack files. Is the substring match still needed? If not, I can switch the containers to a real extension match in a follow-up.

Verification

  • Live run on WH3 after a forced audio cache rebuild: the main window was up 2 s after launch. The audio cache built in the background in 24 s (DAT data 4 s, banks 20 s), down from ~3 min of blank screen.
  • New tests: .data files don't affect the fingerprint; indexes exist after Save; concurrent LoadOrBuildCache callers all get a working cache (fails 6/6 without the lock); stale pack caches are removed and nothing else is. Each test fails without its fix.
  • Test.Audio, Shared.CoreTest and GameWorld.CoreTest all pass.

The new logging shows that 5 sound dats in vanilla WH3 already failed to parse before this change (audio_rules_battle_order_vo, audio_rules_campaign_conversation_vo, audio_rules_campaign_order_vo, english(uk)\advisor\wwise.dat, tools_data.dat). They're probably other formats that belong on the skip list, but I haven't checked, so I left that list unchanged.

🤖 Generated with Claude Code

robert-d-schultz and others added 8 commits November 17, 2026 19:25
SearchFiles matches extensions as substrings, so the ".dat" lookups in the
audio cache also returned the 256 terrain hf_height_map.data files in WH3.
Each was decompressed and parsed as a sound dat, failed, and the failure was
discarded. They also went into the audio fingerprint, and a terrain-only mod
counted as having audio files.

DatLoader also re-resolved every file's path through GetFullPath, which on
the cached game container is a full table scan per file (~0.2s x 270).
It now filters on the paths SearchFiles already returned, and logs dat files
that fail to parse instead of dropping the list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inserting WH3's 3.57M HIRC rows while SQLite keeps three secondary
indexes up to date (one on the effectively random HircId) is the bulk of
the bank-indexing phase. Replaying the real rows: 59.5s with indexes in
place, 10.7s when they are created after the insert.

Save now drops the Hircs/Didx indexes EF created, inserts, then recreates
them from their original SQL inside the same transaction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
App.OnStartup built the game files audio cache on the UI thread before the
main window was shown (added with the audio data cache, PR donkeyProgramming#386). After a
game update that is minutes of a blank screen, for every user, whether or
not they use the audio tools.

It now runs on a background task. AudioRepository already builds a missing
cache on first use; cache load/build now goes through a lock, so an audio
tool opened while the prebuild is still running waits for it instead of
recreating the same file (which the new concurrency test shows collides).

The prebuild also resolved the scoped IAudioRepository from the root
provider, which throws when scope validation is on (debugger attached), so
it never ran there. It now creates its own scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cache file name contains a fingerprint of the game's packs, so every
game update writes a new ~300MB Warhammer III cache and leaves the old one
behind. The audio cache already cleans up after itself; the pack cache
now does the same, removing only files named like this cache with a
different 64-hex fingerprint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Failed to load from cache - Generating new cache" is shown on every first
run and after every game update, where nothing has failed. Say what is
happening and why instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six of the seven "known broken" animations are 0 byte entries in WH3's
anim2.pack. WH3 has 15 such files, so the other nine each logged an Error
with a stack trace ("File empty") on every startup, while the listed ones
logged a Warning each.

Empty files are now skipped generically and reported in one Information
line per container. The list keeps only the pre-WH3 roman_aqueduct entry,
which is not in WH3's packs and so could not be checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The startup audio cache build now logs from a background thread while
the UI thread logs too; the history queue was unsynchronised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DerpyNewb added a commit to DerpyNewb/TheAssetEditorDN that referenced this pull request Nov 19, 2026
… audio cache in the background

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	AssetEditor/App.xaml.cs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant