Repository navigation
Inspect media files in parallel with one ffprobe call each - #19
Merged
Merged
Conversation
Inspection was sequential and started three ffprobe processes for each video file without an ISO header: the format and stream headers, then the same video header twice, for the timecode and for the timing walk. Each start analyses the opening packets, about 0.3 s for Canon XF-AVC. One call now serves all three. The cached timing walk is guarded by the file's revision from before its header was read, so a file replaced meanwhile never caches another version's frame rate. The video header query did not request the format tags, where MXF keeps its timecode, so Canon and other MXF clips arrived without container timecode and inspection searched every audio track for LTC instead. The timecode is now read. Files are inspected on four bounded workers, like fingerprinting, and the outcomes keep input order. Progress counts are published in order, and a cancellation while files are in flight is reported as such. Inspecting eight Canon C70 clips and five other files went from 21.2 s to 0.56 s (2.1 s with one worker), with the timecode of all eight clips (#17).
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #17.
The issue is right that inspection is sequential while fingerprinting and
refinement run on bounded workers. Measuring first showed that most of the
time was work inspection did not need:
XF-AVC 4:2:2 10-bit video. Inspection started three per MXF or MTS file:
the format and stream headers, then the same video header twice, for the
timecode and for the timing walk.
timecode. Canon MXF clips arrived without container timecode (the C70 sample
lost
02:07:09;05), and inspection then decoded three seconds of everyaudio track to look for LTC, two more process starts per track.
Changes:
the timecode. The timing walk receives the frame rate instead of querying
it;
video_timing()keeps its own header query for other callers. Thecached walk is guarded by the file's revision from before its header was
read, so a file replaced during inspection never caches another version's
frame rate.
align_core::par_map, thesame limit as the fingerprint stage. Clips and warnings keep input order,
and the GUI already sorts discovered clips by path. Progress reports each
file as it starts and finishes; counting and publishing happen under one
lock, so the completed count never goes backwards. Cancellation is checked
before each file and again when the workers finish, so files failing after
a cancellation do not turn it into "No readable media".
Measurements on the portable backend (macOS, SSD) with eight copies of the
camera-original C70 clip, two Canon MP4 files, a Canon 6D MOV and two WAV
recordings (1.7 GB):
The full synchronization of the same set took 42.5 s on 0.3.2 and 19.9 s on
this branch (19.7 s together with the #16 fix), with identical matches. The
timecode of all eight C70 clips is now read; 0.3.2 read none.
On the first run, the timing walk reads each MXF or MTS file to the end, so
four workers mean four concurrent readers, as fingerprinting already has. The
result is cached per file revision. HDD and network storage were not measured,
and no setting was added.
Parallel inspection changes no outcome: on the 59-file decoder differential
from the #16 branch (every container and codec Align accepts, damaged and
unreadable files, camera-original Canon footage), each file is inspected,
synchronized, left unmatched or warned about exactly as in 0.3.2, with the
same offsets.
Tests, each failing on the code before it:
mxf_timecode_comes_from_the_format_tags(onmain); it skips only whenFFmpeg lacks the MXF muxer or MPEG-2 encoder, and fails on any other
fixture error.
cancelling_during_inspection_is_reported_as_cancelledinspection_progress_never_goes_backwards(failed 5 of 5 runs on theprevious revision)
Existing timing tests pass unchanged.
Validation:
cargo fmt --all -- --checkcargo clippy --locked --workspace --all-targets -- -D warningscargo test --locked --workspace: 376 passed, 1 ignored