Skip to content

Inspect media files in parallel with one ffprobe call each - #19

Merged
FedyaLight merged 1 commit into
mainfrom
perf/inspect-media
Nov 12, 2026
Merged

FedyaLight merged 1 commit into
mainfrom
perf/inspect-media

Conversation

@FedyaLight

Copy link
Copy Markdown
Owner

Addresses #17.

The issue is right that inspection is sequential while fingerprinting and
refinement run on bounded workers. Measuring first showed that most of the
time was work inspection did not need:

  • Every ffprobe start analyses the opening packets, about 0.3 s for Canon
    XF-AVC 4:2:2 10-bit video. Inspection started three per MXF or MTS file:
    the format and stream headers, then the same video header twice, for the
    timecode and for the timing walk.
  • The video header query did not request the format tags, where MXF keeps its
    timecode. Canon MXF clips arrived without container timecode (the C70 sample
    lost 02:07:09;05), and inspection then decoded three seconds of every
    audio track to look for LTC, two more process starts per track.

Changes:

  • One ffprobe call per file now serves the audio streams, the video header and
    the timecode. The timing walk receives the frame rate instead of querying
    it; video_timing() keeps its own header query for other callers. The
    cached walk is guarded by the file's revision from before its header was
    read, so a file replaced during inspection never caches another version's
    frame rate.
  • Files are inspected on four bounded workers with align_core::par_map, the
    same limit as the fingerprint stage. Clips and warnings keep input order,
    and the GUI already sorts discovered clips by path. Progress reports each
    file as it starts and finishes; counting and publishing happen under one
    lock, so the completed count never goes backwards. Cancellation is checked
    before each file and again when the workers finish, so files failing after
    a cancellation do not turn it into "No readable media".

Measurements on the portable backend (macOS, SSD) with eight copies of the
camera-original C70 clip, two Canon MP4 files, a Canon 6D MOV and two WAV
recordings (1.7 GB):

Inspection First run Repeat
0.3.2 23.9 s 21.2 s
This branch, 1 worker 4.5 s 2.1 s
This branch, 2 workers 2.4 s 1.1 s
This branch, 4 workers 1.2 s 0.56 s

The full synchronization of the same set took 42.5 s on 0.3.2 and 19.9 s on
this branch (19.7 s together with the #16 fix), with identical matches. The
timecode of all eight C70 clips is now read; 0.3.2 read none.

On the first run, the timing walk reads each MXF or MTS file to the end, so
four workers mean four concurrent readers, as fingerprinting already has. The
result is cached per file revision. HDD and network storage were not measured,
and no setting was added.

Parallel inspection changes no outcome: on the 59-file decoder differential
from the #16 branch (every container and codec Align accepts, damaged and
unreadable files, camera-original Canon footage), each file is inspected,
synchronized, left unmatched or warned about exactly as in 0.3.2, with the
same offsets.

Tests, each failing on the code before it:

  • mxf_timecode_comes_from_the_format_tags (on main); it skips only when
    FFmpeg lacks the MXF muxer or MPEG-2 encoder, and fails on any other
    fixture error.
  • cancelling_during_inspection_is_reported_as_cancelled
  • inspection_progress_never_goes_backwards (failed 5 of 5 runs on the
    previous revision)

Existing timing tests pass unchanged.

Validation:

  • cargo fmt --all -- --check
  • cargo clippy --locked --workspace --all-targets -- -D warnings
  • cargo test --locked --workspace: 376 passed, 1 ignored
  • Windows was not run locally.

Inspection was sequential and started three ffprobe processes for each
video file without an ISO header: the format and stream headers, then the
same video header twice, for the timecode and for the timing walk. Each
start analyses the opening packets, about 0.3 s for Canon XF-AVC. One call
now serves all three. The cached timing walk is guarded by the file's
revision from before its header was read, so a file replaced meanwhile
never caches another version's frame rate.

The video header query did not request the format tags, where MXF keeps
its timecode, so Canon and other MXF clips arrived without container
timecode and inspection searched every audio track for LTC instead. The
timecode is now read.

Files are inspected on four bounded workers, like fingerprinting, and the
outcomes keep input order. Progress counts are published in order, and a
cancellation while files are in flight is reported as such. Inspecting
eight Canon C70 clips and five other files went from 21.2 s to 0.56 s
(2.1 s with one worker), with the timecode of all eight clips (#17).
@FedyaLight
FedyaLight merged commit d99a722 into main Nov 12, 2026
14 checks passed
@FedyaLight
FedyaLight deleted the perf/inspect-media branch November 12, 2026 07:29
@FedyaLight FedyaLight mentioned this pull request Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant