Skip to content

oto: report when frames are played on browsers - #332

Merged
hajimehoshi merged 5 commits into
ebitengine:mainfrom
marrasen:js-played
Nov 14, 2026
Merged

hajimehoshi merged 5 commits into
ebitengine:mainfrom
marrasen:js-played

Conversation

@marrasen

@marrasen marrasen commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

This PR was written by Claude (Anthropic), an AI agent, at Marcus's request.

What issue is this addressing?

Updates #311

What type of issue is this addressing?

feature

What this PR does | solves

The web driver now reports when frames are played, with Mux.SetPlayed, so UnplayedSize counts the frames queued after the player's buffer and the audio hardware's delay.

  • AudioWorklet: the worklet processor reports which frame of the mix it outputs at which context time (currentTime), every 100 ms of context time. When it outputs silence for want of data, it clears the report, as the frames after the silence are output later than reported.
  • ScriptProcessorNode: each audioprocess event reports the frame its buffer starts with, at the event's playbackTime.
  • Context time to time.Time: AudioContext.getOutputTimestamp gives the context time of the frame the audio hardware plays now and its performance time, so a context time maps onto performance.now(), and from there onto time.Now().
  • Suspend clears the report. Where getOutputTimestamp is missing, or its performance time is still 0 before output starts, nothing is reported, and UnplayedSize returns the same as BufferedSize.

Testing

CI can't run audio in a browser, so I checked by hand in headless Chrome (--autoplay-policy=no-user-gesture-required) on Linux with PipeWire, playing a sine through a 48 kHz context and logging every 200 ms:

UnplayedSize − BufferedSize outputLatency
AudioWorklet, playing 76–158 ms 56–128 ms
ScriptProcessorNode, playing 127–214 ms 56–128 ms

The difference is the output latency plus what is queued in the worklet or the script processor (2048 frames per transfer by default). After Pause it falls to 0 at the rate of real time, in about the time it showed. After Suspend it is 0, and after Resume the reports come back.

I could not check Firefox: headless Firefox keeps the context suspended without a real user gesture. A browser without getOutputTimestamp keeps today's behavior.

🤖 Generated with Claude Code

The web driver now calls Mux.SetPlayed, so UnplayedSize counts the
frames queued after the player's buffer and the audio hardware's delay.

The worklet processor reports which frame of the mix it outputs at
which context time, every 100 ms, and clears the report when it
outputs silence for want of data. The script processor reports the
frame each buffer starts with at the buffer's playbackTime. The main
thread turns the context time into a time.Time with
AudioContext.getOutputTimestamp. Suspend clears the report.

Where getOutputTimestamp is missing, nothing is reported, and
UnplayedSize returns the same as BufferedSize, as before.

Updates ebitengine#311

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hajimehoshi

Copy link
Copy Markdown
Member

Review of 54c3dd8. I built and vetted it for GOOS=js GOARCH=wasm and ran the internal/mux tests, but did not run it in a browser, so the findings below come from reading the code.

The approach looks right to me. What I checked:

  • Frame indices line up with the mux. The worklet's buf_ is all the ReadFloat32s output joined in order, and silence output doesn't consume it, so frames_ counts mix frames exactly. durationToBufferSize always returns whole frames, so the script processor's frames += len(buf32)/channelCount doesn't drift from mixed.
  • Timing. In the worklet, currentTime is the context time of the block being processed, and frames_ is its first frame. playbackTime is when the script processor's buffer plays. performanceTime + (t − ts.contextTime)·1000 is the correct linear map, whenever the main thread handles the message. Go wasm's monotonic time.Now() reads performance.now(), so subtracting the two is consistent.
  • Main-thread stalls. Until {clear} is handled, played() stops counting at mixed, which is at most one block ahead of what was really played.

Issues

  1. A report already in flight can undo the clear in Suspend (driver_js.go#L289-L292). audioContext.suspend() is asynchronous. The audio thread can still render a few blocks, and worklet reports or audioprocess events may already be queued for the main thread. If one of them is handled after ClearPlayed, SetPlayed accepts it. While suspended, Chrome's getOutputTimestamp stops advancing, so played() keeps counting as if audio were playing for up to reportLifetime (1 s): UnplayedSize slides toward BufferedSize while nothing plays. With reports every 100 ms this is unlikely, more so when the main thread is busy. It affects both the worklet and the script processor.
    Suggested fix: a suspended atomic.Bool, set in Suspend and cleared in Resume, with reportPlayed returning early while it is set. A report from before the suspend that is handled after Resume is harmless, because context time pauses along with the frames and the mapping still holds.

  2. The first worklet report after Resume can be up to 100 ms late (nit). reportedTime_ (L113-L117) keeps its value from before the suspend, and currentTime picks up where it stopped. So UnplayedSize equals BufferedSize until another 100 ms of context time passes. Posting a {reset: true} message to the worklet on Suspend or Resume that sets reportedTime_ = -Infinity would fix it. The script processor already reports on its first event.

  3. The script processor can report less often than SetPlayed requires (edge case). It reports once per buffer (L200-L205), so a 16384-frame BufferSize at 16 kHz or less goes longer than reportLifetime between reports, and the value flips between the real count and BufferedSize. Default buffers are far below that, so a comment is probably enough.

Minor

  • outputTimestamp bool reads as if it holds a value; hasOutputTimestamp would say what it is.
  • The description says "up to 8192 frames by default". The default bufferSizeInBytes is 8192*channelCount, which is 2048 frames per transfer. That matches the measured excess (about 1–2 buffers for the script processor).
  • Only Chrome on Linux was tested. Safari, especially on iOS, would be worth a try. The mux's bounds on at mean a bad getOutputTimestamp there would fall back to BufferedSize, but it would be good to confirm that reports come through.

Review written by Claude (Claude Code) on behalf of hajimehoshi.

Marcus Johansson and others added 2 commits November 9, 2026 22:09
AudioContext.suspend is asynchronous, so a report already queued could
land after Suspend cleared the report. Reports are now dropped from
Suspend until Resume. Resume also tells the worklet to report on its
next output, rather than up to 100 ms later.

Updates ebitengine#311

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A report posted before a Suspend and handled after a quick Resume
passed the suspended check. The worklet now tags each report with a
generation that changes on every Suspend, as the PulseAudio driver
does. The underrun clear has its own flag, so a reset no longer skips
it. A report that maps to a time in the past is dropped, as its output
timestamp is stale.

Updates ebitengine#311

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@marrasen

marrasen commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for the careful review. Fixed in fcf9e79 and 0eaa3d4.

  1. A report in flight after Suspend: added suspended atomic.Bool. Suspend sets it before calling suspend(), Resume clears it, and reportPlayed returns early while it is set. This covers the worklet and the script processor.
    A second review of the fix found a remaining window: a worklet report posted before Suspend and handled after a quick Resume passed that check. So the worklet now tags each report with a generation, as in the PulseAudio fix on oto: report when frames are played on Linux and the BSDs #333. Suspend bumps reportGen, Resume sends it to the worklet, and a report with an older generation is dropped.
  2. The first worklet report after Resume: the {gen} message from Resume also sets reportedTime_ to -Infinity, so the next output reports at once. The underrun clear now has its own flag, reported_, so this reset keeps the {clear} working.
    Reporting at once raises one more case: right after a resume, getOutputTimestamp could still hold its values from before the suspend. A frame rendered now always plays in the future, so reportPlayed now drops a report that maps to a time in the past. In Chrome this never fired in 10 short Suspend→Resume cycles, so it guards other browsers.
  3. Long script processor buffers: added a comment at the report. It says that a buffer lasting longer than reportLifetime, such as 16384 frames at 16 kHz or less, leaves gaps without a report.

Minor:

  • Renamed the field to hasOutputTimestamp.
  • Corrected the description to 2048 frames per transfer by default.
  • Safari still needs a run on a Mac or an iPhone. I will post the result here when we have one.

Testing in headless Chrome 153 on Linux, 48 kHz stereo:

  • Worklet: a difference of 133 to 171 ms while playing, and 0 after Suspend. In 10 Suspend→Resume cycles of 100 to 550 ms, the report was back within 165 ms of every Resume, at 77 to 92 ms.
  • Script processor: 176 to 214 ms while playing, 0 after Suspend, and back within 15 ms of every Resume.
  • GOOS=js GOARCH=wasm go vet is clean.

Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request.

@hajimehoshi

Copy link
Copy Markdown
Member

Thanks, fcf9e79 and 0eaa3d4 handle every point. The generation tag also fixes a mistake in my first review: I said a report from before the suspend, handled after Resume, was harmless, but it isn't while the context hasn't started rendering again and getOutputTimestamp still holds its old values. On 0eaa3d4, GOOS=js GOARCH=wasm go build and go vet, gofmt, and the internal/mux tests pass.

Two new problems:

  1. If Suspend and Resume both run before the worklet module loads, the worklet's reports are dropped until the next Suspend/Resume (driver_js.go#L324-L338). Context.Suspend and Resume don't wait for ready. If both run while addModule is pending, reportGen becomes 1, Resume sees no workletPort and posts nothing, and the node is then created with gen_ = 0. Every report then fails the gen check, and UnplayedSize quietly falls back to BufferedSize. This is rare, but nothing recovers from it. Posting {gen: d.reportGen.Load()} right after creating the node, or passing the generation in processorOptions, would fix it.

  2. if ms < 0 { return } also drops good reports (driver_js.go#L363-L368). A report handled late on the main thread is not stale: the mapping holds whenever it is handled, and SetPlayed accepts a time up to reportLifetime in the past. I measured this with a minimal worklet that reports currentTime every 100 ms, as this one does, in Chromium on macOS (outputLatency 32 ms). A report lands about 28 ms ahead of when its block plays:

    Main thread busy per frame Reports with ms < 0
    0, 8 or 14 ms 0
    30 ms 3 of 49

    One dropped report is harmless, because the previous one still holds. The bad case is an underrun: it happens exactly when the main thread is late, and the report right after the {clear} is then the one most likely to be dropped. So UnplayedSize stays at BufferedSize for 100 ms or longer. Output with lower latency than this leaves even less room.

    Checking the timestamp's age would separate the two cases. In the same runs, performance.now() - ts.performanceTime was at most about 5 ms while playing, and at most 0.3 ms right after a resume in six suspend/resume cycles of 150 to 650 ms. A frozen timestamp would be about as old as the suspension. So dropping a report when that age is over a few hundred ms catches the stale case and keeps the late ones. It also covers a script processor event from before the suspend that fires after Resume, which has no generation check. Like you on Linux, I never saw a stale timestamp after resume.

Minor: workletPort is written in the addModule callback and read in Resume from another goroutine with no synchronization. Wasm runs one thread, so this can't misbehave in practice, but it is formally a data race. An atomic.Pointer or a mutex would fix it.

Review written by Claude (Claude Code) on behalf of hajimehoshi.

Marcus Johansson and others added 2 commits November 11, 2026 10:03
A report handled late on the main thread still maps correctly, so a
time in the past is no reason to drop it. A stale output timestamp is
old, so a report is now dropped when getOutputTimestamp is more than
200 ms old. This also covers a script processor event from before a
Suspend that fires after Resume.

The worklet now gets the report generation when it is created, as a
Suspend and Resume can both run before the module loads. workletPort
is now an atomic.Pointer.

Updates ebitengine#311

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Updates ebitengine#311

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@marrasen

Copy link
Copy Markdown
Contributor Author

Thank you, and for measuring the late reports. Fixed in df77fbd, with a comment fix in 0b120ce.

  1. Suspend and Resume before the module loads: the node now gets the current generation in processorOptions, and the worklet's constructor reads it. A Suspend before the module loads then counts.
  2. Dropping good reports: replaced ms < 0 with your age check. reportPlayed drops a report when performance.now() - performanceTime is more than maxOutputTimestampAge (200 ms). A late report now keeps its mapping, and a frozen timestamp from before a suspend longer than 200 ms is dropped. A shorter suspend can still pass with a stale timestamp, and the next report corrects it within 100 ms.

Minor: workletPort is now an atomic.Pointer[js.Value].

Testing in headless Chrome 153 on Linux, with a copy that prints each dropped report:

  • The age check dropped 0 reports across the worklet and the script processor, including 10 Suspend→Resume cycles in each.
  • With Suspend and Resume both called before ready, the worklet reports normally (58 to 119 ms while playing).
  • GOOS=js GOARCH=wasm go vet is clean.

Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request.

@hajimehoshi hajimehoshi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hajimehoshi
hajimehoshi merged commit e7c4c95 into ebitengine:main Nov 14, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants