Repository navigation
oto: add Player.UnplayedSize, reported on Android - #307
Conversation
c43e32d to
cf783d0
Compare
|
Resolve the conflicts. |
OutputLatency returns how long the sound the context has read from its players takes to be heard: what the fifo queues, and what the stream holds, as its timestamp says. On Android the timestamp counts a Bluetooth headset's own delay where the headset reports it. LoopRead measures it every 100 ms. Other platforms report nothing yet. An application that shows the sound it plays, as a spectrum or a beat, needs it to show what is heard rather than what was read: over Bluetooth earbuds the difference is a quarter of a second. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
OutputLatency is declared in context.go and calls the driver's context, as Suspend, Resume and Err do. Android reports the latency, and the other drivers report none for now; each is where a later change adds its platform. outputlatency_android.go and outputlatency_other.go are gone. TestOutputLatency asks the test's context: a latency reported is zero or more, and one not reported is zero. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cf783d0 to
e478e38
Compare
|
Rebased onto 91d77a3, now that #306 is in. The conflicts were all in Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request. |
|
Cannot we implement the same things for other platforms? It's ok to focus on Android in this PR, but at least we should leave TODO comments |
Each driver that reports no latency yet has a TODO naming what it would use: PulseAudio's latency query and snd_pcm_delay on Linux, IAudioClock and waveOutGetPosition on Windows, the audio queue's time and the device's latency on macOS and iOS, and the AudioContext's latencies on the web. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hajimehoshi
left a comment
There was a problem hiding this comment.
Apart from the shape of the API, I found some issues in the Android measurement. Most of this code would carry over to the player-level design, so they apply either way.
This review was drafted with Claude (Claude Code).
|
Thank you for working on this. Before going into the details, I'd like to settle the shape of the API. Context or playerThe delay after the mux is shared by all the players, so measuring it per context in each driver makes sense. But apps need it per player: which part of this player's data is being heard now. Ebitengine's
Only the mux can get these right, so how about this?
Then Ebitengine only has to call the new method instead of I'd rather not make UnitBytes, like NameWith this, Just a suggestion, nothing decided: if a context-level value is made public, This comment was drafted with Claude (Claude Code). |
…tency The delay after the mux is shared by the players, but an app needs it per player: which of this player's data is heard now, also after Pause, right after Play, and at the end of the source. - The mux counts the samples it mixes, and each player records the spans of the mix its data went into, while they may be unheard. - Each driver reports, through Mux.SetDelayFunc, how many of the frames the mux has mixed are not heard yet. Android does; the others have a TODO naming what they would use. - Player.UnplayedSize returns the bytes read from the source that are not heard yet: what BufferedSize returns, and what was sent on and not played. Where the driver reports nothing, it returns what BufferedSize does. Seek and Reset forget what was sent. - Context.OutputLatency is gone. On Android, LoopRead measures every 100 ms when the frame written to the fifo next will be heard, and Delay counts on from that pair, so the fifo and the stream are sampled together. The stream is asked outside mutex_, and a measurement is kept only if the stream is unchanged and no callback ran meanwhile. It is forgotten wherever the stream is dropped, paused or replaced, clamped at zero, and estimated from the stream's buffer on OpenSL ES, which has no timestamps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The UnplayedSize tests run in testing/synctest bubbles and wait for the mux's loop with synctest.Wait, so their results depend on no machine's scheduler. Their cleanup lets the loop see Stop after the moment it sleeps once a source has ended. sentSpan's literal has a field a line. Doc comments say what each thing does: how spans merge is said where they merge, and ForgetDelayLocked's says only what it forgets. MeasureDelay says why a few tries are enough. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, this is a better shape. 3d1f768 reworks the PR along your outline, and 8913cd7 runs its tests in virtual time:
The six points from your review of the Android code are all in: no The new tests in One thing we haven't explained: once, during the phone test, right after a seek near the end of a track, the app's visuals went silent, as if the music had stopped, while the music played on, and they stayed silent across track changes until the app restarted. The app draws from a history of about 5.5 seconds of the mix, at the position it computes as heard, so this fits Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request. |
|
Thank you for the rework. The shape is what I had in mind, and the six points from the Android review are all addressed. The visuals that went silentFor
I don't know of a documented case where AAudio timestamps get stuck. To contain it whatever it is, how about:
One more lead, though I don't think it is this one: Smaller points
This comment was drafted with Claude (Claude Code). |
…ent while unknown The driver now hands each measurement to the mux with Mux.SetHeard: which frame of the mix is heard at what time. The mux works out the frame heard now from the clock, so nothing is read at two moments, and it no longer calls into the driver. On Android, LoopRead reports every 100 ms, under delay_mutex_, and forgetting the measurement calls Mux.ClearHeard. A report holds for a second, and one that puts a frame more than 2 seconds away is taken as unknown. While the delay is unknown, as during Suspend, a player keeps the spans it sent and leaves them out of UnplayedSize, so they count again once the driver reports. TestUnplayedSize no longer compares UnplayedSize with BufferedSize from two calls, and got/want checks use t.Errorf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks. 8655691, 3d5e24e and da8252d take all of these:
On the emulator, the stress app ran 10 minutes on 8655691: at worst 611 ms behind, the same as before, and no measurement failed. The workflow passed on all nine jobs in my fork: https://github.com/marrasen/oto/actions/runs/38528513752 On the silent visuals: it hasn't happened again, on the phone or in the stress runs, and we have no lead on why it happened. If it comes back, we'll log the raw We'll leave the Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request. |
|
Thank you for the update. I found a crash:
This can happen in an app. I reproduced it with this test in func TestSuspendBeforePlay(t *testing.T) {
if err := oboe.Suspend(); err != nil {
t.Errorf("Suspend before Play: %v", err)
}
}It passes on Could This comment was drafted with Claude (Claude Code). |
Suspend before Play called a Go function that Play sets, and panicked. The test is Hajime's, from his review. Co-authored-by: Hajime Hoshi <hajimehoshi@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for catching that. 6692dc6 does as you suggest: Your test is in Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request. |
|
Sorry, I'm the one who suggested
This comment was drafted with Claude (Claude Code). |
|
Thanks for asking first.
Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request. |
|
I'll reply this tomorrow. I'm sorry for keeping you waiting |
|
No rush! I'm using a fork for my stuff and I totally understand that you've got a lot on your table. Thanks for looking in to this! |
|
Sorry for the late reply, and thank you for the detailed answers. I'm not sure oto needs Two small things on
This comment was drafted with Claude (Claude Code). |
The doc's last sentence told callers how to use the value, not what it is. TestUnplayedSize only checked the value is not negative; the tests in internal/mux cover the behavior. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, that works for us. 64c8b2f drops the last sentence of the godoc and removes Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request. |
hajimehoshi
left a comment
There was a problem hiding this comment.
A few fixes to the docs.
This review was drafted with Claude (Claude Code).
UnplayedSize's doc says "while" in place of "where", as the report can also be cleared for a time, and says data sent before Seek or Reset still plays but is not counted. Play's doc drops the 100 ms, which is not guaranteed. maxReportedDelay's doc reads more plainly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks! 7c0329b takes all three suggestions as written. Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request. |
hajimehoshi
left a comment
There was a problem hiding this comment.
Sorry, one more round on wording. The docs and the code say both "heard" and "played" for the same moment. How about "played" throughout, to match UnplayedSize?
UnplayedSize's doc: the suggestion inline.- The internal code and its comments, for example:
Mux.SetHeard,ClearHeardandheardasSetPlayed,ClearPlayedandplayed, and theheard…fields anddropHeardlikewise,heardFunc,oto_oboe_setHeardandoto_oboe_clearHeardininternal/oboe, andheard_atinbinding_android.cpp,reportHeardand the test names ininternal/mux/unplayed_test.go,- and the TODOs in the drivers.
Two existing comments use "played" for an earlier moment, when the data leaves the player's buffer: Player.Err's doc (suggestion inline) and the comment on srcErr in internal/mux/mux.go. Once "played" means played by the audio hardware, they would read as the later moment. They predate this PR, but could they be reworded here too? For srcErr:
// It is moved to err after the data read before it is mixed or discarded.
This review was drafted with Claude (Claude Code).
Rename the internal "heard" names and comments to "played", to match UnplayedSize: Mux.SetPlayed, ClearPlayed and played, the played... fields and dropPlayed, playedFunc, oto_oboe_setPlayed and oto_oboe_clearPlayed, played_at, reportPlayed and the test names, and the TODOs in the drivers. Three comments used "played" for the earlier moment, when the data leaves the player's buffer. Player.Err's doc now says "sent to the audio hardware", and the two comments on srcErr in internal/mux say "mixed". Take the review's wording for UnplayedSize's doc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks! bcf89a3 says "played" throughout:
The workflow passed on all nine jobs in my fork: https://github.com/marrasen/oto/actions/runs/42475015543 Reply written by Claude (Anthropic), an AI agent, and posted at Marcus's request. |
This change and its description were written by Claude, an AI agent, working with @marrasen on gunim, a GUI framework that plays its sound through oto. Marcus tested it on his phone.
What issue is this addressing?
Updates #311
What type of issue is this addressing?
feature
What this PR does | solves
An application that draws the sound it plays needs to know which part of a player's data is being heard now.
BufferedSizecounts only what the player still holds, but after the mux the sound takes a while longer to reach the listener: about 250 ms over Bluetooth earbuds. This adds one method:So the part of the source being heard now is the bytes read from it less
UnplayedSize, afterPause, right afterPlay, and at the end of the source alike.The mux
Muxcounts the samplesReadFloat32smixes.Mux.SetHeard(frame, at): frameframeof the mix is heard at timeat. The mux works out the frame heard now from the clock, and caps it at the frames mixed.Mux.ClearHeardclears the report.UnplayedSizereturns whatBufferedSizedoes. The spans stay, and count again once the driver reports, as afterResume.SeekandResetdiscard the spans, as their data is from the old position.Android
Every 100 ms,
LoopReadmeasures when the frame written to the fifo next will be heard, fromcalculateLatencyMillisand the frames queued, and hands that pair to the mux: the fifo's write counter, which counts the frames read from Go, and the time. So the fifo and the stream are sampled together.mutex_. Its pointer and a generation counter are copied under the lock, and the result is kept only if the generation is unchanged and no callback ran meanwhile.delay_mutex_, so they arrive in order.The other drivers report nothing yet. Each has a TODO beside its
mux.New, naming what it would use.Context.OutputLatency, from the first version of this PR, is gone.Testing
internal/mux/unplayed_test.godrives the mux as a driver does, with reports it controls: none, steady play, afterPause, a report forgotten and made again, a report over a second old, one over 2 seconds away, the end of the source, and afterSeek. The tests run intesting/synctestbubbles and wait for the mux's loop withsynctest.Wait, so their results depend on no machine's scheduler. They pass with-race -cpu=1,4.On a Pixel 8 Pro (Android 16) with Sony WF-1000XM4 earbuds, in a music player built on gunim, Marcus checked the spectrum and the animations against the sound: in step during play, after pausing and resuming (from the lock screen too), across a change of track, and with the earbuds taken out and put back.
On the Android emulator, a stress app played through gunim and this change for 15 minutes, 2,300 random actions: seeks, mostly near the end of a track, the next track queued, pauses, the speaker suspended and resumed, its buffer resized, bursts of garbage, and the screen and the app's foreground changed from outside. What it computed as heard never fell further behind than the buffer and the device's delay, 610 ms, and no measurement failed. With the driver reporting to the mux, a 10-minute run gave the same: at worst 611 ms behind.
The branch is based on
main, and the workflow passed on it in my fork, on all nine jobs: https://github.com/marrasen/oto/actions/runs/41455412409🤖 Generated with Claude Code