Skip to content

Pipeline the frames: the CPU records a frame while the GPU draws the one before - #259

Merged
BenjaFriend merged 5 commits into
mainfrom
perf/cpu-bound-investigation
Nov 17, 2026
Merged

BenjaFriend merged 5 commits into
mainfrom
perf/cpu-bound-investigation

Conversation

@BenjaFriend

Copy link
Copy Markdown
Member

Problem

On the lighting test level the window showed about 60 fps, and with the cap gone the frame was 2.7 ms: the CPU recorded for about 0.8 ms, then waited about 1.9 ms for the GPU, and the two never worked at the same time. Two causes:

  1. Frame cap of 60 on Wayland. Wayland cannot say which monitor a window is on, so the refresh rate is "not known" and the cap fell back to 60 on a 240 Hz monitor.
  2. The CPU and the GPU took turns. The renderer waited for the fence of the frame it had just submitted, and vkQueuePresentKHR blocks until the GPU is done with the frame on the NVIDIA Wayland driver. Moving only the fence did nothing: the wait moved into the present call.

Changes

  1. Frame cap: with VSync on and the refresh rate not known, there is no cap (the display paces the frames). With VSync off the fallback of 60 stays.
  2. Profile scopes for Vk.Acquire, Vk.Submit and Vk.QueuePresent.
  3. Two frames in flight. The fence wait moved to the start of the frame (the frame slot's last frame).
  4. FrameUploader. What the CPU makes each frame (camera, world matrices, draw commands, occlusion slots and parameters) goes into staging memory of the frame slot, and the frame's first command buffer copies it into GPU buffers. The counters come back through per frame readback buffers. The world matrix buffer is in GPU memory now.
  5. FrameSync and RetireList: descriptor sets and buffers that a frame used are freed when the GPU is done with that frame. Writing a descriptor set that a frame in flight uses (a buffer that grows) waits for an idle device first.
  6. ImGui and the debug pass keep a buffer pair for each frame slot.
  7. The GPU runs one frame at a time. The frames share the G-buffer, depth and pyramid images, so a semaphore chain and a barrier at the start of the upload order them. The CPU records ahead; the GPU frames do not overlap.
  8. Present thread (optional). [Graphics] PresentThread in Config/EngineConf.ini, default true. A thread presents and acquires the images on a second queue of the graphics family, so the render thread does not wait in the driver. It owns both calls because a swap chain is externally synchronized. Off (or with no second queue), the render thread presents a frame after it has submitted the next one, and presents at once when the frame limiter will sleep anyway (FrameLimiter::WillWait).
  9. Fix: the final pass waited for the G-buffer at the color output stage only, and sampled it from the fragment shader (a race that the validation layers reported). It waits at every stage now.
  10. The engine waits for an idle GPU before it frees models and textures at shutdown (VulkanApp::WaitForGpuIdle).

Docs/frame-pipelining.md has the design and the rules for a render pass that writes data each frame.

Measured

Lighting test level, RelWithDebInfo, 1080p window, RTX 3090, VSync off, no cap, 30 Hz sim, spheres turning, 900 ticks:

Build Frame fps
Before (3 runs) 2.71 to 2.72 ms 367 to 369
Present thread off 2.17 ms 460
Present thread on (default, 3 runs) 1.85 to 1.87 ms 536 to 541

Spheres still: 361 to 363 before, 533 to 549 after. Debug build, spinning: about 10 ms before, 7.3 ms after. With VSync on the run is held by the 240 Hz display (231 fps) before and after, but the Wayland fix means it is no longer held at 60.

The frame is GPU bound now: the render thread waits in Vk.Acquire for the GPU. GPU timestamp scopes would be the next step.

Tests

  • ctest: 698/698 pass (RelWithDebInfo). New: FrameSyncTests (slot, IsDone, RetireList), a FrameLimiter::WillWait test, and the RunSettings tests for the cap.
  • The validation layers report nothing, with the present thread on and off, in RelWithDebInfo and Debug, with VSync on and off, with GPU blend, occlusion culling, frustum culling or interpolation off, with screenshots in a window, in the direct and inline exec modes, and with the swap chain made again every 40 to 50 frames (about 70 times in a run).
  • Offscreen pictures are identical to the old build for GPU blend on, occlusion culling off, interpolation off, and frustum culling off with the CPU blend.

Notes

  • The synchronization validation (VK_LAYER_ENABLES=VK_VALIDATION_FEATURE_ENABLE_SYNCHRONIZATION_VALIDATION_EXT) reports SYNC-HAZARD-*-RACING-* between consecutive frames. The frames are ordered by the semaphore chain and the barrier, which the layer does not follow between batches of one queue (it reports nothing when each submit is followed by a fence wait). It also reports the depth attachment of the final render pass, which the pipelining did not change.
  • With --exec threaded --frames-per-tick N the frame that a --screenshot-ticks capture shows can differ from run to run when frames are fast (the game thread and the frames are not locked step by step). Compare runs of one mode, or use --frustum-culling off.
  • Only run by hand on Linux (Wayland, NVIDIA). The other platforms are covered by CI only.

🤖 Generated with Claude Code

https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK

BenjaFriend and others added 5 commits November 17, 2026 13:52
On Wayland the refresh rate is not known, so the cap fell back to 60 and held a 240 Hz
monitor at 60 fps. With VSync on the display paces the frames, so there is no cap then.
With VSync off the fallback of 60 stays.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK
…one before

The renderer waited for each frame's fence right after the submit, and vkQueuePresentKHR
blocks until the GPU is done on the NVIDIA Wayland driver, so the CPU and the GPU took
turns (2.7 ms a frame: 0.8 CPU plus 1.9 GPU). Now two frames are in flight:

- The fence wait moved to the start of the frame (the frame slot's last frame).
- A present thread presents and acquires on a second queue, so the render thread does not
  wait in the driver. It owns the swap chain's two blocking calls (a swap chain is
  externally synchronized).
- FrameUploader copies the data that the CPU makes each frame (camera, world matrices,
  draw commands, occlusion slots and parameters) from per frame staging memory into GPU
  buffers. The counters come back through per frame readback buffers.
- FrameSync and RetireList free what a frame used only when the GPU is done with it.
- ImGui and the debug pass keep a buffer pair for each frame slot.
- The GPU runs one frame at a time: a semaphore chain and a barrier order the frames,
  because they share the G-buffer, depth and pyramid images.
- The final pass waits for the G-buffer at every stage (it sampled it from the fragment
  shader after waiting at the color output stage only).

Lighting test level, RelWithDebInfo, 1080p, RTX 3090, VSync off, no cap: 366 to 538 fps.
Validation layers report nothing, also with the swap chain made again every 40 frames.
Docs/frame-pipelining.md has the design and the rules for a pass.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK
…ault

Off, the main thread presents a frame after it has submitted the next one, so the CPU and
the GPU still overlap (460 fps against 560 with the thread and 368 before the pipelining
on the lighting level). The held frame is shown at once when the frame limiter will sleep
anyway (FrameLimiter::WillWait), so a capped run pays no extra frame of delay.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@BenjaFriend
BenjaFriend merged commit d1df0ea into main Nov 17, 2026
14 checks passed
@BenjaFriend
BenjaFriend deleted the perf/cpu-bound-investigation branch November 17, 2026 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant