Repository navigation
Pipeline the frames: the CPU records a frame while the GPU draws the one before - #259
Merged
Merged
Conversation
On Wayland the refresh rate is not known, so the cap fell back to 60 and held a 240 Hz monitor at 60 fps. With VSync on the display paces the frames, so there is no cap then. With VSync off the fallback of 60 stays. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK
…one before The renderer waited for each frame's fence right after the submit, and vkQueuePresentKHR blocks until the GPU is done on the NVIDIA Wayland driver, so the CPU and the GPU took turns (2.7 ms a frame: 0.8 CPU plus 1.9 GPU). Now two frames are in flight: - The fence wait moved to the start of the frame (the frame slot's last frame). - A present thread presents and acquires on a second queue, so the render thread does not wait in the driver. It owns the swap chain's two blocking calls (a swap chain is externally synchronized). - FrameUploader copies the data that the CPU makes each frame (camera, world matrices, draw commands, occlusion slots and parameters) from per frame staging memory into GPU buffers. The counters come back through per frame readback buffers. - FrameSync and RetireList free what a frame used only when the GPU is done with it. - ImGui and the debug pass keep a buffer pair for each frame slot. - The GPU runs one frame at a time: a semaphore chain and a barrier order the frames, because they share the G-buffer, depth and pyramid images. - The final pass waits for the G-buffer at every stage (it sampled it from the fragment shader after waiting at the color output stage only). Lighting test level, RelWithDebInfo, 1080p, RTX 3090, VSync off, no cap: 366 to 538 fps. Validation layers report nothing, also with the swap chain made again every 40 frames. Docs/frame-pipelining.md has the design and the rules for a pass. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK
…ault Off, the main thread presents a frame after it has submitted the next one, so the CPU and the GPU still overlap (460 fps against 560 with the thread and 368 before the pipelining on the lighting level). The held frame is shown at once when the frame limiter will sleep anyway (FrameLimiter::WillWait), so a capped run pays no extra frame of delay. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the lighting test level the window showed about 60 fps, and with the cap gone the frame was 2.7 ms: the CPU recorded for about 0.8 ms, then waited about 1.9 ms for the GPU, and the two never worked at the same time. Two causes:
vkQueuePresentKHRblocks until the GPU is done with the frame on the NVIDIA Wayland driver. Moving only the fence did nothing: the wait moved into the present call.Changes
Vk.Acquire,Vk.SubmitandVk.QueuePresent.FrameUploader. What the CPU makes each frame (camera, world matrices, draw commands, occlusion slots and parameters) goes into staging memory of the frame slot, and the frame's first command buffer copies it into GPU buffers. The counters come back through per frame readback buffers. The world matrix buffer is in GPU memory now.FrameSyncandRetireList: descriptor sets and buffers that a frame used are freed when the GPU is done with that frame. Writing a descriptor set that a frame in flight uses (a buffer that grows) waits for an idle device first.[Graphics] PresentThreadinConfig/EngineConf.ini, defaulttrue. A thread presents and acquires the images on a second queue of the graphics family, so the render thread does not wait in the driver. It owns both calls because a swap chain is externally synchronized. Off (or with no second queue), the render thread presents a frame after it has submitted the next one, and presents at once when the frame limiter will sleep anyway (FrameLimiter::WillWait).VulkanApp::WaitForGpuIdle).Docs/frame-pipelining.mdhas the design and the rules for a render pass that writes data each frame.Measured
Lighting test level, RelWithDebInfo, 1080p window, RTX 3090, VSync off, no cap, 30 Hz sim, spheres turning, 900 ticks:
Spheres still: 361 to 363 before, 533 to 549 after. Debug build, spinning: about 10 ms before, 7.3 ms after. With VSync on the run is held by the 240 Hz display (231 fps) before and after, but the Wayland fix means it is no longer held at 60.
The frame is GPU bound now: the render thread waits in
Vk.Acquirefor the GPU. GPU timestamp scopes would be the next step.Tests
ctest: 698/698 pass (RelWithDebInfo). New:FrameSyncTests(slot,IsDone,RetireList), aFrameLimiter::WillWaittest, and theRunSettingstests for the cap.directandinlineexec modes, and with the swap chain made again every 40 to 50 frames (about 70 times in a run).Notes
VK_LAYER_ENABLES=VK_VALIDATION_FEATURE_ENABLE_SYNCHRONIZATION_VALIDATION_EXT) reportsSYNC-HAZARD-*-RACING-*between consecutive frames. The frames are ordered by the semaphore chain and the barrier, which the layer does not follow between batches of one queue (it reports nothing when each submit is followed by a fence wait). It also reports the depth attachment of the final render pass, which the pipelining did not change.--exec threaded --frames-per-tick Nthe frame that a--screenshot-tickscapture shows can differ from run to run when frames are fast (the game thread and the frames are not locked step by step). Compare runs of one mode, or use--frustum-culling off.🤖 Generated with Claude Code
https://claude.ai/code/session_017NN37Q2jpV81xZaX3ocojK