From a7fd8dca3bca1778f6fb8dac10ed52079e5cb122 Mon Sep 17 00:00:00 2001 From: Serkyo Date: Mon, 3 Aug 2026 02:52:44 +0200 Subject: [PATCH] docs(workspace): document runtime diagnostics and the authority stream --- ...uthority-stream-for-server-pushed-state.md | 34 ++++++++ docs/diagnostics.md | 83 +++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 docs/adr/0011-authority-stream-for-server-pushed-state.md create mode 100644 docs/diagnostics.md diff --git a/docs/adr/0011-authority-stream-for-server-pushed-state.md b/docs/adr/0011-authority-stream-for-server-pushed-state.md new file mode 100644 index 0000000..f4cf689 --- /dev/null +++ b/docs/adr/0011-authority-stream-for-server-pushed-state.md @@ -0,0 +1,34 @@ +# 0011. A dedicated authority stream for server-pushed state + +- **Status:** Accepted +- **Date:** 2026-07-31 + +## Context + +The simulation is server-authoritative ([ADR-0004](0004-server-authoritative-simulation.md)), so a category of traffic exists that the client never asks for: state the server pushes on its own cadence. Simulation snapshots are the eventual bulk of it; periodic server diagnostics were the first concrete instance. + +Two existing streams could have absorbed that traffic, and both are a poor fit: + +- The **control stream** (stream 0) carries the handshake and disconnect. It is request/response and effectively one-shot per connection. Adding a recurring push to it mixes lifecycle negotiation with steady-state traffic, and a burst of pushed state would sit in the same ordered stream as a disconnect notice that should arrive promptly. +- The **chunk stream** (stream 3) is bidirectional and carries large frames (a 1 MiB cap). Head-of-line blocking is per-stream in QUIC, so a small, time-sensitive state push queued behind a multi-hundred-kilobyte chunk delivery would inherit that chunk's latency. That is precisely the coupling separate streams exist to avoid. + +Stream assignment is a wire contract shared by both peers: `StreamLayout` fixes the ids, and changing one is a protocol break. The decision is therefore made once, ahead of the snapshot work that will depend on it, rather than discovered later. + +## Decision + +Server-pushed authoritative state travels on its own unidirectional-in-practice stream, `StreamLayout::authority` (stream 2), carrying `shared::protocol::authority::AuthorityMessage`. + +- The stream is **server => client only**. Nothing the client sends belongs on it; client input gets its own stream when it lands. +- `AuthorityMessage` is an enum, so new pushed payloads are added as variants rather than as new streams. `ServerStats` is the first variant; simulation snapshots will join it. +- Frames use the existing length-prefixed `postcard` codec with `MAX_AUTHORITY_FRAME_LEN` (64 KiB), well above a fixed-shape diagnostics record, and set to bound what a malformed length prefix can make a peer allocate. +- The async/sync bridge follows the pattern established for chunk delivery ([ADR-0010](0010-net-crate-async-runtime.md)): the simulation loop holds an `AuthoritySink`, a synchronous non-blocking handle wrapping a `tokio` MPSC sender, so neither `server` nor `client` names a `tokio` type. +- A send on a departed connection is logged at debug and dropped. The simulation loop cannot act on a disconnected client, and pushed state is by definition unsolicited, so failure to deliver it is not an error condition for the sender. + +## Consequences + +- Latency of pushed state is independent of chunk delivery volume. A client pulling its initial region at full rate still receives snapshots on time. +- Adding a pushed payload is one enum variant, with no new stream to negotiate on either peer and no `StreamLayout` change. +- The stream layout now commits four ids (control 0, reserved 1, authority 2, chunk LOD0 3). Reassigning any of them is a `PROTOCOL_VERSION` bump. +- Loss and ordering semantics are per-stream: authority messages are ordered relative to each other and unordered relative to chunk deliveries. Anything requiring a snapshot to be interpreted against a specific delivered chunk must carry its own correlation (a tick number or chunk version), rather than relying on arrival order across streams. +- The sink is fire-and-forget and unbounded. That is appropriate for a low-rate diagnostics push, but snapshots at tick rate will need a bound and a drop policy: a slow client must not be allowed to grow the server's queue without limit. This is the known follow-up before snapshots ship. +- Diagnostics being *on* the authority stream rather than beside it means they are subject to the same server-authoritative framing: the client reports what the server measured, never what it inferred. See [`docs/diagnostics.md`](../diagnostics.md). diff --git a/docs/diagnostics.md b/docs/diagnostics.md new file mode 100644 index 0000000..4acb19f --- /dev/null +++ b/docs/diagnostics.md @@ -0,0 +1,83 @@ +# Runtime diagnostics + +How the engine reports on itself: what each crate measures, how those measurements are aggregated into the client's statistics panel, and how the server's own figures reach the client. The panel lives in [`crates/client/src/stats.rs`](../crates/client/src/stats.rs); the sources are spread across `renderer`, `net`, `server`, and `shared`. + +## Why this exists + +Every figure here is **measured, not declared**. The nominal tick rate advertised in the handshake is a constant: it states what the server intends to run at and can never reveal that it is falling behind. The same holds throughout, since a configured frame cap says nothing about achieved frame time and a load radius says nothing about how many chunks are actually resident. Diagnostics exist to close that gap, so the answer to "is this slow, and where" comes from observation rather than from configuration. + +The immediate motivation is that the client is now doing enough work per frame (materialising deliveries, dispatching mesh jobs, ingesting meshes, culling, submitting) that a frame-time regression has several plausible causes and no way to distinguish them by inspection. + +## The measurement layers + +Each crate measures what only it can see, and exposes a plain snapshot type. No crate formats, and no crate reaches into another's internals. + +| Source | Type | What it observes | +|--------|------|------------------| +| `renderer` | `RenderStats` | Uploaded / visible / culled meshes, draw calls, triangles, vertices, geometry bytes, active render mode, projection, swapchain, frames presented and skipped | +| `renderer` | `GpuInfo` | Device name and class, vendor and device ids, driver and API versions, total device-local memory. Queried once, since every field is immutable for the renderer's lifetime | +| `renderer` | `MemoryUsage` | Two independent views of GPU memory: the driver's heap accounting and the renderer's own allocator | +| `net` | `NetStats` | Application counters (chunks received, drops received, subscribes sent) plus QUIC path state (RTT, lost packets, congestion window, path MTU, bytes and datagrams) | +| `client` | `ChunkStats` | Resident chunks, uploaded meshes, in-flight mesh jobs, pending re-meshes, desired-set size | +| `client` | `FrameStats` | Frame count, mean / min / max frame time, achieved FPS over the window | +| `client` | `HostInfo` / `HostUsage` | CPU brand and core count, OS and kernel, then process CPU and memory against system totals | +| `server` | `ServerStats` | Measured TPS, mean and max tick body, tick-budget utilisation, resident and in-flight chunks, connected clients, entities, players, uptime | + +### Separating the immutable from the live + +`GpuInfo` and `HostInfo` are queried once; `MemoryUsage` and `HostUsage` are read per window. The split is deliberate: device name and driver version cannot change while the renderer lives, and re-querying them each window would pay for a string allocation to learn nothing. Live figures are read on demand precisely because they are not cacheable. + +`MemoryUsage` reports the driver's heap figures as `Option`, because they require `VK_EXT_memory_budget`. Where the extension is unavailable the allocator's own figures still report, since this process's suballocations are always knowable even when the driver's total is not. The panel must therefore render a missing driver figure as missing rather than substituting zero, which would read as "no memory in use". + +`decode_driver_version` exists because `VkPhysicalDeviceProperties::driverVersion` is documented as vendor-specific and two vendors deviate from the standard packing: NVIDIA uses a 10/8/8/6-bit layout, and Intel's *Windows* driver uses a 14/18-bit split while its Mesa driver follows the Vulkan convention. The decode is unit-tested per vendor, since a mis-decoded driver version is the kind of wrong-but-plausible output nobody notices. + +## Windowed measurement + +Everything is reported over a **window**, not instantaneously. Both `client::stats::STATS_INTERVAL` and `server::tick_stats::REPORT_INTERVAL` are one second: short enough to surface a stall promptly, long enough that producing a report costs nothing next to the work it summarises. + +A window carries a mean *and* a maximum for exactly one reason: they answer different questions. A mean comfortably inside budget alongside a spiking maximum indicates intermittent stalls, a hitch, whereas a mean at budget indicates sustained overload. Reporting only the mean hides the first case, which is the one users actually feel. + +The server additionally reports `tick_budget_percent`, the share of the nominal tick period consumed by the mean tick body. It is derived rather than measured, but it is the figure that says whether headroom exists; values at or above 100 mean the loop no longer has any. The tick body is timed *excluding* the sleep that pads a tick out to its period, so the number reflects work rather than pacing. + +`TickMeter` computes this with no division-by-zero hazard: a zero period means no budget exists to consume, so utilisation is undefined and reported as zero rather than as infinity. + +### Collection is unconditional; emission is gated + +The panel is toggled with the **F1 + I** chord (see `crates/client/src/debug.rs`), but the toggle gates *emission only*. Accumulation runs whether or not the panel is on, and the window closes on schedule either way. + +This matters more than it sounds. Gating collection on the toggle would make the first window after enabling the panel partial, reporting a fraction of a second of frames as though it were a full window, and the first thing anyone does when something feels wrong is turn the panel on. The figures must already be correct at that moment. + +The panel is emitted through `tracing` at `info` as a multi-line block, consistent with the project-wide prohibition on `println!` for diagnostics. The server formats its own figures the same way, so a dedicated server's log and a client's panel present the same numbers identically. + +## Getting the server's figures to the client + +`ServerStats` is a `shared` protocol type pushed on the authority stream (stream 2) once per window; the stream's design is [ADR-0011](adr/0011-authority-stream-for-server-pushed-state.md). The client drains it non-blockingly each frame and retains the most recent snapshot, so the panel always has a value even though server and client windows are not aligned. + +The retention is intentional: aligning the two cadences would require synchronisation for a display figure. A snapshot up to a second old is the correct trade, and the server's own `uptime_secs` makes staleness visible if it ever matters. + +The server also formats and logs the same `ServerStats` locally, so a dedicated host is diagnosable without a client attached. + +### What is *not* on the wire + +`ServerKind` (integrated, dedicated local, or dedicated remote) is deliberately **not** a protocol field. The client already knows the answer without asking: it either spawned a server in-process or dialled a socket, and a loopback address distinguishes a locally hosted process from a remote one. A server-declared field would be redundant at best and spoofable at worst, so the value is constructed client-side from facts the client already holds. + +The general rule this instances: a diagnostic should be sourced from whichever side *observes* it. The server reports its own tick health because only it can measure that; the client classifies the session because only it knows how the session was established. + +## Concurrency + +Two boundaries are crossed, with a different primitive for each. + +**Client net counters** (`NetCounters`) are incremented on the async chunk task and read on the winit thread, held behind an `Arc` and mutated with **relaxed atomics**. Relaxed is correct here rather than merely cheap: each counter is independent, nothing else is ordered against them, and a reader observing a slightly stale value is reporting a diagnostic figure, not making a decision. Paying for stronger ordering would buy precision nobody consumes. + +**Renderer frame stats** are populated at the end of every successful `draw_frame` and **retained whole** until the next frame replaces them. A reader on the panel's one-second cadence therefore observes a complete, self-consistent frame rather than a half-updated struct, a snapshot at a point rather than field-by-field sampling. That property is what makes it safe for the panel to run on a cadence unrelated to the render loop. + +## Testing + +Formatting and derivation are pure and are tested; live capture is not. + +- `crates/server/src/tests/tick_stats.rs`: window closing, mean and max derivation, budget utilisation including the zero-period case. +- `crates/renderer/src/tests/stats.rs`: `decode_driver_version` per vendor, and `cull_ratio_percent` including the nothing-uploaded case. +- `crates/client/src/tests/stats.rs`: frame accumulation and panel formatting, including absent optional sources. +- `crates/shared/src/tests/session.rs`: `ServerKind` classification from loopback and non-loopback addresses. + +Vulkan device queries, `sysinfo` host readings, and live QUIC path statistics depend on real hardware and a live connection, and are verified by running the client.