84 lines
9 KiB
Markdown
84 lines
9 KiB
Markdown
# Runtime diagnostics
|
|
|
|
How the engine reports on itself: what each crate measures, how those measurements are aggregated into the client's statistics panel, and how the server's own figures reach the client. The panel lives in [`crates/client/src/stats.rs`](../crates/client/src/stats.rs); the sources are spread across `renderer`, `net`, `server`, and `shared`.
|
|
|
|
## Why this exists
|
|
|
|
Every figure here is **measured, not declared**. The nominal tick rate advertised in the handshake is a constant: it states what the server intends to run at and can never reveal that it is falling behind. The same holds throughout, since a configured frame cap says nothing about achieved frame time and a load radius says nothing about how many chunks are actually resident. Diagnostics exist to close that gap, so the answer to "is this slow, and where" comes from observation rather than from configuration.
|
|
|
|
The immediate motivation is that the client is now doing enough work per frame (materialising deliveries, dispatching mesh jobs, ingesting meshes, culling, submitting) that a frame-time regression has several plausible causes and no way to distinguish them by inspection.
|
|
|
|
## The measurement layers
|
|
|
|
Each crate measures what only it can see, and exposes a plain snapshot type. No crate formats, and no crate reaches into another's internals.
|
|
|
|
| Source | Type | What it observes |
|
|
|--------|------|------------------|
|
|
| `renderer` | `RenderStats` | Uploaded / visible / culled meshes, draw calls, triangles, vertices, geometry bytes, active render mode, projection, swapchain, frames presented and skipped |
|
|
| `renderer` | `GpuInfo` | Device name and class, vendor and device ids, driver and API versions, total device-local memory. Queried once, since every field is immutable for the renderer's lifetime |
|
|
| `renderer` | `MemoryUsage` | Two independent views of GPU memory: the driver's heap accounting and the renderer's own allocator |
|
|
| `net` | `NetStats` | Application counters (chunks received, drops received, subscribes sent) plus QUIC path state (RTT, lost packets, congestion window, path MTU, bytes and datagrams) |
|
|
| `client` | `ChunkStats` | Resident chunks, uploaded meshes, in-flight mesh jobs, pending re-meshes, desired-set size |
|
|
| `client` | `FrameStats` | Frame count, mean / min / max frame time, achieved FPS over the window |
|
|
| `client` | `HostInfo` / `HostUsage` | CPU brand and core count, OS and kernel, then process CPU and memory against system totals |
|
|
| `server` | `ServerStats` | Measured TPS, mean and max tick body, tick-budget utilisation, resident and in-flight chunks, connected clients, entities, players, uptime |
|
|
|
|
### Separating the immutable from the live
|
|
|
|
`GpuInfo` and `HostInfo` are queried once; `MemoryUsage` and `HostUsage` are read per window. The split is deliberate: device name and driver version cannot change while the renderer lives, and re-querying them each window would pay for a string allocation to learn nothing. Live figures are read on demand precisely because they are not cacheable.
|
|
|
|
`MemoryUsage` reports the driver's heap figures as `Option`, because they require `VK_EXT_memory_budget`. Where the extension is unavailable the allocator's own figures still report, since this process's suballocations are always knowable even when the driver's total is not. The panel must therefore render a missing driver figure as missing rather than substituting zero, which would read as "no memory in use".
|
|
|
|
`decode_driver_version` exists because `VkPhysicalDeviceProperties::driverVersion` is documented as vendor-specific and two vendors deviate from the standard packing: NVIDIA uses a 10/8/8/6-bit layout, and Intel's *Windows* driver uses a 14/18-bit split while its Mesa driver follows the Vulkan convention. The decode is unit-tested per vendor, since a mis-decoded driver version is the kind of wrong-but-plausible output nobody notices.
|
|
|
|
## Windowed measurement
|
|
|
|
Everything is reported over a **window**, not instantaneously. Both `client::stats::STATS_INTERVAL` and `server::tick_stats::REPORT_INTERVAL` are one second: short enough to surface a stall promptly, long enough that producing a report costs nothing next to the work it summarises.
|
|
|
|
A window carries a mean *and* a maximum for exactly one reason: they answer different questions. A mean comfortably inside budget alongside a spiking maximum indicates intermittent stalls, a hitch, whereas a mean at budget indicates sustained overload. Reporting only the mean hides the first case, which is the one users actually feel.
|
|
|
|
The server additionally reports `tick_budget_percent`, the share of the nominal tick period consumed by the mean tick body. It is derived rather than measured, but it is the figure that says whether headroom exists; values at or above 100 mean the loop no longer has any. The tick body is timed *excluding* the sleep that pads a tick out to its period, so the number reflects work rather than pacing.
|
|
|
|
`TickMeter` computes this with no division-by-zero hazard: a zero period means no budget exists to consume, so utilisation is undefined and reported as zero rather than as infinity.
|
|
|
|
### Collection is unconditional; emission is gated
|
|
|
|
The panel is toggled with the **F1 + I** chord (see `crates/client/src/debug.rs`), but the toggle gates *emission only*. Accumulation runs whether or not the panel is on, and the window closes on schedule either way.
|
|
|
|
This matters more than it sounds. Gating collection on the toggle would make the first window after enabling the panel partial, reporting a fraction of a second of frames as though it were a full window, and the first thing anyone does when something feels wrong is turn the panel on. The figures must already be correct at that moment.
|
|
|
|
The panel is emitted through `tracing` at `info` as a multi-line block, consistent with the project-wide prohibition on `println!` for diagnostics. The server formats its own figures the same way, so a dedicated server's log and a client's panel present the same numbers identically.
|
|
|
|
## Getting the server's figures to the client
|
|
|
|
`ServerStats` is a `shared` protocol type pushed on the authority stream (stream 2) once per window; the stream's design is [ADR-0011](adr/0011-authority-stream-for-server-pushed-state.md). The client drains it non-blockingly each frame and retains the most recent snapshot, so the panel always has a value even though server and client windows are not aligned.
|
|
|
|
The retention is intentional: aligning the two cadences would require synchronisation for a display figure. A snapshot up to a second old is the correct trade, and the server's own `uptime_secs` makes staleness visible if it ever matters.
|
|
|
|
The server also formats and logs the same `ServerStats` locally, so a dedicated host is diagnosable without a client attached.
|
|
|
|
### What is *not* on the wire
|
|
|
|
`ServerKind` (integrated, dedicated local, or dedicated remote) is deliberately **not** a protocol field. The client already knows the answer without asking: it either spawned a server in-process or dialled a socket, and a loopback address distinguishes a locally hosted process from a remote one. A server-declared field would be redundant at best and spoofable at worst, so the value is constructed client-side from facts the client already holds.
|
|
|
|
The general rule this instances: a diagnostic should be sourced from whichever side *observes* it. The server reports its own tick health because only it can measure that; the client classifies the session because only it knows how the session was established.
|
|
|
|
## Concurrency
|
|
|
|
Two boundaries are crossed, with a different primitive for each.
|
|
|
|
**Client net counters** (`NetCounters`) are incremented on the async chunk task and read on the winit thread, held behind an `Arc` and mutated with **relaxed atomics**. Relaxed is correct here rather than merely cheap: each counter is independent, nothing else is ordered against them, and a reader observing a slightly stale value is reporting a diagnostic figure, not making a decision. Paying for stronger ordering would buy precision nobody consumes.
|
|
|
|
**Renderer frame stats** are populated at the end of every successful `draw_frame` and **retained whole** until the next frame replaces them. A reader on the panel's one-second cadence therefore observes a complete, self-consistent frame rather than a half-updated struct, a snapshot at a point rather than field-by-field sampling. That property is what makes it safe for the panel to run on a cadence unrelated to the render loop.
|
|
|
|
## Testing
|
|
|
|
Formatting and derivation are pure and are tested; live capture is not.
|
|
|
|
- `crates/server/src/tests/tick_stats.rs`: window closing, mean and max derivation, budget utilisation including the zero-period case.
|
|
- `crates/renderer/src/tests/stats.rs`: `decode_driver_version` per vendor, and `cull_ratio_percent` including the nothing-uploaded case.
|
|
- `crates/client/src/tests/stats.rs`: frame accumulation and panel formatting, including absent optional sources.
|
|
- `crates/shared/src/tests/session.rs`: `ServerKind` classification from loopback and non-loopback addresses.
|
|
|
|
Vulkan device queries, `sysinfo` host readings, and live QUIC path statistics depend on real hardware and a live connection, and are verified by running the client.
|