diff --git a/AGENTS.md b/AGENTS.md index d006c4e..29d8f05 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,6 +28,7 @@ All paths below are relative to `internal/dashboard/` unless stated otherwise. | Logs | `logs.go`, `log_*.go` | | Reviewed actions and commands | `authorization.go`, `operations.go`, `commands.go`, `sudo.go` | | Settings and themes | `settings.go`, `preferences.go`, `themes.go`, `style.go`, `icons.go` | +| Prometheus and the Metrics suite | `metrics_*.go`, `prometheus*.go`, `internal/promapi/` (repository root) | | SSH | `remote.go`, `internal/remote/` (repository root) | Read related `*_test.go` files as needed. `helpers_test.go` contains adapters @@ -37,7 +38,7 @@ check both implementations when changing shared contracts. The extracted packages do not import `dashboard`, tview or tcell. Keep that boundary: collection and parsing belong outside the UI. Their tests run with -`go test ./internal/hostnet ./internal/dockerinspect ./internal/commandcapture`; +`go test ./internal/hostnet ./internal/dockerinspect ./internal/commandcapture ./internal/promapi`; they do not require loading dashboard fixtures or starting terminal screens. ## Documentation and assets diff --git a/CHANGELOG.md b/CHANGELOG.md index 47fd424..8494fec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ Changes are recorded here before being assigned a release tag. ## Unreleased +## v0.4.0 - 2026-10-27 + +A Metrics suite: Prometheus trend panels, targets, pinned metrics and exporter setup from inside the terminal. + +### Added + +- A sixth suite, **6 · Metrics**, visualises a Prometheus server: built-in trend panels for node_exporter, cAdvisor (standalone or kubelet), Proxmox VE (`prometheus-pve-exporter`) and Prometheus itself, drawn with the shared block/braille/ASCII renderers plus a bar variant, each with an axis naming the window's peak, pressure colours only for values that have a ceiling and gaps for missing samples, and a full-screen view per panel that flicks between panels with the arrow keys; a targets view with scrape state, errors, server version and configuration reload status; pinned metrics chosen from the server's metric names and saved in settings; a host chooser for multi-instance servers; and a setup view that detects the package manager, Docker and Proxmox and offers install steps through the usual approval card, with `sudo` steps authorised like other actions. The server is named with `--prometheus`, `SYSTEMDOC_PROMETHEUS` or `prometheus_url`, or detected from a running Prometheus service or container; **Open in Metrics** appears in the Actions menu for Prometheus and exporter workloads. A bearer token from `SYSTEMDOC_PROMETHEUS_TOKEN` is never stored. The client lives in `internal/promapi` with bounded requests, no redirects and system TLS roots only. + ## v0.3.0 - 2026-10-26 Live GPU telemetry and per-process GPU readings, CPU core inspection, and independently testable backend collectors. diff --git a/README.md b/README.md index 70a968c..8dfbf8b 100644 --- a/README.md +++ b/README.md @@ -43,13 +43,13 @@ systemdoc --docker # Docker containers systemdoc --filter state:failed # focus on failures ``` -## Five feature suites +## Six feature suites ![Containers suite showing a single-node k0s stack with a crash-looping pod selected](docs/assets/containers.png) *The Containers suite with a k0s stack: Docker containers and Kubernetes pods in one list, the Attention filter on, and a crash-looping pod's overview with its events.* -Press `0` anywhere in the live workspace to open the **Control Deck**. It explains and opens the five numbered suites, so a first-time user can understand the product without memorising commands. +Press `0` anywhere in the live workspace to open the **Control Deck**. It explains and opens the six numbered suites, so a first-time user can understand the product without memorising commands. | Suite | What it is for | Rich workflows | | --- | --- | --- | @@ -72,7 +72,7 @@ The suites share one visual language, polling interval, keyboard model, theme sy | Key | Use it for | | --- | --- | -| `0` | Open the five-suite Control Deck | +| `0` | Open the six-suite Control Deck | | `1` / `2` | Services / containers | | `3` | Ports and networking | | `4` | Process Explorer | diff --git a/SECURITY.md b/SECURITY.md index 222c17f..9d24c7c 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -12,7 +12,7 @@ Run Systemdoc as a regular user. Native systemd, journal, Docker, and SSH permis Settings and exports use owner-only permissions. Project definitions contain source paths; those sources, command arguments, labels, inspect JSON, and logs can contain secrets. The Docker overview omits environment values, but it is not a general-purpose redaction layer. Review exports and AI context before sharing. -AI assistance sends the text you review to an external CLI/provider. Authentication belongs to that CLI. SSH uses native host-key and authentication handling; only public keys are installed by key setup. Temporary uploads are private and cleaned up on exit when the connection permits. +The Metrics suite sends PromQL queries to the configured Prometheus server; a bearer token lives only in `SYSTEMDOC_PROMETHEUS_TOKEN`, is sent as a header, never stored and never printed. Credentials inside the URL are rejected. AI assistance sends the text you review to an external CLI/provider. Authentication belongs to that CLI. SSH uses native host-key and authentication handling; only public keys are installed by key setup. Temporary uploads are private and cleaned up on exit when the connection permits. ## Downloads diff --git a/SPEC.md b/SPEC.md index 9400af7..5bf13e7 100644 --- a/SPEC.md +++ b/SPEC.md @@ -20,6 +20,7 @@ A Linux and macOS terminal workspace with two modes: native services (systemd on | Kubernetes | Detected kubectl invocation (`kubectl`, `k0s kubectl`, `k3s kubectl`, `microk8s kubectl`, kind contexts, or `--kubectl`/`SYSTEMDOC_KUBECTL`); node list names the distribution; `get pods -A -o json`, `top pods`, pod JSON/YAML, events, Services, prefixed multi-container logs; `rollout restart` and `delete pod` as reviewed actions | | Host GPU | NVIDIA/DRM device telemetry with bounded per-device history; NVIDIA pmon and DRM fdinfo process usage/memory | | Host network | TCP/UDP socket ownership, interfaces, and successive-counter download/upload rates from `/proc/net/dev` or `netstat -ibn` | +| Prometheus | `internal/promapi` HTTP client with bounded bodies and timeouts, no redirects, bearer header only; exporter panel catalog (node_exporter, cAdvisor, kubelet, Proxmox VE, Prometheus) with placeholder PromQL; install-step data per platform; dashboard discovery from inventory, probe back-off, `query_range` per poll | | Compose | Installed `docker compose`; project files, profiles, environment files, workdir, endpoint | | Settings | JSON under XDG config, owner-only atomic saves | | Operations | Exact-target review, cancellable subprocesses, bounded output, session history | @@ -34,7 +35,9 @@ Versions are pinned in [go.mod](go.mod). Release binaries use `CGO_ENABLED=0` fo action review. It calls `internal/hostnet` for network collection, parsing, filtering and rate calculation, and `internal/dockerinspect` for readable container inspection output. `internal/hostgpu` owns GPU device and process -collection, parsing and interval calculation. These packages own their component tests and +collection, parsing and interval calculation. `internal/promapi` owns the +Prometheus HTTP client, response decoding, the exporter panel catalog and the +install-step data. These packages own their component tests and have no dependency on dashboard state, tview or tcell. Dashboard tests cover their integration with the UI. diff --git a/cmd/systemdoc/main.go b/cmd/systemdoc/main.go index a1e4fdb..bb608ee 100644 --- a/cmd/systemdoc/main.go +++ b/cmd/systemdoc/main.go @@ -20,6 +20,7 @@ func main() { showVersion := flag.Bool("version", false, "Print version") dockerContext := flag.String("docker-context", "", "Pin Docker context for this session") kubectl := flag.String("kubectl", "", "kubectl invocation for the Containers suite (for example \"k0s kubectl\" or \"kubectl --context kind-dev\"); detected when omitted") + prometheus := flag.String("prometheus", "", "Prometheus server URL for the Metrics suite (for example http://localhost:9090); detected from local services and containers when omitted. A bearer token is read from SYSTEMDOC_PROMETHEUS_TOKEN") host := flag.String("ssh", "", "Run the suite on an SSH host or user@host") setupKey := flag.Bool("setup-ssh-key", false, "Install a public SSH key with ssh-copy-id before connecting") sshUser := flag.String("ssh-user", "", "SSH login username (default: SSH configuration)") @@ -61,6 +62,12 @@ func main() { if *kubectl != "" { args = append(args, "--kubectl", *kubectl) } + if *prometheus == "" { + *prometheus = os.Getenv("SYSTEMDOC_PROMETHEUS") + } + if *prometheus != "" { + args = append(args, "--prometheus", *prometheus) + } options := remote.Options{Host: *host, User: *sshUser, Port: *port, Identity: *identity, Binary: *remoteBin, Upload: *upload, UploadBinary: *uploadBinary, Args: args} if *setupKey { if options.Identity == "" { @@ -98,7 +105,7 @@ func main() { if *dockerContext != "" { os.Setenv("DOCKER_CONTEXT", *dockerContext) } - if err := dashboard.Run(dashboard.Options{User: *user, Docker: *docker, Filter: *filter, ActiveOnly: *activeOnly, Kubectl: *kubectl}); err != nil { + if err := dashboard.Run(dashboard.Options{User: *user, Docker: *docker, Filter: *filter, ActiveOnly: *activeOnly, Kubectl: *kubectl, Prometheus: *prometheus}); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } diff --git a/docs/README.md b/docs/README.md index 77f94b6..e9316f8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,11 +5,12 @@ Start with [installation](installation.md), then the [user guide](usage.md). | Guide | What it covers | | --- | --- | | [Installation](installation.md) | Requirements, download, update, uninstall, source builds | -| [User guide](usage.md) | The five feature suites, keyboard controls, logs, management, Compose, AI | +| [User guide](usage.md) | The six feature suites, keyboard controls, logs, management, Compose, AI | | [Configuration](configuration.md) | Settings storage, preferences, theme migration | | [macOS services](macos.md) | Launchd support, builds and validation limits | | [Ports and networking](networking.md) | Host sockets, port owners, connections and interfaces | | [Process Explorer and Storage](host-panels.md) | Process trees, resource sorting, mounts, inodes and deleted open files | +| [Metrics suite](prometheus.md) | Prometheus trend panels for node_exporter, cAdvisor and Proxmox VE, scrape targets, pinned metrics and guided setup | | [Themes](../THEMES.md) | Five gothic Observatory palettes and the shared Nerd Font vocabulary | | [SSH and remote hosts](remote.md) | Remote runs, key setup, static binary upload | | [Troubleshooting](troubleshooting.md) | Common failures and known limits | diff --git a/docs/configuration.md b/docs/configuration.md index 6ff3fb0..50a8503 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -19,7 +19,11 @@ A minimal settings file is: "splash": true, "wrap_logs": false, "nerd_icons": true, - "systemd_active_only": false + "systemd_active_only": false, + "prometheus_url": "http://localhost:9090", + "prometheus_metrics": [ + {"title": "Open files", "name": "process_open_fds", "selector": "{job=\"prometheus\"}", "unit": "count"} + ] } ``` @@ -27,6 +31,8 @@ Missing settings use defaults. Unknown theme names resolve to Cathedral; legacy `SYSTEMDOC_KUBECTL` (or `--kubectl`) pins the kubectl invocation used for Kubernetes pods in the Containers suite, for example `k0s kubectl` or `kubectl --context kind-dev`; leave it unset to detect the stack automatically. +`--prometheus URL` (or `SYSTEMDOC_PROMETHEUS`) names the Prometheus server for the Metrics suite and overrides `prometheus_url` in settings; with neither, a running Prometheus service or container is detected, and failing that `http://localhost:9090` is tried. An optional bearer token is read from `SYSTEMDOC_PROMETHEUS_TOKEN`; it is never written to settings and is not forwarded over SSH. The flag and environment value are forwarded to a remote host, where they name that host's own loopback; the settings value is not, because the remote user has their own settings. `prometheus_metrics` holds pinned panels (`title`, `name`, optional `selector`, `unit`, `rate`, `ceiling`) and `prometheus_instances` remembers which scraped host each exporter family shows. See [the Metrics suite](prometheus.md). + `i` toggles active-only filtering and saves the systemd choice. Docker's filter is session-local. `--active-only` enables it at startup. Favourites and registered Compose project definitions are also saved. Configure projects through the project menu so file ordering, working directory, profiles, environment files, and Docker endpoint are explicit. Saved views (`saved_views`) are managed with **V** or the visible **Views** button. Each named entry stores mode, service manager, system/user scope, filters, sort, favourites-only, layout, pane ratio, inspector tab, drawer visibility and, for Docker, the endpoint. Opening a view restores these session choices; log buffers and backend data are not persisted. Invalid view fields are rejected when opening, and Docker endpoint mismatches do not switch the target. diff --git a/docs/development.md b/docs/development.md index 7badce1..653d810 100644 --- a/docs/development.md +++ b/docs/development.md @@ -31,7 +31,7 @@ python3 scripts/render-preview.py /tmp/systemdoc-cells.json 120x30 /tmp/systemdo For documentation images, capture the real binary instead of fixtures: `make build && scripts/capture-panels.sh` drives `bin/systemdoc` inside a detached 160×44 tmux server with 24-bit colour, walks every panel (dashboard, containers with pods, Docker overview, pod connections, processes, Process Activity, sysdig palette and stream, network speed, storage, constellation, Control Deck), converts each `tmux capture-pane -e` dump with `scripts/ansi-to-cells.py` into the same cell JSON, and renders PNGs into `docs/assets/` through `render-preview.py` and `rsvg-convert`. Run it with the k0s playground up (`playground/k0s.sh up`) so the container captures show a stack. It needs tmux, python3, rsvg-convert and JetBrainsMono Nerd Font. -Set `SYSTEMDOC_VISUAL_MODE=docker` for the container fixture or `approval` for the action card. The exporter includes 160×44, 120×30, and 80×24 frames. Python's standard library is sufficient to write the SVG. JetBrainsMono Nerd Font is required to review glyph spacing accurately. For README PNGs, convert the SVG with `rsvg-convert` from librsvg; this is a documentation-only dependency. +Set `SYSTEMDOC_VISUAL_MODE=docker` for the container fixture, `approval` for the action card, or `metrics` and `metrics-setup` for the Metrics suite. The exporter includes 160×44, 120×30, and 80×24 frames. Python's standard library is sufficient to write the SVG. JetBrainsMono Nerd Font is required to review glyph spacing accurately. For README PNGs, convert the SVG with `rsvg-convert` from librsvg; this is a documentation-only dependency. Review all themes and both modes. Screenshots published in documentation must say that workloads and measurements are fixtures. Never include credentials or real sensitive host configuration. @@ -40,6 +40,7 @@ Review all themes and both modes. Screenshots published in documentation must sa - `cmd/systemdoc`: flags, version, local/remote dispatch. - `internal/dashboard`: UI, systemd/launchd/Docker backends, workflows, state, themes. - `internal/hostnet`: socket collection/parsing, interface counters, rates and filters; no UI dependencies. +- `internal/promapi`: Prometheus HTTP client, exporter panel catalog, install steps; no UI dependencies. - `internal/dockerinspect`: readable Docker inspection output and its JSON fixture; no UI dependencies. - `internal/commandcapture`: bounded structured command output, separate diagnostics and cancellation. - `internal/textutil`: shared plain-text sanitization and formatting. diff --git a/docs/prometheus.md b/docs/prometheus.md new file mode 100644 index 0000000..ce807ab --- /dev/null +++ b/docs/prometheus.md @@ -0,0 +1,78 @@ +# Metrics suite (Prometheus) + +[Documentation index](README.md) · [Configuration](configuration.md) + +Press `6` from any suite, or choose **06 METRICS** in the Control Deck, to open the Metrics suite. It reads one Prometheus server over its HTTP API and shows the exporter families it finds as trend panels, lists scrape targets with server health, and, when no server answers, shows how to install one on this host. + +## Finding the server + +The suite resolves the Prometheus URL in this order: + +1. A workload opened with **Actions → Open in Metrics** (a running Prometheus service or container). +2. `--prometheus URL` or `SYSTEMDOC_PROMETHEUS`. +3. `prometheus_url` in `settings.json`, edited under **Actions → Preferences**. +4. Detection from the current inventory: an active `prometheus.service` (or the Homebrew launchd job `sh.brew.prometheus` on macOS, when the user scope is shown) means `http://localhost:9090`; a running container whose image is `prometheus` uses its published 9090 port. +5. Otherwise the conventional `http://localhost:9090`, so a local Prometheus works with no configuration at all; the setup view says so when nothing answers there. + +An optional bearer token is read from `SYSTEMDOC_PROMETHEUS_TOKEN`. It is sent as a header, never written to settings, never shown on screen and never forwarded over SSH. A username and password inside the URL are rejected. TLS uses the system certificate roots only. + +A server that does not answer is retried with back-off (30 s doubling to ten minutes). `r` probes again immediately. Previous samples stay on screen with the failure in the status line. + +## Panels + +Panels are built in for these exporter families and appear when the server has their series from the last 15 minutes: + +| Family | Detected by | Panels | +| --- | --- | --- | +| node_exporter | `node_cpu_seconds_total` | CPU busy, per-CPU, I/O wait and steal, memory, swap, load against CPU count, filesystem use, disk throughput and busy time, network throughput, uptime, pressure stall, temperatures | +| cAdvisor | `container_cpu_usage_seconds_total` | Per-container CPU, working set, memory against limit, network, age, container count | +| kubelet cAdvisor | the same metric with a `pod` label | The same panels per `namespace/pod`, plus restarts when kube-state-metrics is scraped | +| Proxmox VE (`prometheus-pve-exporter`) | `pve_up` | Node CPU, memory and status; VM and container CPU, memory, network and disk with guest names; storage use; guests running and stopped; HA state | +| Prometheus itself | `prometheus_tsdb_head_series` | Head series, samples appended per second, targets up, slowest scrapes | + +Each trend panel shows the latest value, its direction, a chart of the last 30 minutes with an axis naming the window's peak, and a caption naming the window and the step per column. Charts scale to the window's peak with a little headroom, never above the panel's ceiling, so a trend is visible whether the value is 3% or 90%; the colour of the value, not the scale, says how close it is to its ceiling. Charts follow the `G` graphics setting (blocks, braille, ASCII); `b` switches to bar glyphs. `Enter` opens the selected panel full screen with the chart across the whole terminal and every series listed under it with its own trend; `←` and `→` (or `h` and `l`) flick to the previous and next panel without leaving the view, and `Esc` returns to the grid. Panels the server has no data for are hidden and counted in the status line. Values with an honest ceiling (percentages, usage against a limit) take the green/amber/red pressure colours used everywhere else; unbounded values keep the suite's identity colour. A missing sample draws as a gap (`·`), never as zero, and a window that predates the data starts blank. Panels for per-device or per-container series show the busiest series and name how many more there are; the full-screen view lists them all with the exact PromQL. + +Every poll re-reads the window from the server with `query_range`, so a chart is always the server's view of time rather than a locally accumulated history. Polling is floored at ten seconds for this reason, and only the panels on screen are queried. Rates use a window of four scrape intervals (1m, 2m or 5m). + +When the Prometheus scrapes several hosts, the panels show this machine when its hostname or an address matches an instance, otherwise the first instance; `i` chooses another host and the choice is saved per server and family. + +### Pinning your own metrics + +`a` lists the server's metric names; pick one and give it a title, an optional label filter (`job="node",mode!="idle"`), a unit, whether to show its rate, and an optional ceiling. The pin is saved under `prometheus_metrics` in `settings.json` and appears as a **PIN** panel. `d` removes the selected pin after review. Built-in panels cannot be removed. + +## Targets and health + +`t` lists active scrape targets with state, job, instance, last scrape, duration and the last error, down targets first. The title names the Prometheus version, how many targets are up and whether the last configuration reload succeeded; the detail pane shows the scrape URL and labels. `Enter` opens the full record. + +## Setup when nothing answers + +`s` (shown automatically when no server is configured or reachable) explains where the URL comes from and lists install steps for this host, detected from the package manager (`apt-get`, `dnf`, `pacman`, `zypper`, `brew`), Docker availability and whether the host is a Proxmox VE node: + +- Prometheus and node_exporter through the package manager where the packages exist (Debian, Ubuntu, Arch, Fedora, Homebrew), or as containers. +- cAdvisor as a container. +- `prometheus-pve-exporter` on Proxmox: creating a read-only API user and token with `pveum`, installing with `pipx` or as a container, the `pve.yml` file and the scrape job. +- Scrape configuration snippets, `promtool check config` and a configuration reload. + +Numbered steps run through the same approval card as every other action; steps that need root run with `sudo` after the usual authorisation and are recorded in **Operations**. Steps marked *show only* need a file or a secret that only you can supply, so they are displayed for you to apply by hand. When an install finishes, the suite probes the server again. + +## Keys + +| Key | Action | +| --- | --- | +| p / t / s | Panels, targets, setup | +| Arrows, h/j/k/l, Home, End | Select a panel | +| Enter | Open the selected panel full screen, open a target record or run the selected setup step | +| ← → or h l (full screen) | Flick to the previous or next panel | +| a / d | Pin a metric / unpin the selected pin | +| i | Choose which scraped host a family shows | +| b | Toggle bar glyphs | +| r | Refresh now and probe the server again | +| P | Pause polling | +| G | Cycle block, braille and ASCII graphics | +| e | Review and export the page as text | +| , | Change the polling interval | +| Escape | Return to the dashboard | + +## Limits + +One server at a time. Self-signed certificates are not trusted; place such a server behind a certificate the system trusts. Alerting rules, remote write, recording rules and PromQL editing are not part of the suite. Charts show the server's samples for the last 30 minutes and do not fetch older history. diff --git a/docs/remote.md b/docs/remote.md index 78c65e9..644b5e5 100644 --- a/docs/remote.md +++ b/docs/remote.md @@ -14,7 +14,7 @@ systemdoc --ssh alice@server --docker --docker-context default systemdoc --ssh alice@server --ssh-port 2222 --identity ~/.ssh/server_key ``` -**Actions → Run on remote host** asks for the host and SSH username in separate fields. Leave the username blank to use SSH configuration (or the username in `user@host`). The CLI equivalent is `--ssh-user`; this is separate from `--user`, which selects remote user services (systemd or the launchd GUI domain). Conflicting usernames are rejected before connecting. OpenSSH handles password/key-passphrase prompts, host keys, authentication, agent use, and settings such as ProxyJump. No credentials are copied. The entire application runs remotely, so settings, native service access, Docker access, and AI authentication belong to the remote user. Exiting the remote app returns to the local shell, or the local application when launched through **Actions → Run on remote host**. +**Actions → Run on remote host** asks for the host and SSH username in separate fields. Leave the username blank to use SSH configuration (or the username in `user@host`). The CLI equivalent is `--ssh-user`; this is separate from `--user`, which selects remote user services (systemd or the launchd GUI domain). Conflicting usernames are rejected before connecting. OpenSSH handles password/key-passphrase prompts, host keys, authentication, agent use, and settings such as ProxyJump. No credentials are copied. The entire application runs remotely, so settings, native service access, Docker access, and AI authentication belong to the remote user. `--prometheus` and `SYSTEMDOC_PROMETHEUS` are forwarded and name the remote host's own loopback; `SYSTEMDOC_PROMETHEUS_TOKEN` is not forwarded and must be set in the remote environment. Exiting the remote app returns to the local shell, or the local application when launched through **Actions → Run on remote host**. On the first connection to each host/login, the app offers **Set up key**, **Use existing SSH**, or **Cancel**. Key setup uses `ssh-copy-id` to install only the public key on the remote account. With no selected identity, it creates `~/.ssh/systemdoc_ed25519` if needed using an interactive `ssh-keygen` passphrase prompt. It preserves existing keys. The chosen identity and whether the offer was shown are saved locally; passwords and private-key contents are never stored in Systemdoc settings. The connection form also has **Set up key** for repeating the workflow later. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index fa64302..cb3a093 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -55,9 +55,13 @@ Space and upward scrolling pause follow; `g` resumes. Each stream retains at mos `--upload` requires a matching static Linux ELF or macOS Mach-O executable. A `/tmp` mounted with `noexec` prevents execution: install remotely in an executable location and use `--remote-bin`. Host-key and authentication prompts belong to OpenSSH. See [remote hosts](remote.md). +## Prometheus is unreachable or shows no panels + +The Metrics suite says where its URL came from in the status line. With no `--prometheus`, `SYSTEMDOC_PROMETHEUS` or `prometheus_url`, an active `prometheus.service`, the Homebrew launchd job or a running container with the `prometheus` image is detected, and otherwise `http://localhost:9090` is tried. A server behind authentication needs `SYSTEMDOC_PROMETHEUS_TOKEN`; a 401 or 403 is reported as such. A self-signed certificate fails with a certificate error because only system roots are trusted. A reachable server without node_exporter, cAdvisor or Proxmox series shows the setup view; check `t` to see which targets are down and why, and `r` to probe again after installing an exporter. A URL that redirects (a reverse proxy adding a prefix) must be configured as its final address. + ## Current boundaries -Inventory uses CLI adapters and polling, not direct D-Bus or Docker Engine event subscriptions. Polling can miss short-lived transitions. Activity and operation history are bounded and session-local. There is no batch management, persistent audit database, advanced chart cursor, central multi-host dashboard, automatic reconnect, socket inventory, or built-in existing-file diff editor. Timer browsing lists loaded timers and requires JSON-capable `systemctl list-timers`; it does not include all installed timer files. Rich log search operates on retained buffers and does not fetch older logs. Docker Connections reports container attachments; it is not a Compose topology graph or proof of live traffic. Pod Connections lists Services whose selectors match the pod's labels; that is a routing rule, not proof of traffic, and Ingress, NetworkPolicy and multi-cluster views are not included. Kubernetes support targets one stack per node; several clusters are shown only through the current kubectl context or `--kubectl`. +Inventory uses CLI adapters and polling, not direct D-Bus or Docker Engine event subscriptions. Polling can miss short-lived transitions. Activity and operation history are bounded and session-local. There is no batch management, persistent audit database, advanced chart cursor, central multi-host dashboard, automatic reconnect, socket inventory, or built-in existing-file diff editor. The Metrics suite reads one Prometheus server, shows the last 30 minutes, trusts only system certificate roots and does not edit PromQL, rules or alerts. Timer browsing lists loaded timers and requires JSON-capable `systemctl list-timers`; it does not include all installed timer files. Rich log search operates on retained buffers and does not fetch older logs. Docker Connections reports container attachments; it is not a Compose topology graph or proof of live traffic. Pod Connections lists Services whose selectors match the pod's labels; that is a routing rule, not proof of traffic, and Ingress, NetworkPolicy and multi-cluster views are not included. Kubernetes support targets one stack per node; several clusters are shown only through the current kubectl context or `--kubectl`. Automated tests use fixtures and fake executables. They do not establish compatibility with every terminal, distribution, live AI provider, or authenticated SSH host. Before a release, run the manual checks in [the release guide](releasing.md). Report problems with version, OS/architecture, terminal, backend versions, exact steps, and sanitized output. diff --git a/docs/usage.md b/docs/usage.md index 493625d..ad8ba67 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -6,9 +6,9 @@ *Captured from the real binary against the k0s playground: the masthead names the stack, the Attention filter isolates the failing pods, and the overview shows the crash loop, the last exit and the newest events.* -## The five feature suites +## The six feature suites -Systemdoc is organised around five live operational suites. Press `0` from any suite to open the searchable **Control Deck**, which describes all five and routes directly to one. Press `1`–`5` for immediate switching. +Systemdoc is organised around six live operational suites. Press `0` from any suite to open the searchable **Control Deck**, which describes all six and routes directly to one. Press `1`–`6` for immediate switching. | Suite | Purpose | Use it when you need to… | | --- | --- | --- | @@ -17,6 +17,7 @@ Systemdoc is organised around five live operational suites. Press `0` from any s | **3 · Network** | Host ports, flows, interfaces and speed | Identify a listener or connection owner, filter by port/PID/process/protocol/state, inspect interfaces, watch download/upload throughput, or jump to process detail. | | **4 · Processes** | Host resource, ownership and signal explorer | Rank CPU/RSS, inspect parent/child trees and full commands, find zombies, review a process signal, or pivot from a PID to its service and ports. | | **5 · Storage** | Filesystem capacity and reclaim investigation | Find full mounts or inode pressure, sort/filter storage, and identify deleted files that are still held open. | +| **6 · Metrics** | Prometheus trends, targets and setup | Watch node_exporter, cAdvisor or Proxmox VE panels from your Prometheus server, pin any metric, check scrape targets and server health, or install Prometheus and exporters through reviewed commands. See [the Metrics suite](prometheus.md). | Each suite uses the same polling preference, numbered navigation rail, Deep theme, filter conventions, focus styling and reviewed export model. Observations stay tied to their actual collector; the application does not pretend separately sampled host data is one atomic snapshot. @@ -74,11 +75,12 @@ Press `z` to expand the focused pane. An expanded workload list reveals boot/sub ## Keyboard controls | Key | Action | | --- | --- | -| 0 | Open the Control Deck and descriptions of all five suites | +| 0 | Open the Control Deck and descriptions of all six suites | | 1 / 2 | Systemd / Docker | | 3 | Ports and networking | | 4 | [Process Explorer](host-panels.md) | | 5 | [Disk & Storage](host-panels.md) | +| 6 | [Metrics suite](prometheus.md): a pins a metric, d unpins, i chooses the host, b toggles bar glyphs, t targets, s setup | | F9 / K / Delete | Open reviewed signal actions for the selected process in Process Explorer | | / | Inventory filter | | , | Change live polling interval | diff --git a/internal/dashboard/control_deck.go b/internal/dashboard/control_deck.go index ee6143a..0768601 100644 --- a/internal/dashboard/control_deck.go +++ b/internal/dashboard/control_deck.go @@ -2,7 +2,7 @@ package dashboard import "fmt" -// controlDeck makes the five feature suites explicit while routing to the +// controlDeck makes the six feature suites explicit while routing to the // existing live views, so discovery never creates a second source of truth. func (w *workspace) controlDeck() { services := fleetTotals(w.items[0]) @@ -33,22 +33,27 @@ func (w *workspace) controlDeck() { "Mount capacity, free space, inode pressure, deleted files held open, filters, sorting and private evidence export", func() { w.navigateToSuite(4) }, }, + { + "06 " + w.icon(iconMetrics) + " METRICS · Prometheus trends", + "Built-in panels for node_exporter, cAdvisor and Proxmox VE, pinned metrics, scrape targets and server health, plus guided setup when no Prometheus is reachable", + func() { w.navigateToSuite(5) }, + }, } - w.choose("control-deck", w.iconLabel(iconEye, "CONTROL DECK · five operational suites"), choices) + w.choose("control-deck", w.iconLabel(iconEye, "CONTROL DECK · six operational suites"), choices) +} + +// suitePage is what every non-main suite page offers the navigation code. +type suitePage interface { + suiteIndex() int + close() + focusPrimary() } func (w *workspace) navigateToSuite(target int) { _, page := w.pages.GetFrontPage() - switch current := page.(type) { - case *networkPage: - if target == 2 { - w.app.SetFocus(current.table) - return - } - current.close() - case *hostPage: - if target == current.tab { - w.app.SetFocus(current.table) + if current, ok := page.(suitePage); ok { + if current.suiteIndex() == target { + current.focusPrimary() return } current.close() diff --git a/internal/dashboard/dashboard.go b/internal/dashboard/dashboard.go index f7880f1..bf8552d 100644 --- a/internal/dashboard/dashboard.go +++ b/internal/dashboard/dashboard.go @@ -358,7 +358,8 @@ func (w *workspace) buildDashboard() { w.toolbarButtons = append(w.toolbarButtons, network) w.commandBar.AddItem(network, suiteButtonWidths[2], 0, false) for i, label := range initialSuiteLabels[3:] { - button := tview.NewButton(label).SetSelectedFunc(func() { w.hostPage(i + 3) }) + suite := i + 3 + button := tview.NewButton(label).SetSelectedFunc(func() { w.openHostTab(suite) }) w.toolbarButtons = append(w.toolbarButtons, button) w.commandBar.AddItem(button, suiteButtonWidths[i+3], 0, false) } @@ -432,14 +433,10 @@ func (w *workspace) layoutDashboard(width, height int) { } w.lastWidth, w.lastHeight, w.layoutDetail = width, height, focusDetail w.root.Clear() - suiteButtons := append(w.modeButtons[:], w.toolbarButtons[:3]...) + suiteButtons := append(w.modeButtons[:], w.toolbarButtons[:len(hostTabNames)-2]...) setSuiteLabels(suiteButtons, width, w.settings.NerdIcons) for i, button := range suiteButtons { - buttonWidth := suiteButtonWidths[i] - if width < 100 { - buttonWidth = max(6, width/5) - } - w.commandBar.ResizeItem(button, buttonWidth, 0) + w.commandBar.ResizeItem(button, suiteButtonWidth(i, width), 0) } headerHeight := mastheadHeight(width, height) w.header.SetBorderPadding(0, 0, 1, 0) @@ -503,7 +500,7 @@ func (w *workspace) styleDashboard() { } w.table.SetBorderAttributes(tcell.AttrBold) w.detail.SetBorderAttributes(tcell.AttrBold) - w.commandBar.GetItem(5).(*tview.Box).SetBackgroundColor(tcell.GetColor(p.background)) + w.commandBar.GetItem(len(hostTabNames)).(*tview.Box).SetBackgroundColor(tcell.GetColor(p.background)) w.table.SetTitle(" " + w.iconLabel(iconEye, "WORKLOADS · S sort · Enter inspect ")).SetTitleColor(tcell.GetColor(p.accent)) w.table.SetSelectedStyle(tcell.StyleDefault.Background(tcell.GetColor(p.accent)).Foreground(tcell.GetColor(p.background)).Bold(true)) w.header.SetBackgroundColor(tcell.GetColor(p.background)) diff --git a/internal/dashboard/data.go b/internal/dashboard/data.go index 29d1358..879a7a1 100644 --- a/internal/dashboard/data.go +++ b/internal/dashboard/data.go @@ -32,6 +32,9 @@ type workload struct { MemoryPeak, IORead, IOWrite uint64 HasMemoryPeak, HasIO bool CGroup string + // Ports is the docker ps publish list ("0.0.0.0:9090->9090/tcp, :::9090->9090/tcp"), + // kept so endpoint discovery needs no second inspection. + Ports string // Stats is the container's docker stats JSON line from the bulk sample. Stats string } @@ -159,11 +162,11 @@ func listDockerContainers(ctx context.Context) ([]workload, error) { if line == "" { continue } - var row struct{ ID, Names, State, Status, Image, Labels string } + var row struct{ ID, Names, State, Status, Image, Labels, Ports string } if err := json.Unmarshal([]byte(line), &row); err != nil { return nil, err } - result = append(result, workload{ID: row.ID, Name: row.Names, State: row.State, Detail: row.Status, Project: containerProject(row.Labels), Description: row.Image}) + result = append(result, workload{ID: row.ID, Name: row.Names, State: row.State, Detail: row.Status, Project: containerProject(row.Labels), Description: row.Image, Ports: row.Ports}) } return result, nil } diff --git a/internal/dashboard/features_test.go b/internal/dashboard/features_test.go index 91fa52a..b28eaa5 100644 --- a/internal/dashboard/features_test.go +++ b/internal/dashboard/features_test.go @@ -45,16 +45,16 @@ func TestSavedViewRestoresWorkspaceAndRoundTrips(t *testing.T) { } } -func TestControlDeckDocumentsFiveSuites(t *testing.T) { +func TestControlDeckDocumentsSixSuites(t *testing.T) { w := testWorkspace() w.controlDeck() _, page := w.pages.GetFrontPage() panel := page.(*overlay).content.(*tview.Flex) list := panel.GetItem(1).(*tview.List) - if list.GetItemCount() != 5 { + if list.GetItemCount() != 6 { t.Fatalf("control deck has %d suites", list.GetItemCount()) } - for i, want := range []string{"SERVICES", "CONTAINERS", "NETWORK", "PROCESSES", "STORAGE"} { + for i, want := range hostTabNames { name, description := list.GetItemText(i) if !strings.Contains(name, want) || strings.TrimSpace(description) == "" { t.Fatalf("suite %d is not self-documenting: %q / %q", i, name, description) diff --git a/internal/dashboard/helpers_test.go b/internal/dashboard/helpers_test.go index 6c4b21e..18b5f2e 100644 --- a/internal/dashboard/helpers_test.go +++ b/internal/dashboard/helpers_test.go @@ -75,3 +75,5 @@ func hostReadoutFor(p palette, usage hostUtilisation, width int) string { } return " " + hostReadout(p, usage, width) } + +func resetPrometheusState() { resetPrometheus() } diff --git a/internal/dashboard/host_flair.go b/internal/dashboard/host_flair.go index d342345..f24484f 100644 --- a/internal/dashboard/host_flair.go +++ b/internal/dashboard/host_flair.go @@ -19,6 +19,8 @@ func panelHue(p palette, tab int) string { return hexColourOf(blend(tcell.GetColor(p.accent), tcell.GetColor(p.glow), 0.55)) case 4: return p.glow + case 5: + return hexColourOf(blend(tcell.GetColor(p.accent), tcell.GetColor(p.glow), 0.3)) default: return p.accent } diff --git a/internal/dashboard/host_ui.go b/internal/dashboard/host_ui.go index 41457c8..da6bc3f 100644 --- a/internal/dashboard/host_ui.go +++ b/internal/dashboard/host_ui.go @@ -18,9 +18,18 @@ import ( "systemdoc/internal/textutil" ) -var hostTabNames = []string{"SERVICES", "CONTAINERS", "NETWORK", "PROCESSES", "STORAGE"} -var hostTabIcons = []iconRole{iconServices, iconContainers, iconNetwork, iconProcesses, iconStorage} -var suiteButtonWidths = []int{15, 15, 14, 15, 14} +var hostTabNames = []string{"SERVICES", "CONTAINERS", "NETWORK", "PROCESSES", "STORAGE", "METRICS"} +var hostTabIcons = []iconRole{iconServices, iconContainers, iconNetwork, iconProcesses, iconStorage, iconMetrics} +var suiteButtonWidths = []int{15, 15, 14, 15, 14, 14} + +// suiteButtonWidth is the rail cell for suite i; narrow terminals share the +// width evenly so every numbered suite stays reachable. +func suiteButtonWidth(i, width int) int { + if width < 100 { + return max(6, width/len(hostTabNames)) + } + return suiteButtonWidths[i] +} // tview centres a button label inside its cell, so a button sized to its exact // label leaves no gap and the next label reads as part of this one. The extra @@ -33,7 +42,7 @@ func setSuiteLabels(buttons []*tview.Button, width int, nerd bool) { for i, button := range buttons { name := hostTabNames[i] if width < 100 { - name = []string{"SVC", "CTR", "NET", "PROC", "DISK"}[i] + name = []string{"SVC", "CTR", "NET", "PROC", "DISK", "METR"}[i] } button.SetLabel(fmt.Sprintf("%d %s %s", i+1, iconFor(nerd, hostTabIcons[i]), name)) } @@ -47,11 +56,13 @@ func (w *workspace) openHostTab(tab int) { w.networkPage() case 3, 4: w.hostPage(tab) + case 5: + w.metricsPage() } } // hostNavigation is the suite rail every non-main page shares with the main -// dashboard: five numbered suites on the left, a spacer, and the controls that +// dashboard: six numbered suites on the left, a spacer, and the controls that // apply everywhere on the right. Matching the main rail's geometry is what // keeps the top bar from reshuffling when the user changes suite. func (w *workspace) hostNavigation(active int, choose func(int)) (*tview.Flex, []*tview.Button, *tview.Button) { @@ -135,7 +146,7 @@ type hostPage struct { lastPoll time.Time note [2]string failure [2]string - modeButtons [5]*tview.Button + modeButtons [6]*tview.Button viewButtons [2]*tview.Button deckButton *tview.Button width, height int @@ -199,11 +210,7 @@ func (h *hostPage) Draw(screen tcell.Screen) { h.width, h.height = width, height setSuiteLabels(h.modeButtons[:], width, h.w.settings.NerdIcons) for i, button := range h.modeButtons { - buttonWidth := suiteButtonWidths[i] - if width < 100 { - buttonWidth = max(6, width/5) - } - h.navigation.ResizeItem(button, buttonWidth, 0) + h.navigation.ResizeItem(button, suiteButtonWidth(i, width), 0) } deckWidth := controlButtonWidth(h.deckButton.GetLabel()) if width < 140 { @@ -392,7 +399,7 @@ func (h *hostPage) input(e *tcell.EventKey) *tcell.EventKey { w.controlDeck() case ',': w.pollingDialog() - case '1', '2', '3', '4', '5': + case '1', '2', '3', '4', '5', '6': target := int(e.Rune() - '1') if target != h.tab { h.close() @@ -1226,3 +1233,6 @@ func (h *hostPage) deletedRows() ([]string, [3]hostCard) { {title: "OWNERS", headline: fmt.Sprintf("%d processes", len(owners)), visual: "Still holding files open", note: "Sampled by lsof +L1"}, } } + +func (h *hostPage) suiteIndex() int { return h.tab } +func (h *hostPage) focusPrimary() { h.w.app.SetFocus(h.table) } diff --git a/internal/dashboard/icons.go b/internal/dashboard/icons.go index 8234970..544d9fa 100644 --- a/internal/dashboard/icons.go +++ b/internal/dashboard/icons.go @@ -35,6 +35,7 @@ const ( iconLock iconStoryline iconConstellation + iconMetrics ) type iconPair struct{ nerd, fallback string } @@ -69,6 +70,7 @@ var iconVocabulary = map[iconRole]iconPair{ iconLock: {"\uf023", "!"}, iconStoryline: {"\uf1da", "TIME"}, iconConstellation: {"\uf1b3", "MAP"}, + iconMetrics: {"\uf201", "MTRC"}, } func iconFor(nerd bool, role iconRole) string { diff --git a/internal/dashboard/icons_test.go b/internal/dashboard/icons_test.go index 88c492f..a88f41a 100644 --- a/internal/dashboard/icons_test.go +++ b/internal/dashboard/icons_test.go @@ -8,7 +8,7 @@ import ( ) func TestIconVocabularyHasReadableNerdAndFallbackGlyphs(t *testing.T) { - for role := iconEye; role <= iconConstellation; role++ { + for role := iconEye; role <= iconMetrics; role++ { pair, ok := iconVocabulary[role] if !ok || strings.TrimSpace(pair.nerd) == "" || strings.TrimSpace(pair.fallback) == "" { t.Fatalf("icon role %d is incomplete: %+v", role, pair) @@ -21,7 +21,7 @@ func TestIconVocabularyHasReadableNerdAndFallbackGlyphs(t *testing.T) { func TestSuiteLabelsKeepKeysAndNamesInBothIconModes(t *testing.T) { for _, nerd := range []bool{false, true} { - buttons := make([]*tview.Button, 5) + buttons := make([]*tview.Button, len(hostTabNames)) for i := range buttons { buttons[i] = tview.NewButton("") } diff --git a/internal/dashboard/masthead_test.go b/internal/dashboard/masthead_test.go index 3574d24..0008670 100644 --- a/internal/dashboard/masthead_test.go +++ b/internal/dashboard/masthead_test.go @@ -34,6 +34,9 @@ func TestMastheadIsUniformAcrossPages(t *testing.T) { w.networkPage() capture("network") w.pages.RemovePage("network") + w.metricsPage() + capture("metrics") + w.pages.RemovePage("metrics") for _, tab := range []int{3, 4} { w.hostPage(tab) _, page := w.pages.GetFrontPage() diff --git a/internal/dashboard/metrics_chart.go b/internal/dashboard/metrics_chart.go new file mode 100644 index 0000000..194963f --- /dev/null +++ b/internal/dashboard/metrics_chart.go @@ -0,0 +1,204 @@ +package dashboard + +import ( + "fmt" + "strconv" + "strings" + "time" + + "systemdoc/internal/promapi" +) + +// trendStyle says how renderTrend draws one history. +type trendStyle struct { + width, rows int + fixedMax float64 // <= 0 scales to the window's own maximum + mode string // blocks, braille or ascii + bars bool // one glyph column and one blank column per sample +} + +// renderTrend draws the newest samples that fit the width as an area chart +// rows high. It follows signalArea's rules: eight levels per row, a reading +// above zero never draws as nothing, and a negative sample is a gap (·). +// Samples before the first known one are blank rather than gaps, so a window +// that predates the data starts clean. With two rows and bars off the output +// otherwise matches signalArea exactly. +func renderTrend(values []float64, s trendStyle) []string { + rows := max(1, s.rows) + perSample := 1 + if s.bars { + perSample = 2 + } + values = trimHistory(values, max(1, s.width/perSample)) + maximum := s.fixedMax + if maximum <= 0 { + for _, value := range values { + maximum = max(maximum, value) + } + } + maximum = max(maximum, 1) + mode := graphMode(s.mode) + glyphs := []rune(" ▁▂▃▄▅▆▇█") + if mode == "braille" { + glyphs = []rune(" ⡀⡄⡆⡇⣇⣧⣷⣿") + } + levels := rows * 8 + out := make([]strings.Builder, rows) + seen := false // a known sample has been drawn + for i, value := range values { + level := -1 + if value >= 0 { + seen = true + level = max(0, min(levels, int(value/maximum*float64(levels)))) + if value > 0 && level == 0 { + level = 1 + } + } + for row := 0; row < rows; row++ { + // out[0] is the top row; row r from the bottom holds levels r*8..r*8+8. + b := &out[rows-1-row] + switch { + case level < 0 && !seen: + b.WriteRune(' ') + case level < 0: + b.WriteRune('·') + case mode == "ascii": + if level-row*8 > 0 { + b.WriteRune('#') + } else { + b.WriteRune('.') + } + default: + b.WriteRune(glyphs[max(0, min(8, level-row*8))]) + } + if s.bars && i < len(values)-1 { + b.WriteRune(' ') + } + } + } + lines := make([]string, rows) + for i := range out { + lines[i] = strings.TrimRight(out[i].String(), " ") + } + return lines +} + +// formatMetric renders a value in its unit; a negative value is unknown. +func formatMetric(value float64, unit string) string { + if value < 0 { + return "—" + } + switch unit { + case promapi.UnitPercent: + return strconv.FormatFloat(value, 'f', 1, 64) + "%" + case promapi.UnitBytes: + return hostBytes(int64(value), false) + case promapi.UnitBytesPerSecond: + return hostBytes(int64(value), false) + "/s" + case promapi.UnitSeconds: + return formatSeconds(value) + case promapi.UnitCelsius: + return strconv.FormatFloat(value, 'f', 0, 64) + "°C" + case promapi.UnitCount: + return formatCount(value) + case promapi.UnitPerSecond: + return strconv.FormatFloat(value, 'f', 2, 64) + "/s" + case promapi.UnitLoad: + return strconv.FormatFloat(value, 'f', 2, 64) + default: + return strconv.FormatFloat(value, 'g', 4, 64) + } +} + +func formatSeconds(seconds float64) string { + switch { + case seconds < 1: + return strconv.FormatFloat(seconds*1000, 'f', 0, 64) + " ms" + case seconds < 60: + return strconv.FormatFloat(seconds, 'f', 1, 64) + " s" + case seconds < 3600: + return strconv.FormatFloat(seconds/60, 'f', 1, 64) + " m" + case seconds < 86400: + return strconv.FormatFloat(seconds/3600, 'f', 1, 64) + " h" + default: + return strconv.FormatFloat(seconds/86400, 'f', 1, 64) + " d" + } +} + +func formatCount(value float64) string { + switch { + case value >= 1e9: + return strconv.FormatFloat(value/1e9, 'f', 1, 64) + "G" + case value >= 1e6: + return strconv.FormatFloat(value/1e6, 'f', 1, 64) + "M" + case value >= 1e3: + return strconv.FormatFloat(value/1e3, 'f', 1, 64) + "k" + default: + return strconv.FormatFloat(value, 'f', 0, 64) + } +} + +// axisChart draws a trend with a small axis on the left naming the top of +// the scale, so a flat line and a full line can be told apart at a glance. +// The scale is the window's peak plus a quarter of headroom, never above the +// panel's ceiling (s.fixedMax); colour, not scale, carries the ceiling. +func axisChart(values []float64, unit string, s trendStyle) []string { + peak := 0.0 + for _, value := range values { + peak = max(peak, value) + } + scale := peak * 1.25 + if s.fixedMax > 0 { + scale = min(scale, s.fixedMax) + } + top := formatMetric(scale, unit) + width := max(displayWidth(top), 1) + s.width -= width + 2 + s.fixedMax = scale + rows := renderTrend(values, s) + for i := range rows { + label := "" + switch i { + case 0: + label = top + case len(rows) - 1: + label = "0" + } + rows[i] = fmt.Sprintf("%*s ┤%s", width, label, rows[i]) + } + return rows +} + +// knownSpan counts the columns from the first known sample to the end. +func knownSpan(values []float64) int { + for i, value := range values { + if value >= 0 { + return len(values) - i + } + } + return 0 +} + +// trendCaption names the window a chart covers: "15:02 ─ 15:17 · 15s/col". +func trendCaption(newest time.Time, step time.Duration, points, width int) string { + if newest.IsZero() || points == 0 { + return "collecting samples" + } + oldest := newest.Add(-step * time.Duration(points-1)) + span := oldest.Format("15:04") + " ─ " + newest.Format("15:04") + if width < 28 { + return span + } + return span + " · " + shortDuration(step) + "/col" +} + +func shortDuration(d time.Duration) string { + switch { + case d < time.Minute: + return strconv.Itoa(int(d.Seconds())) + "s" + case d < time.Hour: + return strconv.Itoa(int(d.Minutes())) + "m" + default: + return strconv.Itoa(int(d.Hours())) + "h" + } +} diff --git a/internal/dashboard/metrics_chart_test.go b/internal/dashboard/metrics_chart_test.go new file mode 100644 index 0000000..8d84683 --- /dev/null +++ b/internal/dashboard/metrics_chart_test.go @@ -0,0 +1,126 @@ +package dashboard + +import ( + "strings" + "testing" + "time" + + "systemdoc/internal/promapi" +) + +func TestRenderTrendMatchesSignalAreaAtTwoRows(t *testing.T) { + values := []float64{0, 3, 12.5, -1, 50, 99, 100, 0.001, 42, 7} + for _, mode := range graphModes { + for _, fixed := range []float64{0, 100} { + got := strings.Join(renderTrend(values, trendStyle{width: 40, rows: 2, fixedMax: fixed, mode: mode}), "\n") + want := signalArea(values, fixed, mode) + if got != strings.TrimRight(want, " ") { + t.Errorf("%s fixed=%v\n got %q\nwant %q", mode, fixed, got, want) + } + } + } +} + +func TestRenderTrendRulesAtMoreRows(t *testing.T) { + lines := renderTrend([]float64{0.001, 0, -1, 100}, trendStyle{width: 10, rows: 6, fixedMax: 100, mode: "blocks"}) + if len(lines) != 6 { + t.Fatalf("rows: %d", len(lines)) + } + bottom := []rune(lines[5]) + if bottom[0] != '▁' { + t.Errorf("a non-zero reading must show at least one level, got %q", string(bottom[0])) + } + if bottom[1] != ' ' { + t.Errorf("zero draws nothing, got %q", string(bottom[1])) + } + for _, line := range lines { + if []rune(line)[2] != '·' { + t.Errorf("gap must show in every row: %q", line) + } + } + if []rune(lines[0])[3] != '█' { + t.Errorf("full value must fill the top row: %q", lines[0]) + } + ascii := renderTrend([]float64{50, -1, 100}, trendStyle{width: 10, rows: 3, fixedMax: 100, mode: "ascii"}) + for _, line := range ascii { + if strings.Trim(line, "#.·") != "" { + t.Errorf("ascii uses only # . and gaps: %q", line) + } + } +} + +func TestRenderTrendTrimsAndBars(t *testing.T) { + values := make([]float64, 100) + for i := range values { + values[i] = float64(i) + } + lines := renderTrend(values, trendStyle{width: 20, rows: 2, mode: "blocks"}) + if len([]rune(lines[1])) != 20 || []rune(lines[1])[19] != '█' { + t.Errorf("newest 20 samples expected, got %q", lines[1]) + } + bars := renderTrend(values, trendStyle{width: 20, rows: 2, mode: "blocks", bars: true}) + if runes := []rune(bars[1]); len(runes) != 19 || runes[1] != ' ' || runes[0] == ' ' { + t.Errorf("bars alternate glyph and blank over half the width, got %q", bars[1]) + } +} + +func TestFormatMetricAndCaption(t *testing.T) { + cases := map[string]string{ + formatMetric(-1, promapi.UnitPercent): "—", + formatMetric(42.26, promapi.UnitPercent): "42.3%", + formatMetric(1536, promapi.UnitBytes): "1.5 KiB", + formatMetric(2048, promapi.UnitBytesPerSecond): "2.0 KiB/s", + formatMetric(0.0123, promapi.UnitSeconds): "12 ms", + formatMetric(90, promapi.UnitSeconds): "1.5 m", + formatMetric(172800, promapi.UnitSeconds): "2.0 d", + formatMetric(48213, promapi.UnitCount): "48.2k", + formatMetric(1.5, promapi.UnitPerSecond): "1.50/s", + formatMetric(0.73, promapi.UnitLoad): "0.73", + formatMetric(63, promapi.UnitCelsius): "63°C", + formatMetric(3.14159, promapi.UnitNone): "3.142", + trendCaption(time.Time{}, 0, 0, 40): "collecting samples", + trendCaption(time.Date(2024, 5, 1, 15, 17, 0, 0, time.UTC), 15*time.Second, 61, 40): "15:02 ─ 15:17 · 15s/col", + trendCaption(time.Date(2024, 5, 1, 15, 17, 0, 0, time.UTC), 15*time.Second, 61, 20): "15:02 ─ 15:17", + } + for got, want := range cases { + if got != want { + t.Errorf("got %q want %q", got, want) + } + } +} + +func BenchmarkRenderTrend(b *testing.B) { + values := make([]float64, 152) + for i := range values { + values[i] = float64(i % 100) + } + for i := 0; i < b.N; i++ { + renderTrend(values, trendStyle{width: 152, rows: 6, fixedMax: 100, mode: "braille"}) + } +} + +func TestRenderTrendLeadingGapsAreBlankAndAxisNamesThePeak(t *testing.T) { + values := []float64{-1, -1, 5, -1, 20} + rows := renderTrend(values, trendStyle{width: 10, rows: 2, mode: "blocks"}) + bottom := []rune(rows[1]) + if bottom[0] != ' ' || bottom[1] != ' ' || bottom[3] != '·' { + t.Fatalf("leading gaps blank, interior gaps dotted: %q", rows[1]) + } + if knownSpan(values) != 3 { + t.Fatalf("known span %d", knownSpan(values)) + } + axis := axisChart(values, promapi.UnitPercent, trendStyle{width: 20, rows: 3, mode: "blocks", fixedMax: 100}) + if len(axis) != 3 || !strings.HasPrefix(axis[0], "25.0% ┤") || !strings.HasPrefix(axis[2], " 0 ┤") || !strings.HasPrefix(axis[1], " ┤") { + t.Fatalf("axis rows: %q", axis) + } + if width := displayWidth(axis[2]); width > 20 { + t.Fatalf("axis chart wider than asked: %d", width) + } + if strings.HasSuffix(axis[0], " ") || strings.HasSuffix(axis[0], "█") { + t.Fatalf("the peak reaches into the top row with headroom above it: %q", axis[0]) + } + capped := axisChart([]float64{90, 95}, promapi.UnitPercent, trendStyle{width: 20, rows: 2, mode: "blocks", fixedMax: 100}) + if !strings.HasPrefix(capped[0], "100.0% ┤") { + t.Fatalf("headroom must stop at the ceiling: %q", capped[0]) + } +} diff --git a/internal/dashboard/metrics_data.go b/internal/dashboard/metrics_data.go new file mode 100644 index 0000000..f32bb24 --- /dev/null +++ b/internal/dashboard/metrics_data.go @@ -0,0 +1,462 @@ +package dashboard + +import ( + "context" + "net" + "os" + "sort" + "strings" + "sync" + "time" + + "systemdoc/internal/promapi" +) + +const ( + metricsPollFloor = 10 // seconds; every poll re-reads a window per panel + metricsWindow = 30 * time.Minute + metricsDetectEvery = 5 * time.Minute + metricsWorkers = 4 + metricsQueryLimit = 5 * time.Second +) + +type metricsState int + +const ( + metricsUnconfigured metricsState = iota + metricsProbing + metricsUnreachable + metricsNoFamilies + metricsReady +) + +// metricsPanel is one chart on the page: a catalog panel or a pinned metric +// plus whatever the last poll returned for it. +type metricsPanel struct { + spec promapi.Panel + family string + pinned bool + pin pinnedMetric + ceiling float64 + series []metricsSeries + step time.Duration + newest time.Time + failure string + empty bool // last poll returned no series +} + +// metricsSeries is one line of a panel. values holds one column per step +// with -1 for a missing sample; latest is -1 when unknown. +type metricsSeries struct { + label string + values []float64 + latest float64 +} + +// metricsDetection is what one detection pass learned about the server. +type metricsDetection struct { + families []promapi.Family + names map[string]bool + instances map[string][]string // family ID → instance label values + nodes []string // Proxmox node names + targets []promapi.Target + interval time.Duration + runtime promapi.RuntimeInfo +} + +// metricsResult is what one poll produced for one panel. +type metricsResult struct { + id string + series []metricsSeries + ceiling float64 + step time.Duration + newest time.Time + err error +} + +// detectMetrics asks the server which exporter families it has series for. +// Every call is bounded to the names the catalog cares about. +func detectMetrics(ctx context.Context, client *promapi.Client) (metricsDetection, error) { + since := time.Now().Add(-15 * time.Minute) + d := metricsDetection{names: map[string]bool{}, instances: map[string][]string{}} + names, err := client.MetricNames(ctx, promapi.DetectNames(), since) + if err != nil { + return d, err + } + for _, name := range names { + d.names[name] = true + } + kubelet := false + if d.names["container_cpu_usage_seconds_total"] { + samples, err := client.Query(ctx, `count(container_cpu_usage_seconds_total{pod!=""})`, time.Time{}) + kubelet = err == nil && len(samples) > 0 && samples[0].Value > 0 + } + d.families = promapi.DetectFamilies(names, kubelet) + for _, family := range d.families { + if family.Scope == promapi.ScopeInstance && len(family.Detect) > 0 { + d.instances[family.ID], _ = client.LabelValues(ctx, "instance", family.Detect[:1], since) + } + if family.Scope == promapi.ScopePVENode { + samples, _ := client.Query(ctx, "pve_node_info", time.Time{}) + for _, sample := range samples { + d.nodes = append(d.nodes, sample.Labels["name"]) + } + sort.Strings(d.nodes) + } + } + d.targets, _ = client.Targets(ctx) + for _, target := range d.targets { + if target.ScrapeInterval > 0 && (d.interval == 0 || target.ScrapeInterval < d.interval) { + d.interval = target.ScrapeInterval + } + } + d.runtime, _ = client.RuntimeInfo(ctx) + return d, nil +} + +// chooseInstance picks which scraped host to show when a Prometheus knows +// several: a saved choice, then this machine, then the only one, then the first. +func chooseInstance(candidates []string, saved, hostname string, localAddrs []string) string { + if len(candidates) == 0 { + return "" + } + for _, candidate := range candidates { + if candidate == saved { + return saved + } + } + short, _, _ := strings.Cut(hostname, ".") + for _, candidate := range candidates { + host := candidate + if h, _, err := net.SplitHostPort(candidate); err == nil { + host = h + } + switch { + case host == "localhost", host == "127.0.0.1", host == "::1": + return candidate + case strings.EqualFold(host, hostname), short != "" && strings.EqualFold(host, short): + return candidate + } + for _, addr := range localAddrs { + if host == addr { + return candidate + } + } + } + return candidates[0] +} + +func localAddresses() []string { + var out []string + addrs, _ := net.InterfaceAddrs() + for _, addr := range addrs { + if ip, ok := addr.(*net.IPNet); ok { + out = append(out, ip.IP.String()) + } + } + return out +} + +// panelsFor builds the page's panel list from detected families and pins. +// A panel whose metric is missing uses its fallback query or is left out. +func panelsFor(families []promapi.Family, names map[string]bool, pins []pinnedMetric) []*metricsPanel { + var panels []*metricsPanel + for _, family := range families { + for _, spec := range family.Panels { + if spec.Detect != "" && !names[spec.Detect] { + if spec.Fallback == "" { + continue + } + spec.Query = spec.Fallback + } + panels = append(panels, &metricsPanel{spec: spec, family: family.ID, ceiling: spec.Ceiling}) + } + } + for _, pin := range pins { + if pin.validate() != nil { + continue + } + spec := promapi.Panel{ID: "pin." + pin.Name + pin.Selector, Title: pin.Title, Unit: pin.Unit, Query: pin.expr(), Ceiling: pin.Ceiling, Kind: promapi.KindArea, Range: true} + panels = append(panels, &metricsPanel{spec: spec, family: "pinned", pinned: true, pin: pin, ceiling: pin.Ceiling}) + } + return panels +} + +// queryPanels runs every panel's query with a small worker pool. columns is +// how many samples a chart can show; the step follows from it. +func queryPanels(ctx context.Context, client *promapi.Client, panels []*metricsPanel, selectors map[string]promapi.Selector, columns int, interval time.Duration) []metricsResult { + results := make([]metricsResult, len(panels)) + jobs := make(chan int) + var wg sync.WaitGroup + for i := 0; i < metricsWorkers; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for i := range jobs { + results[i] = queryPanel(ctx, client, panels[i], selectors[panels[i].family], columns, interval) + } + }() + } + for i := range panels { + jobs <- i + } + close(jobs) + wg.Wait() + return results +} + +func queryPanel(ctx context.Context, client *promapi.Client, panel *metricsPanel, sel promapi.Selector, columns int, interval time.Duration) metricsResult { + ctx, cancel := context.WithTimeout(ctx, metricsQueryLimit) + defer cancel() + spec := panel.spec + result := metricsResult{id: spec.ID, ceiling: spec.Ceiling, newest: time.Now().Truncate(time.Second)} + expr := spec.Expr(sel) + if spec.CeilingQ != "" { + if samples, err := client.Query(ctx, spec.CeilingExpr(sel), time.Time{}); err == nil && len(samples) > 0 && samples[0].Known { + result.ceiling = samples[0].Value + } + } + if !spec.Range { + samples, err := client.Query(ctx, expr, time.Time{}) + result.err = err + for _, sample := range samples { + latest := -1.0 + if sample.Known { + latest = sample.Value + } + result.series = append(result.series, metricsSeries{label: seriesLabel(sample.Labels, spec.Legend), latest: latest}) + } + result.series = sortSeries(result.series, spec.TopN) + return result + } + result.step = promapi.StepFor(metricsWindow, columns, interval) + start := result.newest.Add(-result.step * time.Duration(columns-1)) + series, err := client.QueryRange(ctx, expr, start, result.newest, result.step) + result.err = err + for _, s := range series { + values := bucketPoints(s.Points, start, result.step, columns) + result.series = append(result.series, metricsSeries{label: seriesLabel(s.Labels, spec.Legend), values: values, latest: lastKnown(values)}) + } + result.series = sortSeries(result.series, spec.TopN) + return result +} + +// bucketPoints places samples on a fixed grid so every chart column is one +// step and a missing step stays a gap. +func bucketPoints(points []promapi.Point, start time.Time, step time.Duration, columns int) []float64 { + values := make([]float64, columns) + for i := range values { + values[i] = -1 + } + for _, point := range points { + i := int(point.At.Sub(start) / step) + if point.Known && i >= 0 && i < columns { + values[i] = point.Value + } + } + return values +} + +func lastKnown(values []float64) float64 { + for i := len(values) - 1; i >= 0; i-- { + if values[i] >= 0 { + return values[i] + } + } + return -1 +} + +// seriesLabel names a series by the legend labels, joined with "/". +func seriesLabel(labels promapi.Labels, legend string) string { + if legend == "" { + return "" + } + var parts []string + for _, key := range strings.Split(legend, "/") { + value := labels[key] + if value == "" { + value = "?" + } + parts = append(parts, value) + } + return strings.Join(parts, "/") +} + +// sortSeries puts the busiest series first and keeps at most top of them. +func sortSeries(series []metricsSeries, top int) []metricsSeries { + sort.SliceStable(series, func(i, j int) bool { return series[i].latest > series[j].latest }) + if top <= 0 { + top = promapi.DefaultTop + } + if len(series) > top { + series = series[:top] + } + return series +} + +// refresh polls the server off the UI thread and applies the outcome back on it. +func (m *metricsPage) refresh() { + if m.busy || m.ctx.Err() != nil { + return + } + m.lastPoll = time.Now() + endpoint := m.w.prometheusEndpoint() + m.endpoint = endpoint + if endpoint.URL == "" { + m.state = metricsUnconfigured + m.render() + return + } + if m.state == metricsUnconfigured { + m.state = metricsProbing + } + m.busy = true + m.updateStatus() + generation := m.generation + detect := time.Since(m.detectedAt) > metricsDetectEvery + panels := m.visiblePanels() + selectors := m.selectors() + columns := m.chartWidth() + pins := append([]pinnedMetric(nil), m.w.settings.PrometheusMetrics...) + saved := m.w.settings.PrometheusInstances + interval := m.interval + count := m.columns * m.visibleRows + ctx := m.ctx + go func() { + client, build, problem := currentPrometheus(ctx, endpoint) + var detection *metricsDetection + var results []metricsResult + var detectErr error + if client != nil && detect { + d, err := detectMetrics(ctx, client) + detection, detectErr = &d, err + if err == nil { + interval = d.interval + selectors = selectorsFor(d.families, d.instances, d.nodes, saved, endpoint.URL, d.interval) + panels = panelsFor(d.families, d.names, pins) + if len(panels) > count { + panels = panels[:count] + } + } + } + if client != nil && detectErr == nil { + results = queryPanels(ctx, client, panels, selectors, columns, interval) + } + m.w.queue(func() { + if ctx.Err() != nil || generation != m.generation { + return + } + m.busy = false + m.build = build + m.apply(client != nil, problem, detection, detectErr, results) + }) + }() +} + +// apply folds a poll's outcome into the page state and redraws. +func (m *metricsPage) apply(reachable bool, problem string, d *metricsDetection, detectErr error, results []metricsResult) { + switch { + case !reachable: + m.state = metricsUnreachable + m.failure = problem + case detectErr != nil: + m.state = metricsUnreachable + m.failure = detectErr.Error() + dropPrometheus(detectErr) + default: + m.failure = "" + if d != nil { + m.families, m.names, m.instances, m.nodes = d.families, d.names, d.instances, d.nodes + m.targets, m.interval, m.runtime, m.detectedAt = d.targets, d.interval, d.runtime, time.Now() + m.panels = panelsFor(d.families, d.names, m.w.settings.PrometheusMetrics) + m.selected = min(m.selected, max(0, len(m.panels)-1)) + m.hidden = 0 + } + m.state = metricsReady + if len(m.panels) == 0 { + m.state = metricsNoFamilies + } + m.sampledAt = time.Now() + } + byID := map[string]*metricsPanel{} + for _, panel := range m.panels { + byID[panel.spec.ID] = panel + } + failed := 0 + for _, result := range results { + panel := byID[result.id] + if panel == nil { + continue + } + if result.err != nil { + failed++ + panel.failure = firstLine(result.err.Error()) + continue + } + panel.failure = "" + panel.series, panel.ceiling, panel.step, panel.newest = result.series, result.ceiling, result.step, result.newest + panel.empty = len(result.series) == 0 + } + if len(results) > 0 && failed == len(results) { + m.pollFailures++ + if m.pollFailures >= 6 { + m.state = metricsUnreachable + m.failure = "every panel query failed; see the panel notes" + retryPrometheusProbe() + } + } else { + m.pollFailures = 0 + } + m.hideEmpty() + if m.state == metricsReady && !m.autoRouted { + m.autoRouted = true + if m.view == metricsSetupView { + m.view = metricsPanelsView + } + } + if m.state != metricsReady && m.state != metricsNoFamilies && !m.autoRouted { + m.view = metricsSetupView + } + m.render() +} + +// hideEmpty drops panels the server answered with no series, so a host +// without a collector does not show empty cards. They return on the next +// detection pass if data appears. +func (m *metricsPage) hideEmpty() { + kept := m.panels[:0] + for _, panel := range m.panels { + if panel.empty && panel.failure == "" { + m.hidden++ + continue + } + kept = append(kept, panel) + } + m.panels = kept + m.selected = min(m.selected, max(0, len(m.panels)-1)) +} + +// selectors returns the per-family selectors for the next poll. +func (m *metricsPage) selectors() map[string]promapi.Selector { + return selectorsFor(m.families, m.instances, m.nodes, m.w.settings.PrometheusInstances, m.endpoint.URL, m.interval) +} + +// selectorsFor decides, per family, which host and rate window the queries +// use. saved holds the user's host choices keyed by "url|family". +func selectorsFor(families []promapi.Family, instances map[string][]string, nodes []string, saved map[string]string, url string, interval time.Duration) map[string]promapi.Selector { + hostname, _ := os.Hostname() + local := localAddresses() + window := promapi.RateWindow(interval) + selectors := map[string]promapi.Selector{"pinned": {Window: window}} + for _, family := range families { + sel := promapi.Selector{Window: window} + switch family.Scope { + case promapi.ScopeInstance: + sel.Instance = chooseInstance(instances[family.ID], saved[url+"|"+family.ID], hostname, local) + case promapi.ScopePVENode: + sel.Node = chooseInstance(nodes, saved[url+"|"+family.ID], hostname, local) + } + selectors[family.ID] = sel + } + return selectors +} diff --git a/internal/dashboard/metrics_focus.go b/internal/dashboard/metrics_focus.go new file mode 100644 index 0000000..9a85057 --- /dev/null +++ b/internal/dashboard/metrics_focus.go @@ -0,0 +1,154 @@ +package dashboard + +import ( + "fmt" + "strings" + "time" + + "github.com/gdamore/tcell/v2" + "github.com/rivo/tview" +) + +// The full-screen view shows one panel with a chart across the whole +// terminal and every series listed under it. Left and right flick between +// panels; the page keeps polling the shown panel at full width. + +func (m *metricsPage) expand() { + if m.selected >= len(m.panels) || m.focus != nil { + return + } + p := m.w.palette() + view := textView().SetDynamicColors(true).SetWrap(false).SetScrollable(true) + view.SetBackgroundColor(tcell.GetColor(p.surface)).SetBorder(true).SetBorderAttributes(tcell.AttrBold).SetTitleColor(tcell.GetColor(p.accent)) + view.SetInputCapture(m.focusInput) + m.w.illuminatePanel(view.Box, func() bool { return true }, func(p palette) string { return panelHue(p, metricsTab) }, true) + m.focus = view + m.w.pages.AddPage("metrics-focus", view, true, true) + m.w.app.SetFocus(view) + m.repoll() + m.refresh() + m.renderFocus() +} + +func (m *metricsPage) closeFocus() { + if m.focus == nil { + return + } + m.focus = nil + m.w.pages.RemovePage("metrics-focus") + m.repoll() + m.rebuildGrid() + m.render() + m.focusPrimary() +} + +// repoll asks for fresh samples at the current chart width right away. +func (m *metricsPage) repoll() { + m.generation++ + m.busy = false + m.lastPoll = time.Time{} +} + +func (m *metricsPage) focusInput(e *tcell.EventKey) *tcell.EventKey { + switch e.Key() { + case tcell.KeyEscape, tcell.KeyEnter: + m.closeFocus() + return nil + case tcell.KeyLeft, tcell.KeyUp: + m.flick(-1) + return nil + case tcell.KeyRight, tcell.KeyDown: + m.flick(1) + return nil + case tcell.KeyHome: + m.flick(-len(m.panels)) + return nil + case tcell.KeyEnd: + m.flick(len(m.panels)) + return nil + } + switch e.Rune() { + case 'h', 'k': + m.flick(-1) + case 'l', 'j': + m.flick(1) + case 'G': + m.w.cycleGraphMode() + m.render() + case 'b': + m.bars = !m.bars + m.render() + case 'r': + m.repoll() + m.refresh() + case 'q': + m.closeFocus() + default: + return e + } + return nil +} + +// flick moves to another panel without leaving the full-screen view. +func (m *metricsPage) flick(delta int) { + if len(m.panels) == 0 { + return + } + m.selected = max(0, min(len(m.panels)-1, m.selected+delta)) + m.repoll() + m.refresh() + m.renderFocus() +} + +func (m *metricsPage) renderFocus() { + if m.focus == nil || m.selected >= len(m.panels) { + return + } + p := m.w.palette() + hue := panelHue(p, metricsTab) + panel := m.panels[m.selected] + family := strings.ToUpper(panel.family) + if panel.pinned { + family = "PIN" + } + m.focus.SetTitle(fmt.Sprintf(" %s · %s · %d of %d · ← → flick · Esc back ", tview.Escape(family), tview.Escape(panel.spec.Title), m.selected+1, len(m.panels))) + width, chartWidth := m.width-4, m.chartWidth() + chartRows := max(3, m.height-12) + var lines []string + switch { + case panel.failure != "": + lines = append(lines, "["+p.error+"]query failed: "+tview.Escape(panel.failure)+"[-]") + case len(panel.series) == 0: + lines = append(lines, "["+p.muted+"]"+panelEmptyText(panel)+"[-]") + case panel.spec.Range: + lead := panel.series[0] + lines = append(lines, m.headline(panel, lead, width)+" ["+p.muted+"]"+tview.Escape(panel.spec.Note)+"[-]", "") + for _, row := range axisChart(lead.values, panel.spec.Unit, trendStyle{width: chartWidth + metricsAxisWidth, rows: chartRows, fixedMax: panel.ceiling, mode: m.w.settings.GraphMode, bars: m.bars}) { + lines = append(lines, gradientText(row, hue, p.accent)) + } + lines = append(lines, "["+p.muted+"]"+tview.Escape(m.cardCaption(panel, width))+"[-]", "") + lines = append(lines, m.seriesTable(panel, width)...) + default: + lines = append(lines, m.headline(panel, panel.series[0], width)+" ["+p.muted+"]"+tview.Escape(panel.spec.Note)+"[-]", "") + lines = append(lines, m.renderBars(panel, len(panel.series), width)...) + } + lines = append(lines, "", "["+p.muted+"]"+tview.Escape(panel.spec.Expr(m.selectors()[panel.family]))+"[-]") + m.focus.SetText(strings.Join(lines, "\n")) +} + +// seriesTable lists every series with its latest value and a one-row trend. +func (m *metricsPage) seriesTable(panel *metricsPanel, width int) []string { + p := m.w.palette() + if len(panel.series) < 2 { + return nil + } + labelWidth := min(28, width/3) + sparkWidth := max(8, width-labelWidth-14) + lines := []string{"[" + p.muted + "]SERIES[-]"} + for _, series := range panel.series { + name := ellipsize(seriesName(series.label, panel.spec.Legend), labelWidth) + spark := signalChart(trimHistory(series.values, sparkWidth), m.w.settings.GraphMode) + lines = append(lines, fmt.Sprintf("%-*s %10s %s", labelWidth, tview.Escape(name), formatMetric(series.latest, panel.spec.Unit), gradientText(spark, panelHue(p, metricsTab), p.accent))) + } + return lines +} diff --git a/internal/dashboard/metrics_pins.go b/internal/dashboard/metrics_pins.go new file mode 100644 index 0000000..c933598 --- /dev/null +++ b/internal/dashboard/metrics_pins.go @@ -0,0 +1,151 @@ +package dashboard + +import ( + "fmt" + "strconv" + "strings" + "time" + + "github.com/rivo/tview" + + "systemdoc/internal/promapi" +) + +var pinUnits = []string{promapi.UnitNone, promapi.UnitPercent, promapi.UnitBytes, promapi.UnitBytesPerSecond, promapi.UnitSeconds, promapi.UnitCount, promapi.UnitPerSecond, promapi.UnitLoad, promapi.UnitCelsius} +var pinUnitLabels = []string{"plain number", "percent", "bytes", "bytes per second", "seconds", "count", "per second", "load", "°C"} + +// addMetric fetches the server's metric names and offers them in a +// searchable list; the chosen one goes to pinDialog. +func (m *metricsPage) addMetric() { + if m.endpoint.URL == "" { + m.w.message("No Prometheus", "Configure or detect a server first; s opens setup.") + return + } + endpoint, ctx := m.endpoint, m.ctx + go func() { + client, _, problem := currentPrometheus(ctx, endpoint) + var names []string + var err error + if client != nil { + names, err = client.MetricNames(ctx, nil, time.Now().Add(-15*time.Minute)) + } + m.w.queue(func() { + switch { + case ctx.Err() != nil: + case client == nil: + m.w.message("Prometheus unreachable", problem) + case err != nil: + m.w.message("Cannot list metrics", err.Error()) + default: + m.pickMetric(names) + } + }) + }() +} + +func (m *metricsPage) pickMetric(names []string) { + choices := make([]choice, 0, len(names)) + for _, name := range names { + name := name + choices = append(choices, choice{name, "Pin this metric as a panel", func() { m.pinDialog(name) }}) + } + m.w.choose("metrics-add", m.w.iconLabel(iconMetrics, "ADD METRIC · type to search"), choices) +} + +// pinDialog asks how to show one metric and saves the pin. +func (m *metricsPage) pinDialog(name string) { + w := m.w + form := tview.NewForm() + form.AddInputField("Title", name, 40, nil, nil). + AddInputField("Label filter (job=\"node\",mode!=\"idle\")", "", 40, nil, nil). + AddDropDown("Unit", pinUnitLabels, 0, nil). + AddCheckbox("Show the rate of change", false, nil). + AddInputField("Ceiling (blank scales to the window)", "", 12, nil, nil) + form.SetBorder(true).SetTitle(" PIN · " + tview.Escape(name) + " ") + close := func() { w.pages.RemovePage("metrics-pin"); m.focusPrimary() } + form.AddButton("Save", func() { + pin := pinnedMetric{Title: strings.TrimSpace(form.GetFormItem(0).(*tview.InputField).GetText()), Name: name} + if labels := strings.TrimSpace(form.GetFormItem(1).(*tview.InputField).GetText()); labels != "" { + pin.Selector = "{" + strings.Trim(labels, "{}") + "}" + } + unit, _ := form.GetFormItem(2).(*tview.DropDown).GetCurrentOption() + pin.Unit = pinUnits[unit] + pin.Rate = form.GetFormItem(3).(*tview.Checkbox).IsChecked() + if text := strings.TrimSpace(form.GetFormItem(4).(*tview.InputField).GetText()); text != "" { + ceiling, err := strconv.ParseFloat(text, 64) + if err != nil { + w.message("Invalid ceiling", "Enter a number or leave it blank.") + return + } + pin.Ceiling = ceiling + } + if err := m.savePin(pin); err != nil { + w.message("Cannot pin metric", err.Error()) + return + } + close() + }).AddButton("Cancel", close) + w.pages.AddPage("metrics-pin", centered(form, 80, 15), true, true) + w.app.SetFocus(form) +} + +// savePin validates a pin, stores it in settings and shows it as the last panel. +func (m *metricsPage) savePin(pin pinnedMetric) error { + if err := pin.validate(); err != nil { + return err + } + if len(m.w.settings.PrometheusMetrics) >= maxPinnedMetrics { + return fmt.Errorf("at most %d pinned metrics; unpin one first", maxPinnedMetrics) + } + for _, other := range m.w.settings.PrometheusMetrics { + if other.Title == pin.Title { + return fmt.Errorf("the title %q is already used", pin.Title) + } + } + next := m.w.settings + next.PrometheusMetrics = append(append([]pinnedMetric(nil), m.w.settings.PrometheusMetrics...), pin) + if !m.w.persistViewSettings(next) { + return fmt.Errorf("settings were not saved") + } + m.rebuildPanels() + m.selectPanel(len(m.panels) - 1) + return nil +} + +// removePin drops one pin from settings and the page. +func (m *metricsPage) removePin(pin pinnedMetric) { + next := m.w.settings + next.PrometheusMetrics = nil + for _, other := range m.w.settings.PrometheusMetrics { + if other != pin { + next.PrometheusMetrics = append(next.PrometheusMetrics, other) + } + } + if m.w.persistViewSettings(next) { + m.rebuildPanels() + m.selectPanel(m.selected) + } +} + +// unpinSelected removes the selected pinned panel after review. +func (m *metricsPage) unpinSelected() { + if m.selected >= len(m.panels) { + return + } + panel := m.panels[m.selected] + if !panel.pinned { + m.w.message("Built-in panel", "Only pinned metrics can be removed; built-in panels follow the detected exporters.") + return + } + w := m.w + w.confirm("unpin · "+panel.spec.Title, "Removes the pinned metric "+panel.pin.Name+panel.pin.Selector+" from settings.json. The server is not changed.", func() { m.removePin(panel.pin) }) +} + +// rebuildPanels re-derives the panel list after pins change and polls again. +func (m *metricsPage) rebuildPanels() { + m.panels = panelsFor(m.families, m.names, m.w.settings.PrometheusMetrics) + m.generation++ + m.busy = false + m.lastPoll = time.Time{} + m.render() +} diff --git a/internal/dashboard/metrics_setup.go b/internal/dashboard/metrics_setup.go new file mode 100644 index 0000000..deee0f7 --- /dev/null +++ b/internal/dashboard/metrics_setup.go @@ -0,0 +1,147 @@ +package dashboard + +import ( + "fmt" + "os" + "os/exec" + "runtime" + "strings" + + "github.com/gdamore/tcell/v2" + "github.com/rivo/tview" + + "systemdoc/internal/promapi" +) + +// detectInstall learns what this host can install with and fills the setup +// steps; it runs once per page, off the UI thread. +func (m *metricsPage) detectInstall() { + c := promapi.InstallContext{GOOS: runtime.GOOS, DistroID: distroID(), Docker: dockerUsable(m.ctx), Proxmox: isProxmox()} + for _, manager := range []string{"apt-get", "dnf", "pacman", "zypper", "brew"} { + if _, err := exec.LookPath(manager); err == nil { + c.PackageManager = manager + break + } + } + steps := promapi.InstallSteps(c) + m.w.queue(func() { + if m.ctx.Err() == nil { + m.install = steps + m.render() + } + }) +} + +func distroID() string { + data, err := os.ReadFile("/etc/os-release") + if err != nil { + return "" + } + for _, line := range strings.Split(string(data), "\n") { + if value, ok := strings.CutPrefix(line, "ID="); ok { + return strings.Trim(value, `"`) + } + } + return "" +} + +func isProxmox() bool { + if _, err := os.Stat("/etc/pve"); err == nil { + return true + } + _, err := exec.LookPath("pveversion") + return err == nil +} + +// renderSetup shows the connection state, where the URL comes from, and the +// install steps for this host. Runnable steps are numbered; the rest are +// shown for the user to apply by hand. +func (m *metricsPage) renderSetup() { + p := m.w.palette() + var out strings.Builder + head := func(text string) { fmt.Fprintf(&out, "\n[%s::b]%s[-::-]\n", panelHue(p, metricsTab), text) } + muted := func(text string) { fmt.Fprintf(&out, "[%s]%s[-]\n", p.muted, tview.Escape(text)) } + head("CONNECTION") + switch m.state { + case metricsUnconfigured: + muted("No Prometheus URL is configured and none was detected among running services or containers.") + case metricsProbing: + muted("Probing " + m.endpoint.URL + " …") + case metricsUnreachable: + if m.endpoint.Source == "default" { + muted("Nothing is configured or detected, and nothing answers on the conventional " + m.endpoint.URL) + } else { + fmt.Fprintf(&out, "[%s]%s is not answering: %s[-]\n", p.error, tview.Escape(m.endpoint.URL), tview.Escape(m.failure)) + } + muted("Retries back off up to ten minutes · r probes again now") + default: + fmt.Fprintf(&out, "[%s]%s answers[-] · prometheus %s · %s\n", p.success, tview.Escape(m.endpoint.URL), tview.Escape(m.build.Version), tview.Escape(m.endpoint.Source)) + } + path, _ := settingsPath() + muted("Set the URL with --prometheus, SYSTEMDOC_PROMETHEUS or prometheus_url in " + path) + muted("A bearer token is read from SYSTEMDOC_PROMETHEUS_TOKEN and never stored") + if len(m.install) == 0 { + head("INSTALL") + muted("Detecting package manager and Docker …") + m.setup.SetText(out.String()) + return + } + component := "" + for i, step := range m.install { + if step.Component != component { + component = step.Component + head(strings.ToUpper(strings.NewReplacer("-", " ", "_", " ").Replace(component))) + } + marker, colour := fmt.Sprintf(" %2d ", i+1), p.text + if i == m.installStep { + marker, colour = fmt.Sprintf("▸%2d ", i+1), p.accent + } + fmt.Fprintf(&out, "[%s]%s%s[-]", colour, marker, tview.Escape(step.Title)) + if step.Sudo { + fmt.Fprintf(&out, " [%s]sudo[-]", p.warning) + } + if !step.Runnable { + fmt.Fprintf(&out, " [%s]show only[-]", p.muted) + } + out.WriteString("\n") + if len(step.Argv) > 0 { + fmt.Fprintf(&out, " [%s]%s[-]\n", p.muted, tview.Escape(strings.Join(step.Argv, " "))) + } + if i == m.installStep && step.Note != "" { + fmt.Fprintf(&out, " [%s]%s[-]\n", p.muted, tview.Escape(strings.ReplaceAll(step.Note, "\n", "\n "))) + } + } + m.setup.SetText(out.String()) +} + +func (m *metricsPage) moveStep(key tcell.Key) { + if key == tcell.KeyUp { + m.installStep = max(0, m.installStep-1) + } else { + m.installStep = min(len(m.install)-1, m.installStep+1) + } + m.renderSetup() +} + +// runSelectedStep reviews the chosen install command and runs it through the +// same approval, sudo and operation history as every other action. +func (m *metricsPage) runSelectedStep() { + if m.installStep >= len(m.install) { + return + } + step := m.install[m.installStep] + if !step.Runnable || len(step.Argv) == 0 { + m.w.message(step.Title, step.Note) + return + } + w := m.w + detail := strings.Join(step.Argv, " ") + "\n\n" + step.Note + run := func() { w.execute(step.Component, step.Argv[0], step.Argv[1:]) } + if step.Sudo { + detail = "sudo -n -- " + detail + run = func() { + w.ensureSudo(func() { w.execute(step.Component, "sudo", append([]string{"-n", "--"}, step.Argv...)) }) + } + } + w.confirm("install · "+step.Title, detail, run) +} diff --git a/internal/dashboard/metrics_ui.go b/internal/dashboard/metrics_ui.go new file mode 100644 index 0000000..3122d67 --- /dev/null +++ b/internal/dashboard/metrics_ui.go @@ -0,0 +1,849 @@ +package dashboard + +import ( + "context" + "fmt" + "sort" + "strings" + "time" + + "github.com/gdamore/tcell/v2" + "github.com/rivo/tview" + + "systemdoc/internal/promapi" + "systemdoc/internal/textutil" +) + +const ( + metricsTab = 5 + metricsPageName = "metrics" +) + +type metricsView int + +const ( + metricsPanelsView metricsView = iota + metricsTargetsView + metricsSetupView +) + +var metricsViewLabels = []string{"p PANELS", "t TARGETS", "s SETUP"} + +// metricsPage is the sixth suite: Prometheus trends, scrape targets and setup. +type metricsPage struct { + *tview.Flex + w *workspace + ctx context.Context + cancel context.CancelFunc + header *tview.TextView + navigation *tview.Flex + modeButtons [6]*tview.Button + deckButton *tview.Button + viewBar *tview.Flex + viewButtons [3]*tview.Button + grid *tview.Flex + cards []*tview.TextView + table *tview.Table + setup *tview.TextView + detail *tview.TextView + status *tview.TextView + footer *tview.TextView + view metricsView + state metricsState + endpoint promEndpoint + build promapi.BuildInfo + runtime promapi.RuntimeInfo + families []promapi.Family + names map[string]bool + instances map[string][]string + nodes []string + targets []promapi.Target + interval time.Duration + detectedAt time.Time + panels []*metricsPanel + selected int + firstRow int + columns int + visibleRows int + cardHeight int + bars bool + busy, paused bool + generation int + pollFailures int + lastPoll time.Time + sampledAt time.Time + failure string + autoRouted bool + install []promapi.InstallStep + installStep int + focus *tview.TextView // full-screen view of the selected panel, nil when closed + hidden int // panels the server had no data for + width int + height int +} + +// metricsAxisWidth is the room a card keeps for the chart's axis labels. +const metricsAxisWidth = 9 + +func (w *workspace) metricsPage() *metricsPage { + if _, page := w.pages.GetFrontPage(); page != nil { + if m, ok := page.(*metricsPage); ok { + m.focusPrimary() + return m + } + } + ctx, cancel := context.WithCancel(w.ctx) + m := &metricsPage{Flex: tview.NewFlex().SetDirection(tview.FlexRow), w: w, ctx: ctx, cancel: cancel, columns: 1, visibleRows: 1, cardHeight: 6} + p := w.palette() + background, surface := tcell.GetColor(p.background), tcell.GetColor(p.surface) + m.SetBackgroundColor(background) + m.header = textView().SetDynamicColors(true).SetTextColor(tcell.GetColor(p.accent)) + m.status = textView().SetDynamicColors(true).SetWrap(false).SetTextColor(tcell.GetColor(p.muted)) + m.footer = textView().SetTextColor(tcell.GetColor(p.muted)) + for _, v := range []*tview.TextView{m.header, m.status, m.footer} { + v.SetBackgroundColor(background) + } + hue := func(p palette) string { return panelHue(p, metricsTab) } + m.detail = textView().SetDynamicColors(true).SetWrap(true).SetScrollable(true).SetTextColor(tcell.GetColor(p.text)) + m.detail.SetBackgroundColor(surface).SetBorder(true).SetTitle(" TARGET · Enter full record ").SetTitleColor(tcell.GetColor(p.accent)).SetBorderAttributes(tcell.AttrBold) + w.illuminate(m.detail.Box, m.detail.HasFocus, hue) + m.table = tview.NewTable().SetSelectable(true, false).SetFixed(1, 0) + m.table.SetBackgroundColor(background).SetBorder(true).SetTitleColor(tcell.GetColor(p.accent)).SetBorderAttributes(tcell.AttrBold) + m.table.SetSelectedStyle(tcell.StyleDefault.Background(tcell.GetColor(p.accent)).Foreground(background).Bold(true)) + m.table.SetSelectionChangedFunc(func(int, int) { m.renderTargetDetail() }) + m.table.SetSelectedFunc(func(int, int) { m.inspectTarget() }) + w.illuminate(m.table.Box, m.table.HasFocus, hue) + m.setup = textView().SetDynamicColors(true).SetWrap(true).SetScrollable(true).SetTextColor(tcell.GetColor(p.text)) + m.setup.SetBackgroundColor(surface).SetBorder(true).SetTitle(" SETUP · j/k choose a step · Enter reviews and runs it ").SetTitleColor(tcell.GetColor(p.accent)).SetBorderAttributes(tcell.AttrBold) + w.illuminate(m.setup.Box, m.setup.HasFocus, hue) + m.grid = tview.NewFlex().SetDirection(tview.FlexRow) + m.grid.SetBackgroundColor(background) + nav, modeButtons, deck := w.hostNavigation(metricsTab, func(target int) { + if target == metricsTab { + m.focusPrimary() + } else { + m.close() + w.openHostTab(target) + } + }) + copy(m.modeButtons[:], modeButtons) + m.deckButton, m.navigation = deck, nav + m.viewBar = tview.NewFlex() + m.viewBar.SetBackgroundColor(background) + for i, label := range metricsViewLabels { + view := metricsView(i) + button := tview.NewButton(label).SetSelectedFunc(func() { m.switchView(view) }) + m.viewButtons[i] = button + m.viewBar.AddItem(button, controlButtonWidth(label)+2, 0, false) + } + m.viewBar.AddItem(tview.NewBox().SetBackgroundColor(background), 0, 1, false) + m.AddItem(m.header, 1, 0, false).AddItem(nav, 1, 0, false).AddItem(m.viewBar, 1, 0, false). + AddItem(m.grid, 0, 1, true).AddItem(m.table, 0, 0, false).AddItem(m.setup, 0, 0, false). + AddItem(m.detail, 0, 0, false).AddItem(m.status, 1, 0, false).AddItem(m.footer, 1, 0, false) + m.SetInputCapture(m.input) + w.pages.AddPage(metricsPageName, m, true, true) + m.render() + m.refresh() + go m.detectInstall() + go func() { + ticker := time.NewTicker(time.Second) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + w.queueQuiet(func() { + if ctx.Err() != nil { + return + } + if front, _ := w.pages.GetFrontPage(); (front == metricsPageName || front == "metrics-focus") && !m.paused && pollDue(m.lastPoll, time.Now(), m.pollSeconds()) { + m.refresh() + } + }) + } + } + }() + return m +} + +func (m *metricsPage) close() { + m.cancel() + m.w.pages.RemovePage(metricsPageName) + m.w.app.SetFocus(m.w.table) +} + +func (m *metricsPage) suiteIndex() int { return metricsTab } + +func (m *metricsPage) focusPrimary() { + switch m.view { + case metricsTargetsView: + m.w.app.SetFocus(m.table) + case metricsSetupView: + m.w.app.SetFocus(m.setup) + default: + if card := m.selectedCard(); card != nil { + m.w.app.SetFocus(card) + } else { + m.w.app.SetFocus(m.grid) + } + } +} + +// pollSeconds floors polling because every poll re-reads a window per panel. +func (m *metricsPage) pollSeconds() int { return max(metricsPollFloor, m.w.settings.RefreshSeconds) } + +func (m *metricsPage) Draw(screen tcell.Screen) { + _, _, width, height := m.GetRect() + if m.width != width || m.height != height { + m.width, m.height = width, height + setSuiteLabels(m.modeButtons[:], width, m.w.settings.NerdIcons) + for i, button := range m.modeButtons { + m.navigation.ResizeItem(button, suiteButtonWidth(i, width), 0) + } + deckWidth := controlButtonWidth(m.deckButton.GetLabel()) + if width < 140 { + deckWidth = 0 + } + m.navigation.ResizeItem(m.deckButton, deckWidth, 0) + m.ResizeItem(m.header, mastheadHeight(width, height), 0) + m.header.SetBorderPadding(0, 0, 1, 0) + m.layout() + m.render() + } + m.Flex.Draw(screen) + p := m.w.palette() + rails := map[*tview.Box]string{} + for _, card := range m.cards { + rails[card.Box] = panelHue(p, metricsTab) + } + paintSurfaces(screen, m.Flex, p, m.header, m.table, rails, m.w.limitedColours) +} + +// layout sizes the grid for the terminal: up to three cards across and as +// many rows as fit, each card tall enough for a chart. +func (m *metricsPage) layout() { + m.columns = 1 + if m.width >= 150 { + m.columns = 3 + } else if m.width >= 110 { + m.columns = 2 + } + chartRows := 2 + switch { + case m.height >= 40: + chartRows = 5 + case m.height >= 28: + chartRows = 3 + } + m.cardHeight = chartRows + 4 + body := m.height - mastheadHeight(m.width, m.height) - 4 + detailHeight := 0 + if m.view == metricsTargetsView && m.height >= 22 { + detailHeight = 7 + } + m.visibleRows = max(1, (body-detailHeight)/m.cardHeight) + m.ResizeItem(m.detail, detailHeight, 0) + m.ResizeItem(m.grid, 0, boolInt(m.view == metricsPanelsView)) + m.ResizeItem(m.table, 0, boolInt(m.view == metricsTargetsView)) + m.ResizeItem(m.setup, 0, boolInt(m.view == metricsSetupView)) + m.rebuildGrid() +} + +func boolInt(b bool) int { + if b { + return 1 + } + return 0 +} + +// chartWidth is how many samples a chart can draw: a card's inner width +// less the axis, or the whole terminal in the full-screen view. +func (m *metricsPage) chartWidth() int { + if m.width == 0 { + return 40 + } + if m.focus != nil { + return max(8, min(240, m.width-metricsAxisWidth-4)) + } + return max(8, min(120, m.width/m.columns-metricsAxisWidth-4)) +} + +// visiblePanels are the cards on screen, or the one panel shown full screen; +// only they are polled. +func (m *metricsPage) visiblePanels() []*metricsPanel { + if m.focus != nil && m.selected < len(m.panels) { + return m.panels[m.selected : m.selected+1] + } + perPage := m.columns * m.visibleRows + start := min(m.firstRow*m.columns, len(m.panels)) + return m.panels[start:min(start+perPage, len(m.panels))] +} + +// rebuildGrid lays the visible cards out in rows, reusing card widgets. +func (m *metricsPage) rebuildGrid() { + p := m.w.palette() + visible := m.visiblePanels() + for len(m.cards) < len(visible) { + i := len(m.cards) + card := textView().SetDynamicColors(true).SetWrap(false) + card.SetBackgroundColor(tcell.GetColor(p.surface)).SetBorder(true).SetBorderAttributes(tcell.AttrBold) + m.w.illuminatePanel(card.Box, card.HasFocus, func(p palette) string { return panelHue(p, metricsTab) }, true) + card.SetMouseCapture(func(action tview.MouseAction, event *tcell.EventMouse) (tview.MouseAction, *tcell.EventMouse) { + if action == tview.MouseLeftClick { + m.selected = m.firstRow*m.columns + i + m.focusPrimary() + m.render() + } + return action, event + }) + m.cards = append(m.cards, card) + } + m.cards = m.cards[:len(visible)] + m.grid.Clear() + for row := 0; row*m.columns < len(visible); row++ { + line := tview.NewFlex() + line.SetBackgroundColor(tcell.GetColor(p.background)) + for col := 0; col < m.columns; col++ { + if i := row*m.columns + col; i < len(visible) { + line.AddItem(m.cards[i], 0, 1, false) + } else { + line.AddItem(tview.NewBox().SetBackgroundColor(tcell.GetColor(p.background)), 0, 1, false) + } + } + m.grid.AddItem(line, m.cardHeight, 0, false) + } + m.grid.AddItem(tview.NewBox().SetBackgroundColor(tcell.GetColor(p.background)), 0, 1, false) +} + +func (m *metricsPage) selectedCard() *tview.TextView { + i := m.selected - m.firstRow*m.columns + if m.view != metricsPanelsView || i < 0 || i >= len(m.cards) { + return nil + } + return m.cards[i] +} + +// render redraws everything from the page state; it never touches the network. +func (m *metricsPage) render() { + p := m.w.palette() + m.header.SetText(m.w.masthead("Metrics", "", hostReadout(p, m.w.hostUsage, m.width)+" "+m.w.signalsHint(p), m.width, m.height)) + for i, button := range m.viewButtons { + fg, bg := p.muted, p.background + if metricsView(i) == m.view { + fg, bg = p.background, panelHue(p, metricsTab) + } + button.SetStyle(tcell.StyleDefault.Foreground(tcell.GetColor(fg)).Background(tcell.GetColor(bg)).Bold(metricsView(i) == m.view)) + } + switch m.view { + case metricsPanelsView: + m.renderPanels() + case metricsTargetsView: + m.renderTargets() + case metricsSetupView: + m.renderSetup() + } + if m.focus != nil { + m.renderFocus() + } + m.updateStatus() + m.updateFooter() +} + +func (m *metricsPage) renderPanels() { + visible := m.visiblePanels() + if len(m.cards) != len(visible) { + m.rebuildGrid() + } + for i, panel := range visible { + m.renderCard(m.cards[i], panel) + } +} + +// renderCard fills one card: a title, the latest value, the trend and a caption. +func (m *metricsPage) renderCard(card *tview.TextView, panel *metricsPanel) { + p := m.w.palette() + hue := panelHue(p, metricsTab) + family := strings.ToUpper(panel.family) + if panel.pinned { + family = "PIN" + } + card.SetTitle(" " + tview.Escape(family+" · "+panel.spec.Title) + " ").SetTitleColor(tcell.GetColor(hue)) + chartWidth, chartRows := m.chartWidth(), m.cardHeight-4 + var lines []string + if panel.failure != "" { + lines = append(lines, "["+p.error+"]query failed[-]", "["+p.muted+"]"+tview.Escape(ellipsize(panel.failure, chartWidth))+"[-]") + } else if len(panel.series) == 0 { + lines = append(lines, "["+p.muted+"]"+panelEmptyText(panel)+"[-]", "["+p.muted+"]"+tview.Escape(ellipsize(panel.spec.Note, chartWidth))+"[-]") + } else { + lead := panel.series[0] + lines = append(lines, m.headline(panel, lead, chartWidth)) + if panel.spec.Range { + for _, row := range axisChart(lead.values, panel.spec.Unit, trendStyle{width: chartWidth + metricsAxisWidth, rows: chartRows, fixedMax: panel.ceiling, mode: m.w.settings.GraphMode, bars: m.bars}) { + lines = append(lines, gradientText(row, hue, p.accent)) + } + lines = append(lines, "["+p.muted+"]"+tview.Escape(m.cardCaption(panel, chartWidth))+"[-]") + } else { + lines = append(lines, m.renderBars(panel, chartRows+1, chartWidth+metricsAxisWidth)...) + } + } + card.SetText(strings.Join(lines, "\n")) +} + +// headline is a panel's latest value, coloured by pressure when the value +// has a ceiling, with the series name and the direction of change. +func (m *metricsPage) headline(panel *metricsPanel, lead metricsSeries, width int) string { + p := m.w.palette() + colour := panelHue(p, metricsTab) + if panel.ceiling > 0 && lead.latest >= 0 { + colour = pressureHue(p, int(lead.latest/panel.ceiling*100)) + } + text := "[" + colour + "::b]" + formatMetric(lead.latest, panel.spec.Unit) + "[-::-]" + if name := seriesName(lead.label, panel.spec.Legend); name != "" { + text += " [" + p.text + "]" + tview.Escape(ellipsize(name, width/2)) + "[-]" + } + if panel.spec.Range { + text += " [" + p.muted + "]" + signalDirection(lead.values) + "[-]" + } + return text +} + +// seriesName shows a bare numeric label with its legend key ("cpu 6") and +// a metric-name legend without its exporter prefix ("load15"). +func seriesName(label, legend string) string { + label = textutil.Clean(label) + switch { + case label == "": + return "" + case legend == "__name__": + if _, short, ok := strings.Cut(label, "_"); ok { + return short + } + case !strings.Contains(legend, "/") && strings.Trim(label, "0123456789") == "": + return legend + " " + label + } + return label +} + +// panelEmptyText distinguishes a panel that has not been polled yet from one +// the server answered with nothing. +func panelEmptyText(panel *metricsPanel) string { + if panel.newest.IsZero() { + return "loading …" + } + return "no data" +} + +// cardCaption names the window and the other series a chart does not draw. +func (m *metricsPage) cardCaption(panel *metricsPanel, width int) string { + caption := trendCaption(panel.newest, panel.step, knownSpan(panel.series[0].values), width) + if len(panel.series) > 1 { + caption += " · +" + fmt.Sprint(len(panel.series)-1) + " more" + } + return ellipsize(caption, width) +} + +// renderBars draws one meter per series for instant-query panels. +func (m *metricsPage) renderBars(panel *metricsPanel, rows, width int) []string { + p := m.w.palette() + var lines []string + labelWidth := min(18, width/3) + for i, series := range panel.series { + if i >= rows { + lines[rows-1] = "[" + p.muted + "]+" + fmt.Sprint(len(panel.series)-rows+1) + " more[-]" + break + } + value := formatMetric(series.latest, panel.spec.Unit) + label := ellipsize(seriesName(series.label, panel.spec.Legend), labelWidth) + meterWidth := width - labelWidth - displayWidth(value) - 3 + colour := panelHue(p, metricsTab) + if panel.ceiling > 0 && series.latest >= 0 { + colour = pressureHue(p, int(series.latest/panel.ceiling*100)) + } + line := fmt.Sprintf("%-*s ", labelWidth, tview.Escape(label)) + if meterWidth >= 4 && panel.ceiling > 0 { + line += "[" + colour + "]" + signalMeter(max(0, series.latest), panel.ceiling, meterWidth, m.w.settings.GraphMode) + "[-] " + } + lines = append(lines, line+"["+colour+"]"+value+"[-]") + } + return lines +} + +func (m *metricsPage) renderTargets() { + p := m.w.palette() + m.table.Clear() + title := " TARGETS" + if m.build.Version != "" { + title += " · prometheus " + m.build.Version + } + up := 0 + for _, target := range m.targets { + if target.Health == "up" { + up++ + } + } + if len(m.targets) > 0 { + title += fmt.Sprintf(" · up %d/%d", up, len(m.targets)) + } + if !m.runtime.StartTime.IsZero() { + title += " · config " + if m.runtime.ReloadConfigSuccess { + title += "OK" + } else { + title += "FAILED" + } + } + m.table.SetTitle(tview.Escape(title) + " ") + headers := []string{"STATE", "JOB", "INSTANCE", "LAST SCRAPE", "DURATION", "ERROR"} + if m.width < 110 { + headers = []string{"STATE", "JOB", "INSTANCE", "ERROR"} + } + for col, name := range headers { + m.table.SetCell(0, col, tview.NewTableCell(name).SetTextColor(tcell.GetColor(p.muted)).SetSelectable(false).SetAttributes(tcell.AttrBold)) + } + rows := append([]promapi.Target(nil), m.targets...) + sortTargets(rows) + for i, target := range rows { + colour := p.muted + switch target.Health { + case "up": + colour = p.success + case "down": + colour = p.error + } + cells := map[string]string{ + "STATE": strings.ToUpper(target.Health), + "JOB": target.Job, + "INSTANCE": target.Instance, + "LAST SCRAPE": agoText(target.LastScrape), + "DURATION": formatMetric(target.ScrapeDuration.Seconds(), promapi.UnitSeconds), + "ERROR": ellipsize(target.LastError, 40), + } + for col, name := range headers { + cell := tview.NewTableCell(tview.Escape(cells[name])).SetTextColor(tcell.GetColor(p.text)) + if name == "STATE" || name == "ERROR" && target.LastError != "" { + cell.SetTextColor(tcell.GetColor(colour)) + } + if name == "DURATION" && target.ScrapeDuration > 5*time.Second { + cell.SetTextColor(tcell.GetColor(p.warning)) + } + m.table.SetCell(i+1, col, cell) + } + } + if row, _ := m.table.GetSelection(); row < 1 && len(rows) > 0 { + m.table.Select(1, 0) + } + m.renderTargetDetail() +} + +// sortTargets lists down targets first; Targets already sorted by job and instance. +func sortTargets(targets []promapi.Target) { + sort.SliceStable(targets, func(i, j int) bool { return targets[i].Health == "down" && targets[j].Health != "down" }) +} + +func agoText(at time.Time) string { + if at.IsZero() { + return "never" + } + return formatSeconds(time.Since(at).Seconds()) + " ago" +} + +func (m *metricsPage) currentTarget() (promapi.Target, bool) { + rows := append([]promapi.Target(nil), m.targets...) + sortTargets(rows) + row, _ := m.table.GetSelection() + if row < 1 || row > len(rows) { + return promapi.Target{}, false + } + return rows[row-1], true +} + +func (m *metricsPage) renderTargetDetail() { + p := m.w.palette() + target, ok := m.currentTarget() + if !ok { + m.detail.SetText("[" + p.muted + "]No scrape targets reported.[-]") + return + } + m.detail.SetText(targetRecord(target, p)) +} + +func targetRecord(target promapi.Target, p palette) string { + var out strings.Builder + line := func(key, value string) { + fmt.Fprintf(&out, "[%s]%-10s[-] %s\n", p.muted, key, tview.Escape(value)) + } + line("url", target.ScrapeURL) + line("health", target.Health) + line("scraped", agoText(target.LastScrape)+" · "+formatMetric(target.ScrapeDuration.Seconds(), promapi.UnitSeconds)+" · every "+shortDuration(target.ScrapeInterval)) + if target.LastError != "" { + line("error", target.LastError) + } + var labels []string + for key, value := range target.Labels { + labels = append(labels, key+"="+value) + } + sort.Strings(labels) + line("labels", strings.Join(labels, " ")) + return strings.TrimRight(out.String(), "\n") +} + +func (m *metricsPage) inspectTarget() { + if target, ok := m.currentTarget(); ok { + m.w.message("Target · "+target.Job+" · "+target.Instance, targetRecord(target, m.w.palette())) + } +} + +func (m *metricsPage) switchView(view metricsView) { + m.view = view + m.generation++ + m.busy = false + m.layout() + m.render() + m.focusPrimary() + m.lastPoll = time.Time{} +} + +// focusFamily jumps to an exporter's first panel, or to the targets view. +func (m *metricsPage) focusFamily(family string) { + if family == "targets" { + m.switchView(metricsTargetsView) + return + } + for i, panel := range m.panels { + if panel.family == family { + m.selectPanel(i) + return + } + } + m.failure = "" + m.render() +} + +func (m *metricsPage) selectPanel(i int) { + if len(m.panels) == 0 { + return + } + m.selected = max(0, min(len(m.panels)-1, i)) + row := m.selected / m.columns + if row < m.firstRow { + m.firstRow = row + } else if row >= m.firstRow+m.visibleRows { + m.firstRow = row - m.visibleRows + 1 + } + m.rebuildGrid() + m.render() + m.focusPrimary() +} + +func (m *metricsPage) input(e *tcell.EventKey) *tcell.EventKey { + w := m.w + switch e.Key() { + case tcell.KeyEscape: + if m.view == metricsTargetsView && w.app.GetFocus() == m.detail { + w.app.SetFocus(m.table) + return nil + } + m.close() + return nil + case tcell.KeyTab: + if m.view == metricsTargetsView { + if w.app.GetFocus() == m.table { + w.app.SetFocus(m.detail) + } else { + w.app.SetFocus(m.table) + } + return nil + } + case tcell.KeyEnter: + switch m.view { + case metricsPanelsView: + m.expand() + case metricsSetupView: + m.runSelectedStep() + default: + return e + } + return nil + case tcell.KeyUp, tcell.KeyDown, tcell.KeyLeft, tcell.KeyRight, tcell.KeyHome, tcell.KeyEnd: + if m.view == metricsPanelsView { + m.movePanel(e.Key()) + return nil + } + if m.view == metricsSetupView && (e.Key() == tcell.KeyUp || e.Key() == tcell.KeyDown) { + m.moveStep(e.Key()) + return nil + } + return e + } + switch e.Rune() { + case '0': + w.controlDeck() + case ',': + w.pollingDialog() + case '1', '2', '3', '4', '5': + m.close() + w.openHostTab(int(e.Rune() - '1')) + case '6': + m.focusPrimary() + case 'p': + m.switchView(metricsPanelsView) + case 't': + m.switchView(metricsTargetsView) + case 's': + m.switchView(metricsSetupView) + case 'a': + m.addMetric() + case 'd': + m.unpinSelected() + case 'b': + m.bars = !m.bars + m.render() + case 'r': + retryPrometheusProbe() + m.detectedAt = time.Time{} + m.refresh() + case 'P': + m.paused = !m.paused + m.updateStatus() + case 'G': + w.cycleGraphMode() + m.render() + case 'i': + m.chooseInstance() + case 'e': + w.reviewSnapshot(m.report()) + case 'j', 'k', 'h', 'l': + keys := map[rune]tcell.Key{'j': tcell.KeyDown, 'k': tcell.KeyUp, 'h': tcell.KeyLeft, 'l': tcell.KeyRight} + return m.input(tcell.NewEventKey(keys[e.Rune()], 0, tcell.ModNone)) + default: + return e + } + return nil +} + +func (m *metricsPage) movePanel(key tcell.Key) { + step := map[tcell.Key]int{tcell.KeyUp: -m.columns, tcell.KeyDown: m.columns, tcell.KeyLeft: -1, tcell.KeyRight: 1}[key] + switch key { + case tcell.KeyHome: + m.selectPanel(0) + case tcell.KeyEnd: + m.selectPanel(len(m.panels) - 1) + default: + m.selectPanel(m.selected + step) + } +} + +// chooseInstance lets the user pick which scraped host a family shows. +func (m *metricsPage) chooseInstance() { + if m.selected >= len(m.panels) { + return + } + panel := m.panels[m.selected] + candidates := m.instances[panel.family] + if panel.family == "pve" { + candidates = m.nodes + } + if len(candidates) < 2 { + m.w.message("One host", "Prometheus reports a single "+panel.family+" instance; nothing to choose.") + return + } + var choices []choice + for _, candidate := range candidates { + candidate := candidate + choices = append(choices, choice{candidate, "Show " + panel.family + " panels for this host", func() { + next := m.w.settings + next.PrometheusInstances = map[string]string{} + for k, v := range m.w.settings.PrometheusInstances { + next.PrometheusInstances[k] = v + } + next.PrometheusInstances[m.endpoint.URL+"|"+panel.family] = candidate + if m.w.persistViewSettings(next) { + m.generation++ + m.busy = false + m.refresh() + } + }}) + } + m.w.choose("metrics-instance", "Choose host · "+panel.family, choices) +} + +func (m *metricsPage) updateStatus() { + p := m.w.palette() + series := 0 + for _, panel := range m.panels { + series += len(panel.series) + } + state := "waiting for first sample" + if !m.sampledAt.IsZero() { + state = "sampled " + m.sampledAt.Format("15:04:05") + } + if m.busy { + state += " · refreshing" + } + if m.paused { + state += " · PAUSED" + } + source := "no endpoint" + if m.endpoint.URL != "" { + source = m.endpoint.URL + " (" + m.endpoint.Source + ")" + } + lead := fmt.Sprintf(" %d panels · %d series · %s · poll %ds · %s · ", len(m.panels), series, state, m.pollSeconds(), source) + note, colour := m.stateNote(), p.muted + if m.failure != "" { + note, colour = "Prometheus unreachable; previous sample retained: "+m.failure, p.error + } + note = ellipsize(textutil.Clean(strings.ReplaceAll(note, "\n", " · ")), max(0, m.width-displayWidth(lead)-1)) + m.status.SetText(fmt.Sprintf("%s[%s]%s[-]", tview.Escape(lead), colour, tview.Escape(note))) +} + +func (m *metricsPage) stateNote() string { + switch m.state { + case metricsUnconfigured: + return "No Prometheus configured or detected · s setup" + case metricsProbing: + return "Probing " + m.endpoint.URL + case metricsNoFamilies: + return "Reachable, but no node_exporter, cAdvisor or Proxmox series yet · s setup" + } + var names []string + for _, family := range m.families { + names = append(names, family.Title) + } + note := strings.Join(names, " · ") + if pins := len(m.w.settings.PrometheusMetrics); pins > 0 { + note += fmt.Sprintf(" · %d pinned", pins) + } + if m.hidden > 0 { + note += fmt.Sprintf(" · %d without data hidden", m.hidden) + } + if note == "" { + return "Ready" + } + return note +} + +func (m *metricsPage) updateFooter() { + text := " p panels · t targets · s setup · a pin · d unpin · Enter full screen · i host · b bars · r refresh · P pause · G graphics · , polling · e export · Esc back" + if m.width < 100 { + text = " p/t/s views · Enter full screen · i host · b bars · r · G · Esc" + } + m.footer.SetText(text) +} + +// report is the plain-text export of the current page. +func (m *metricsPage) report() string { + var out strings.Builder + fmt.Fprintf(&out, "Prometheus metrics · %s · %s\n\n", m.endpoint.URL, time.Now().Format(time.RFC3339)) + for _, panel := range m.panels { + fmt.Fprintf(&out, "%s · %s\n", strings.ToUpper(panel.family), panel.spec.Title) + for _, series := range panel.series { + fmt.Fprintf(&out, " %-32s %s\n", series.label, formatMetric(series.latest, panel.spec.Unit)) + } + } + out.WriteString("\nTARGETS\n") + for _, target := range m.targets { + fmt.Fprintf(&out, " %-6s %-16s %-28s %s\n", target.Health, target.Job, target.Instance, target.LastError) + } + return textutil.Clean(out.String()) +} diff --git a/internal/dashboard/metrics_ui_test.go b/internal/dashboard/metrics_ui_test.go new file mode 100644 index 0000000..5cee4cf --- /dev/null +++ b/internal/dashboard/metrics_ui_test.go @@ -0,0 +1,362 @@ +package dashboard + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/gdamore/tcell/v2" + + "systemdoc/internal/promapi" +) + +// sampleDetection is what a server with node_exporter looks like. +func sampleDetection() metricsDetection { + node, _ := promapi.FamilyByID("node") + names := map[string]bool{} + for _, name := range promapi.DetectNames() { + names[name] = strings.HasPrefix(name, "node_") + } + return metricsDetection{ + families: []promapi.Family{node}, + names: names, + instances: map[string][]string{"node": {"localhost:9100"}}, + targets: []promapi.Target{{Job: "node", Instance: "localhost:9100", Health: "up", ScrapeInterval: 15 * time.Second}, {Job: "pve", Instance: "pve:9221", Health: "down", LastError: "connection refused [red]x"}}, + interval: 15 * time.Second, + } +} + +func sampleResults(panels []*metricsPanel, columns int) []metricsResult { + var results []metricsResult + for _, panel := range panels { + values := make([]float64, columns) + for i := range values { + values[i] = float64(i % 100) + } + values[columns/2] = -1 + results = append(results, metricsResult{id: panel.spec.ID, ceiling: panel.spec.Ceiling, step: 15 * time.Second, newest: time.Now(), series: []metricsSeries{{label: "[red]cpu0", values: values, latest: 42}}}) + } + return results +} + +func metricsFixture(t *testing.T, width, height int) (*metricsPage, tcell.SimulationScreen) { + t.Helper() + t.Setenv("SYSTEMDOC_PROMETHEUS", "") + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) // saving preferences must never touch the real settings file + w := testWorkspace() + screen := tcell.NewSimulationScreen("UTF-8") + w.app.SetScreen(screen) + screen.SetSize(width, height) + w.settings.PrometheusURL = "http://example.invalid:9090" + m := w.metricsPage() + m.endpoint = w.prometheusEndpoint() + w.app.ForceDraw() + d := sampleDetection() + m.apply(true, "", &d, nil, nil) + w.app.ForceDraw() + m.apply(true, "", nil, nil, sampleResults(m.visiblePanels(), m.chartWidth())) + w.app.ForceDraw() + return m, screen +} + +func screenText(screen tcell.SimulationScreen) string { + width, height := screen.Size() + var out strings.Builder + for y := 0; y < height; y++ { + out.WriteString(screenRow(screen, y, width)) + out.WriteString("\n") + } + return out.String() +} + +func TestMetricsPageRendersAtThreeSizes(t *testing.T) { + for _, size := range [][3]int{{160, 44, 3}, {120, 30, 2}, {80, 24, 1}} { + m, screen := metricsFixture(t, size[0], size[1]) + if m.columns != size[2] || len(m.cards) == 0 { + t.Fatalf("%dx%d: columns %d cards %d", size[0], size[1], m.columns, len(m.cards)) + } + text := screenText(screen) + if !strings.Contains(text, "NODE · CPU busy") || !strings.Contains(text, "42.0%") { + t.Fatalf("%dx%d is missing panel content:\n%s", size[0], size[1], text) + } + if strings.Contains(text, "awaiting") { + t.Fatalf("%dx%d still shows the waiting state", size[0], size[1]) + } + m.close() + } +} + +func TestMetricsLabelValuesAreEscapedAndGapsDrawn(t *testing.T) { + m, screen := metricsFixture(t, 120, 30) + defer m.close() + text := screenText(screen) + if !strings.Contains(text, "[red]cpu0") { + t.Fatalf("label markup was interpreted instead of shown:\n%s", text) + } + if !strings.Contains(text, "·") { + t.Fatal("a missing sample must draw as a gap") + } + p := m.w.palette() + width, height := screen.Size() + for y := 0; y < height; y++ { + for x := 0; x < width; x++ { + r, _, style, _ := screen.GetContent(x, y) + fg, _, _ := style.Decompose() + if r == 'c' && fg == tcell.ColorRed && p.error != "#ff0000" { + t.Fatal("label text rendered in red, so markup leaked through") + } + } + } +} + +func TestMetricsGraphModesChangeGlyphs(t *testing.T) { + m, screen := metricsFixture(t, 120, 30) + defer m.close() + seen := map[string]string{} + for range graphModes { + seen[m.w.settings.GraphMode] = screenText(screen) + m.input(tcell.NewEventKey(tcell.KeyRune, 'G', 0)) + m.w.app.ForceDraw() + } + if len(seen) != 3 || seen["blocks"] == seen["ascii"] || !strings.Contains(seen["ascii"], "#") { + t.Fatalf("graph modes did not change the chart: %v", len(seen)) + } + m.input(tcell.NewEventKey(tcell.KeyRune, 'b', 0)) + m.w.app.ForceDraw() + if !m.bars { + t.Fatal("b must toggle bar glyphs") + } +} + +func TestMetricsEmptyStateShowsSetup(t *testing.T) { + t.Setenv("SYSTEMDOC_PROMETHEUS", "") + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + w := testWorkspace() + screen := tcell.NewSimulationScreen("UTF-8") + w.app.SetScreen(screen) + screen.SetSize(120, 30) + m := w.metricsPage() + defer m.close() + m.install = promapi.InstallSteps(promapi.InstallContext{GOOS: "linux", DistroID: "debian", PackageManager: "apt-get"}) + m.apply(false, "", nil, nil, nil) // an unreachable server routes to setup + m.state = metricsUnconfigured // the state refresh() reports without any URL + m.render() + w.app.ForceDraw() + if m.view != metricsSetupView { + t.Fatalf("view %d state %d", m.view, m.state) + } + text := screenText(screen) + for _, want := range []string{"CONNECTION", "No Prometheus URL", "settings.json", "apt-get install", "NODE EXPORTER"} { + if !strings.Contains(text, want) { + t.Fatalf("setup view lacks %q:\n%s", want, text) + } + } + m.input(tcell.NewEventKey(tcell.KeyEnter, 0, 0)) + if front, _ := w.pages.GetFrontPage(); front != "confirm" { + t.Fatalf("Enter must open the approval card, front page is %s", front) + } +} + +func TestMetricsTargetsViewAndNavigation(t *testing.T) { + m, screen := metricsFixture(t, 120, 30) + m.input(tcell.NewEventKey(tcell.KeyRune, 't', 0)) + m.w.app.ForceDraw() + text := screenText(screen) + if !strings.Contains(text, "UP") || !strings.Contains(text, "pve:9221") || !strings.Contains(text, "[red]x") || !strings.Contains(text, "url") { + t.Fatalf("targets view:\n%s", text) + } + if row, _ := m.table.GetSelection(); row != 1 || m.w.app.GetFocus() != m.table { + t.Fatalf("table focus/selection: row %d", row) + } + m.input(tcell.NewEventKey(tcell.KeyTab, 0, 0)) + if m.w.app.GetFocus() != m.detail { + t.Fatal("Tab must move to the detail pane") + } + m.input(tcell.NewEventKey(tcell.KeyEscape, 0, 0)) + if m.w.app.GetFocus() != m.table { + t.Fatal("Escape returns to the table") + } + m.input(tcell.NewEventKey(tcell.KeyRune, '4', 0)) + if front, _ := m.w.pages.GetFrontPage(); front != "processes" { + t.Fatalf("4 must open Processes, got %s", front) + } + _, page := m.w.pages.GetFrontPage() + page.(*hostPage).close() + m.w.openHostTab(5) + if front, _ := m.w.pages.GetFrontPage(); front != "metrics" { + t.Fatalf("suite 6 must open Metrics, got %s", front) + } + _, page = m.w.pages.GetFrontPage() + page.(*metricsPage).input(tcell.NewEventKey(tcell.KeyEscape, 0, 0)) + if front, _ := m.w.pages.GetFrontPage(); front != "main" || m.w.app.GetFocus() != m.w.table { + t.Fatalf("Escape must return to the dashboard, got %s", front) + } +} + +func TestMetricsPanelSelectionScrolls(t *testing.T) { + m, _ := metricsFixture(t, 80, 24) + defer m.close() + if m.visibleRows < 1 || len(m.panels) <= m.visibleRows { + t.Skip("not enough panels to scroll") + } + m.input(tcell.NewEventKey(tcell.KeyEnd, 0, 0)) + if m.selected != len(m.panels)-1 || m.firstRow == 0 || m.selectedCard() == nil { + t.Fatalf("End: selected %d firstRow %d", m.selected, m.firstRow) + } + m.input(tcell.NewEventKey(tcell.KeyHome, 0, 0)) + if m.selected != 0 || m.firstRow != 0 { + t.Fatalf("Home: selected %d firstRow %d", m.selected, m.firstRow) + } +} + +// fakePrometheus answers enough of the API for one node_exporter host. +func fakePrometheus(t *testing.T) *httptest.Server { + t.Helper() + vector := func(w http.ResponseWriter, value string) { + fmt.Fprintf(w, `{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"localhost:9100"},"value":[%d,"%s"]}]}}`, time.Now().Unix(), value) + } + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/v1/status/buildinfo": + fmt.Fprint(w, `{"status":"success","data":{"version":"2.53.1"}}`) + case "/api/v1/status/runtimeinfo": + fmt.Fprint(w, `{"status":"success","data":{"startTime":"2024-05-01T08:00:00Z","reloadConfigSuccess":true}}`) + case "/api/v1/label/__name__/values": + fmt.Fprint(w, `{"status":"success","data":["node_cpu_seconds_total","node_memory_MemTotal_bytes"]}`) + case "/api/v1/label/instance/values": + fmt.Fprint(w, `{"status":"success","data":["localhost:9100"]}`) + case "/api/v1/targets": + fmt.Fprint(w, `{"status":"success","data":{"activeTargets":[{"labels":{"job":"node","instance":"localhost:9100"},"health":"up","scrapeInterval":"15s","lastScrape":"2024-05-01T10:00:00Z"}]}}`) + case "/api/v1/query": + vector(w, "8") + case "/api/v1/query_range": + now := time.Now().Unix() + fmt.Fprintf(w, `{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"cpu":"0"},"values":[[%d,"12.5"],[%d,"37.5"]]}]}}`, now-15, now) + default: + http.NotFound(w, r) + } + })) +} + +func TestMetricsPageLiveAgainstFakeServer(t *testing.T) { + defer resetPrometheusState() + resetPrometheusState() + server := fakePrometheus(t) + defer server.Close() + t.Setenv("SYSTEMDOC_PROMETHEUS", server.URL) + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + ctx, cancel := context.WithCancel(context.Background()) + w := newWorkspace(ctx, false) + w.colourDepthKnown = true + screen := tcell.NewSimulationScreen("UTF-8") + w.app.SetScreen(screen) + screen.SetSize(120, 30) + done := make(chan error, 1) + go func() { done <- w.app.Run() }() + defer func() { cancel(); w.app.Stop(); <-done }() + var m *metricsPage + w.app.QueueUpdateDraw(func() { m = w.metricsPage() }) + deadline := time.Now().Add(5 * time.Second) + for { + var ready bool + var text string + w.app.QueueUpdateDraw(func() { + ready = m != nil && m.state == metricsReady && len(m.panels) > 0 && len(m.panels[0].series) > 0 + text = screenText(screen) + }) + if ready { + if !strings.Contains(text, "37.5%") || !strings.Contains(text, "NODE · CPU busy") { + t.Fatalf("live page:\n%s", text) + } + return + } + if time.Now().After(deadline) { + t.Fatalf("page never became ready; state %v failure %q", m.state, m.failure) + } + time.Sleep(20 * time.Millisecond) + } +} + +func TestMetricsPinPersistsAndUnpins(t *testing.T) { + m, screen := metricsFixture(t, 120, 30) + defer m.close() + pin := pinnedMetric{Title: "Open files", Name: "process_open_fds", Selector: `{job="prometheus"}`, Unit: promapi.UnitCount} + if err := m.savePin(pin); err != nil { + t.Fatal(err) + } + if err := m.savePin(pin); err == nil { + t.Fatal("a duplicate title must be rejected") + } + last := m.panels[len(m.panels)-1] + if !last.pinned || last.spec.Query != `process_open_fds{job="prometheus"}` || m.selected != len(m.panels)-1 { + t.Fatalf("pinned panel: %+v selected %d", last.spec, m.selected) + } + m.w.app.ForceDraw() + if text := screenText(screen); !strings.Contains(text, "PIN · Open files") { + t.Fatalf("pinned card not shown:\n%s", text) + } + saved, _ := readSettings() + if len(saved.PrometheusMetrics) != 1 || saved.PrometheusMetrics[0] != pin { + t.Fatalf("settings: %+v", saved.PrometheusMetrics) + } + m.input(tcell.NewEventKey(tcell.KeyRune, 'd', 0)) + if front, _ := m.w.pages.GetFrontPage(); front != "confirm" { + t.Fatalf("d must ask for approval, front page is %s", front) + } + m.w.pages.RemovePage("confirm") + m.removePin(pin) + saved, _ = readSettings() + if len(saved.PrometheusMetrics) != 0 || m.panels[len(m.panels)-1].pinned { + t.Fatal("unpin must remove the pin from settings and the page") + } +} + +func TestMetricsFullScreenFlicksBetweenPanels(t *testing.T) { + m, screen := metricsFixture(t, 120, 30) + defer m.close() + m.input(tcell.NewEventKey(tcell.KeyEnter, 0, 0)) + if m.focus == nil { + t.Fatal("Enter must open the full-screen view") + } + if front, _ := m.w.pages.GetFrontPage(); front != "metrics-focus" || m.w.app.GetFocus() != m.focus { + t.Fatalf("front page %s", front) + } + m.w.app.ForceDraw() + text := screenText(screen) + if !strings.Contains(text, "1 of") || !strings.Contains(text, "NODE · CPU busy") || !strings.Contains(text, "┤") { + t.Fatalf("full-screen view:\n%s", text) + } + if got := m.chartWidth(); got < 100 { + t.Fatalf("full screen must poll at full width, got %d columns", got) + } + if len(m.visiblePanels()) != 1 { + t.Fatal("only the shown panel is polled while full screen") + } + m.focusInput(tcell.NewEventKey(tcell.KeyRight, 0, 0)) + if m.selected != 1 { + t.Fatalf("Right must flick to the next panel, selected %d", m.selected) + } + m.focusInput(tcell.NewEventKey(tcell.KeyRune, 'h', 0)) + if m.selected != 0 { + t.Fatalf("h must flick back, selected %d", m.selected) + } + m.focusInput(tcell.NewEventKey(tcell.KeyEscape, 0, 0)) + if m.focus != nil || m.w.app.GetFocus() != m.selectedCard() { + t.Fatal("Escape must close the view and return to the card") + } +} + +func TestMetricsHidesPanelsWithoutData(t *testing.T) { + m, _ := metricsFixture(t, 120, 30) + defer m.close() + before := len(m.panels) + id := m.panels[0].spec.ID + m.apply(true, "", nil, nil, []metricsResult{{id: id}}) + if len(m.panels) != before-1 || m.hidden != 1 || m.panels[0].spec.ID == id { + t.Fatalf("empty panel must be hidden: %d → %d panels, hidden %d", before, len(m.panels), m.hidden) + } +} diff --git a/internal/dashboard/network_ui.go b/internal/dashboard/network_ui.go index 57ec757..cb4473e 100644 --- a/internal/dashboard/network_ui.go +++ b/internal/dashboard/network_ui.go @@ -31,7 +31,7 @@ type networkPage struct { summary *tview.Flex cards [3]*tview.TextView buttons [4]*tview.Button - modeButtons [5]*tview.Button + modeButtons [6]*tview.Button snapshot hostnet.Snapshot visible []hostnet.Socket interfaces []hostnet.Interface @@ -55,10 +55,7 @@ func (n *networkPage) Draw(screen tcell.Screen) { n.lastWidth, n.lastHeight = width, height setSuiteLabels(n.modeButtons[:], width, n.w.settings.NerdIcons) for i, button := range n.modeButtons { - buttonWidth := suiteButtonWidths[i] - if width < 100 { - buttonWidth = max(6, width/5) - } + buttonWidth := suiteButtonWidth(i, width) n.suiteNavigation.ResizeItem(button, buttonWidth, 0) } deckWidth := controlButtonWidth(n.deckButton.GetLabel()) @@ -220,7 +217,7 @@ func (w *workspace) networkPage() { n.updateStatus() case 'e': w.reviewSnapshot(n.report()) - case '1', '2', '4', '5': + case '1', '2', '4', '5', '6': n.close() w.openHostTab(int(e.Rune() - '1')) case 'p': @@ -847,3 +844,6 @@ func (n *networkPage) processDetails(s hostnet.Socket) { }) }() } + +func (n *networkPage) suiteIndex() int { return 2 } +func (n *networkPage) focusPrimary() { n.w.app.SetFocus(n.table) } diff --git a/internal/dashboard/preferences.go b/internal/dashboard/preferences.go index 6f12594..799a624 100644 --- a/internal/dashboard/preferences.go +++ b/internal/dashboard/preferences.go @@ -9,6 +9,8 @@ import ( "github.com/gdamore/tcell/v2" "github.com/rivo/tview" + + "systemdoc/internal/promapi" ) func (w *workspace) preferences() { @@ -28,7 +30,8 @@ func (w *workspace) preferences() { AddInputField("Accent override (#RRGGBB)", w.settings.Accent, 12, nil, nil). AddInputField("Background override (#RRGGBB)", w.settings.Background, 12, nil, nil). AddCheckbox("Nerd Font interface (recommended)", w.settings.NerdIcons, nil). - AddDropDown("Signal graphics", graphModes, graphModeIndex(w.settings.GraphMode), nil) + AddDropDown("Signal graphics", graphModes, graphModeIndex(w.settings.GraphMode), nil). + AddInputField("Prometheus URL (blank = detect)", w.settings.PrometheusURL, 44, nil, nil) form.SetBorder(true).SetTitle(" Preferences · saved under XDG config/systemdoc ") form.AddButton("Save", func() { seconds, err := strconv.Atoi(form.GetFormItem(0).(*tview.InputField).GetText()) @@ -61,7 +64,18 @@ func (w *workspace) preferences() { w.settings.WrapLogs = form.GetFormItem(3).(*tview.Checkbox).IsChecked() w.settings.Splash = form.GetFormItem(4).(*tview.Checkbox).IsChecked() w.settings.NerdIcons = form.GetFormItem(7).(*tview.Checkbox).IsChecked() + prometheusURL := strings.TrimSpace(form.GetFormItem(9).(*tview.InputField).GetText()) + if prometheusURL != "" { + if _, err := promapi.New(prometheusURL, ""); err != nil { + w.message("Invalid Prometheus URL", err.Error()) + return + } + } _, w.settings.GraphMode = form.GetFormItem(8).(*tview.DropDown).GetCurrentOption() + if prometheusURL != w.settings.PrometheusURL { + w.settings.PrometheusURL = prometheusURL + resetPrometheus() + } w.detail.SetWrap(w.settings.WrapLogs) w.lastWidth = 0 w.applyTheme() @@ -69,7 +83,7 @@ func (w *workspace) preferences() { w.savePreferences() w.app.SetFocus(w.table) }).AddButton("Cancel", func() { w.pages.RemovePage("preferences"); w.app.SetFocus(w.table) }) - w.pages.AddPage("preferences", centered(form, 88, 29), true, true) + w.pages.AddPage("preferences", centered(form, 88, 31), true, true) } func graphModeIndex(value string) int { @@ -102,11 +116,8 @@ func (w *workspace) updatePollButton() { if w.pages == nil { return } - if page, ok := w.pages.GetPage("network").(*networkPage); ok { - page.updateStatus() - } - for _, name := range []string{"processes", "storage"} { - if page, ok := w.pages.GetPage(name).(*hostPage); ok { + for _, name := range []string{"network", "processes", "storage", "metrics"} { + if page, ok := w.pages.GetPage(name).(interface{ updateStatus() }); ok { page.updateStatus() } } @@ -199,10 +210,10 @@ func (w *workspace) splashBodyText() string { return cached.text } eye := gradientText("─────────── "+w.icon(iconEye)+" ───────────", p.accent, p.glow) - text := fmt.Sprintf("%s\n%s\n\n%s\n[%s]SYSTEM OBSERVATORY[-]\n\n[%s]%s OBSERVE[-] · [%s]%s INSPECT[-] · [%s]%s ACT[-]\n\n[%s::b]01 %s SERVICES[-::-] [%s::b]02 %s CONTAINERS[-::-] [%s::b]03 %s NETWORK[-::-]\n[%s::b]04 %s PROCESSES[-::-] [%s::b]05 %s STORAGE[-::-]\n\n[%s::b]%s[-::-] Connecting to %s and Docker · polling every %ds\n[%s]PRESS ANY KEY TO ENTER[-]", + text := fmt.Sprintf("%s\n%s\n\n%s\n[%s]SYSTEM OBSERVATORY[-]\n\n[%s]%s OBSERVE[-] · [%s]%s INSPECT[-] · [%s]%s ACT[-]\n\n[%s::b]01 %s SERVICES[-::-] [%s::b]02 %s CONTAINERS[-::-] [%s::b]03 %s NETWORK[-::-]\n[%s::b]04 %s PROCESSES[-::-] [%s::b]05 %s STORAGE[-::-] [%s::b]06 %s METRICS[-::-]\n\n[%s::b]%s[-::-] Connecting to %s and Docker · polling every %ds\n[%s]PRESS ANY KEY TO ENTER[-]", wordmarkBlock("SYSTEMDOC", p.accent, p.glow), eye, gradientText("S Y S T E M D O C", p.glow, p.accent), p.muted, p.accent, w.icon(iconEye), p.glow, w.icon(iconSearch), p.warning, w.icon(iconActions), - p.success, w.icon(iconServices), p.accent, w.icon(iconContainers), p.glow, w.icon(iconNetwork), p.warning, w.icon(iconProcesses), p.success, w.icon(iconStorage), + p.success, w.icon(iconServices), p.accent, w.icon(iconContainers), p.glow, w.icon(iconNetwork), p.warning, w.icon(iconProcesses), p.success, w.icon(iconStorage), p.accent, w.icon(iconMetrics), p.accent, splashSpinnerSlot, tview.Escape(key.manager), key.refresh, p.muted) splashCache.Store(&splashBody{key: key, text: text}) return text diff --git a/internal/dashboard/prometheus.go b/internal/dashboard/prometheus.go new file mode 100644 index 0000000..1f6b217 --- /dev/null +++ b/internal/dashboard/prometheus.go @@ -0,0 +1,282 @@ +package dashboard + +import ( + "context" + "fmt" + "os" + "regexp" + "strings" + "sync" + "time" + "unicode/utf8" + + "systemdoc/internal/promapi" +) + +// maxPinnedMetrics bounds the user's own panels so settings stay small. +const maxPinnedMetrics = 64 + +// pinnedMetric is a user-chosen metric shown as its own panel in the Metrics +// suite. Name and Selector are kept apart so the picker can validate each. +type pinnedMetric struct { + Title string `json:"title"` + Name string `json:"name"` + Selector string `json:"selector,omitempty"` // {job="node"} style + Unit string `json:"unit,omitempty"` + Rate bool `json:"rate,omitempty"` + Ceiling float64 `json:"ceiling,omitempty"` +} + +var pinnedSelector = regexp.MustCompile(`^\{[^{}\n]*\}$`) + +func (m pinnedMetric) validate() error { + if !promapi.ValidMetricName(m.Name) { + return fmt.Errorf("metric name %q is not valid", m.Name) + } + if m.Selector != "" && !pinnedSelector.MatchString(m.Selector) { + return fmt.Errorf("selector must look like {job=\"node\"}") + } + if title := strings.TrimSpace(m.Title); title == "" || utf8.RuneCountInString(title) > 40 { + return fmt.Errorf("title must be 1–40 characters") + } + if m.Ceiling < 0 { + return fmt.Errorf("ceiling cannot be negative") + } + return nil +} + +// expr is the PromQL for the pin; $win is filled in by promapi.Expand. +func (m pinnedMetric) expr() string { + expr := m.Name + m.Selector + if m.Rate { + return "rate(" + expr + "[$win])" + } + return expr +} + +// promEndpoint is a resolved Prometheus URL and where it came from, so the +// Metrics suite can say "settings" or "detected: prometheus.service". +type promEndpoint struct { + URL string + Source string +} + +// prometheusEndpoint runs on the UI loop. An "Open in Metrics" choice wins, +// then the flag or environment, then settings, then the inventory, and +// finally the conventional local port, which the probe will report as +// unreachable when nothing listens there. +func (w *workspace) prometheusEndpoint() promEndpoint { + if w.promSelected.URL != "" { + return w.promSelected + } + if url := strings.TrimSpace(os.Getenv("SYSTEMDOC_PROMETHEUS")); url != "" { + return promEndpoint{URL: url, Source: "flag or environment"} + } + if w.settings.PrometheusURL != "" { + return promEndpoint{URL: w.settings.PrometheusURL, Source: "settings"} + } + if endpoint, ok := discoverPrometheus(w.items[0], w.items[1]); ok { + return endpoint + } + return promEndpoint{URL: "http://localhost:9090", Source: "default"} +} + +// discoverPrometheus looks for a running Prometheus in the inventory the +// suites already collected; it never runs a command. +func discoverPrometheus(services, containers []workload) (promEndpoint, bool) { + for _, item := range services { + if endpoint, ok := prometheusWorkload(0, item); ok { + return endpoint, true + } + } + for _, item := range containers { + if endpoint, ok := prometheusWorkload(1, item); ok { + return endpoint, true + } + } + return promEndpoint{}, false +} + +// prometheusWorkload reports whether one row is a running Prometheus server +// and the URL it answers on. +func prometheusWorkload(mode int, item workload) (promEndpoint, bool) { + if !isActive(item) { + return promEndpoint{}, false + } + detected := promEndpoint{URL: "http://localhost:9090", Source: "detected: " + item.Name} + if mode == 0 { + switch { + case item.ID == "prometheus.service", strings.HasSuffix(item.ID, ".prometheus.service"): + return detected, true + case usesLaunchd() && (item.ID == "sh.brew.prometheus" || item.ID == "homebrew.mxcl.prometheus") && item.PID > 0: + return detected, true + } + return promEndpoint{}, false + } + if isPod(item) || imageBase(item.Description) != "prometheus" { + return promEndpoint{}, false + } + if item.Ports == "" { + return detected, true // host networking + } + host, port, ok := publishedPort(item.Ports, "9090/tcp") + if !ok { + return promEndpoint{}, false + } + detected.URL = "http://" + host + ":" + port + return detected, true +} + +// metricsFamilyFor names the exporter family a workload belongs to, so the +// Actions menu can jump to its panels. +func metricsFamilyFor(item workload) (string, bool) { + text := strings.ToLower(item.Name + " " + item.ID + " " + item.Description) + switch { + case strings.Contains(text, "node-exporter"), strings.Contains(text, "node_exporter"): + return "node", true + case strings.Contains(text, "cadvisor"): + return "cadvisor", true + case strings.Contains(text, "pve-exporter"), strings.Contains(text, "pve_exporter"): + return "pve", true + case strings.Contains(text, "prometheus"): + return "targets", true + } + return "", false +} + +// imageBase returns the last path element of an image reference without its +// tag or digest: quay.io/prometheus/prometheus:v2.51 → prometheus. +func imageBase(image string) string { + image = strings.TrimSpace(image) + if at := strings.Index(image, "@"); at >= 0 { + image = image[:at] + } + if slash := strings.LastIndex(image, "/"); slash >= 0 { + image = image[slash+1:] + } + if colon := strings.Index(image, ":"); colon >= 0 { + image = image[:colon] + } + return image +} + +// publishedPort finds the host side of a docker ps publish list for one +// container port. Wildcard binds map to localhost. +func publishedPort(ports, containerPort string) (host, port string, ok bool) { + for _, mapping := range strings.Split(ports, ",") { + mapping = strings.TrimSpace(mapping) + hostSide, target, found := strings.Cut(mapping, "->") + if !found || target != containerPort { + continue + } + colon := strings.LastIndex(hostSide, ":") + if colon < 0 { + continue + } + host, port = strings.Trim(hostSide[:colon], "[]"), hostSide[colon+1:] + if host == "0.0.0.0" || host == "::" || host == "" { + host = "localhost" + } + return host, port, true + } + return "", "", false +} + +// Probe state is shared by page goroutines and the UI thread, like kube. +var prom struct { + mu sync.Mutex + client *promapi.Client + url string + build promapi.BuildInfo + err string + probed time.Time + failures int +} + +// currentPrometheus returns a client for endpoint, probing the server's build +// info when nothing is cached. Failed probes back off like the kubectl probe +// (30 s doubling to 10 min); a changed URL is probed at once. +func currentPrometheus(ctx context.Context, endpoint promEndpoint) (*promapi.Client, promapi.BuildInfo, string) { + prom.mu.Lock() + if prom.url != endpoint.URL { + prom.client, prom.url, prom.build, prom.err, prom.probed, prom.failures = nil, endpoint.URL, promapi.BuildInfo{}, "", time.Time{}, 0 + } + if prom.client != nil { + client, build := prom.client, prom.build + prom.mu.Unlock() + return client, build, "" + } + wait := min(time.Duration(30<9090/tcp, :::9091->9090/tcp"}, "http://localhost:9091"}, + {"bound address", 1, workload{ID: "abc", State: "running", Description: "prom/prometheus", Ports: "192.168.1.5:9090->9090/tcp"}, "http://192.168.1.5:9090"}, + {"host network", 1, workload{ID: "abc", State: "running", Description: "bitnami/prometheus@sha256:abcd"}, "http://localhost:9090"}, + {"other port only", 1, workload{ID: "abc", State: "running", Description: "prom/prometheus", Ports: "0.0.0.0:8080->8080/tcp"}, ""}, + {"node exporter image", 1, workload{ID: "abc", State: "running", Description: "quay.io/prometheus/node-exporter"}, ""}, + {"pve exporter image", 1, workload{ID: "abc", State: "running", Description: "prompve/prometheus-pve-exporter"}, ""}, + {"stopped container", 1, workload{ID: "abc", State: "exited", Description: "prom/prometheus"}, ""}, + {"pod never matches", 1, workload{ID: podPrefix + "monitoring/prometheus-0", State: "running", Description: "prom/prometheus"}, ""}, + } + for _, c := range cases { + endpoint, ok := prometheusWorkload(c.mode, c.item) + if ok != (c.url != "") || endpoint.URL != c.url { + t.Errorf("%s: want %q, got %q (%v)", c.name, c.url, endpoint.URL, ok) + } + if ok && !strings.HasPrefix(endpoint.Source, "detected: ") { + t.Errorf("%s: source %q", c.name, endpoint.Source) + } + } + endpoint, ok := discoverPrometheus([]workload{{ID: "sshd.service", State: "active"}}, []workload{{ID: "x", State: "running", Description: "prom/prometheus"}}) + if !ok || endpoint.URL != "http://localhost:9090" { + t.Errorf("discovery across inventories: %+v %v", endpoint, ok) + } +} + +func TestMetricsFamilyFor(t *testing.T) { + cases := map[string]workload{ + "node": {ID: "prometheus-node-exporter.service"}, + "cadvisor": {ID: "abc", Description: "ghcr.io/google/cadvisor:v0.52.1"}, + "pve": {ID: "abc", Name: "pve-exporter"}, + "targets": {ID: "prometheus.service"}, + "": {ID: "nginx.service", Description: "nginx"}, + } + for want, item := range cases { + family, ok := metricsFamilyFor(item) + if family != want || ok != (want != "") { + t.Errorf("%+v: want %q, got %q", item, want, family) + } + } +} + +func TestPublishedPortAndImageBase(t *testing.T) { + if host, port, ok := publishedPort("[::]:9090->9090/tcp", "9090/tcp"); !ok || host != "localhost" || port != "9090" { + t.Errorf("ipv6 wildcard: %s %s %v", host, port, ok) + } + if _, _, ok := publishedPort("9090/tcp", "9090/tcp"); ok { + t.Error("exposed-only port must not count as published") + } + for image, want := range map[string]string{"prometheus": "prometheus", "prom/prometheus:latest": "prometheus", "quay.io/prometheus/prometheus@sha256:ff": "prometheus", "localhost:5000/team/prometheus:v1": "prometheus"} { + if got := imageBase(image); got != want { + t.Errorf("%s: got %q", image, got) + } + } +} + +func TestPrometheusEndpointPrecedence(t *testing.T) { + t.Setenv("SYSTEMDOC_PROMETHEUS", "") + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + w := testWorkspace() + if got := w.prometheusEndpoint(); got.Source != "default" || got.URL != "http://localhost:9090" { + t.Errorf("default: %+v", got) + } + w.items[1] = []workload{{ID: "x", State: "running", Description: "prom/prometheus"}} + if got := w.prometheusEndpoint(); got.URL != "http://localhost:9090" || !strings.HasPrefix(got.Source, "detected") { + t.Errorf("discovery: %+v", got) + } + w.settings.PrometheusURL = "http://settings:9090" + if got := w.prometheusEndpoint(); got.Source != "settings" { + t.Errorf("settings: %+v", got) + } + t.Setenv("SYSTEMDOC_PROMETHEUS", "http://env:9090") + if got := w.prometheusEndpoint(); got.URL != "http://env:9090" { + t.Errorf("environment: %+v", got) + } + w.promSelected = promEndpoint{URL: "http://chosen:9090", Source: "detected: prom"} + if got := w.prometheusEndpoint(); got.URL != "http://chosen:9090" { + t.Errorf("selection: %+v", got) + } +} + +func TestPrometheusProbeBackoff(t *testing.T) { + defer resetPrometheusState() + resetPrometheusState() + hits, status := 0, http.StatusInternalServerError + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits++ + w.WriteHeader(status) + w.Write([]byte(`{"status":"success","data":{"version":"2.53.1"}}`)) + })) + defer server.Close() + ctx := context.Background() + endpoint := promEndpoint{URL: server.URL} + if client, _, err := currentPrometheus(ctx, endpoint); client != nil || err == "" { + t.Fatalf("failed probe: client %v err %q", client, err) + } + currentPrometheus(ctx, endpoint) + if hits != 1 { + t.Fatalf("back-off ignored: %d hits", hits) + } + retryPrometheusProbe() + currentPrometheus(ctx, endpoint) + if hits != 2 { + t.Fatalf("explicit retry: %d hits", hits) + } + status = http.StatusOK + currentPrometheus(ctx, promEndpoint{URL: server.URL + "/"}) // a changed URL is probed at once + client, build, err := currentPrometheus(ctx, promEndpoint{URL: server.URL + "/"}) + if hits != 3 || client == nil || build.Version != "2.53.1" || err != "" { + t.Fatalf("success cached: hits %d client %v build %+v err %q", hits, client, build, err) + } + if _, _, ok := prometheusStatus(); !ok { + t.Error("status must report the cached client") + } +} + +func TestPrometheusTokenNeverPersisted(t *testing.T) { + dir := t.TempDir() + t.Setenv("XDG_CONFIG_HOME", dir) + t.Setenv("SYSTEMDOC_PROMETHEUS_TOKEN", "super-secret") + if err := writeSettings(settings{SettingsVersion: 1, PrometheusURL: "http://localhost:9090", PrometheusMetrics: []pinnedMetric{{Title: "Load", Name: "node_load1"}}}); err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(filepath.Join(dir, "systemdoc", "settings.json")) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(data), "super-secret") || !strings.Contains(string(data), `"prometheus_url": "http://localhost:9090"`) || !strings.Contains(string(data), `"node_load1"`) { + t.Errorf("settings file: %s", data) + } +} + +func TestPinnedMetricValidate(t *testing.T) { + good := pinnedMetric{Title: "Load", Name: "node_load1", Selector: `{job="node"}`, Rate: true} + if err := good.validate(); err != nil { + t.Fatal(err) + } + if got := good.expr(); got != `rate(node_load1{job="node"}[$win])` { + t.Errorf("expr: %s", got) + } + for name, bad := range map[string]pinnedMetric{ + "name": {Title: "x", Name: "1bad"}, + "selector": {Title: "x", Name: "ok", Selector: `job="node"`}, + "title": {Name: "ok"}, + "ceiling": {Title: "x", Name: "ok", Ceiling: -1}, + } { + if bad.validate() == nil { + t.Errorf("%s must be rejected", name) + } + } +} + +func TestListDockerContainersKeepsPorts(t *testing.T) { + dir := t.TempDir() + script := "#!/bin/sh\nprintf '%s\\n' '{\"ID\":\"abc\",\"Names\":\"prom\",\"State\":\"running\",\"Status\":\"Up\",\"Image\":\"prom/prometheus\",\"Ports\":\"0.0.0.0:9090->9090/tcp\"}'\n" + if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(script), 0700); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", dir) + items, err := listDockerContainers(context.Background()) + if err != nil || len(items) != 1 || items[0].Ports != "0.0.0.0:9090->9090/tcp" { + t.Fatalf("items %+v err %v", items, err) + } +} + +func TestActionsOfferOpenInMetrics(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + w := testWorkspace() + w.mode = 1 + w.visible = []workload{{ID: "abc", Name: "prom", State: "running", Description: "prom/prometheus"}} + w.selected[1] = "abc" + screen := tcell.NewSimulationScreen("UTF-8") + w.app.SetScreen(screen) + screen.SetSize(120, 40) + w.actions() + for _, r := range "Open in Metrics" { + w.app.GetFocus().InputHandler()(tcell.NewEventKey(tcell.KeyRune, r, 0), func(tview.Primitive) {}) + } + w.app.ForceDraw() + var text strings.Builder + for y := 0; y < 40; y++ { + text.WriteString(screenRow(screen, y, 120)) + } + if !strings.Contains(text.String(), "Open in Metrics") || !strings.Contains(text.String(), "localhost:9090") { + t.Fatalf("actions menu lacks the Prometheus entry:\n%s", text.String()) + } +} diff --git a/internal/dashboard/refresh.go b/internal/dashboard/refresh.go index 3d62d12..3d1d1a3 100644 --- a/internal/dashboard/refresh.go +++ b/internal/dashboard/refresh.go @@ -25,6 +25,7 @@ func (w *workspace) load() { if w.mode == 1 { retryKubeProbe() } + retryPrometheusProbe() invalidateUnitFiles() w.startInventory(w.mode) } diff --git a/internal/dashboard/remote.go b/internal/dashboard/remote.go index ff79b4f..7236017 100644 --- a/internal/dashboard/remote.go +++ b/internal/dashboard/remote.go @@ -42,6 +42,12 @@ func (w *workspace) remoteDialog() { if userScope.IsChecked() { options.Args = []string{"--user"} } + // A URL from the flag or environment names the remote host's own + // loopback once it runs there; a settings value stays local because the + // remote user has their own settings file. + if endpoint := strings.TrimSpace(os.Getenv("SYSTEMDOC_PROMETHEUS")); endpoint != "" { + options.Args = append(options.Args, "--prometheus", endpoint) + } if err := options.Validate(); err != nil { status.SetText(" " + err.Error()).SetTextColor(tcell.GetColor(w.palette().error)) return diff --git a/internal/dashboard/settings.go b/internal/dashboard/settings.go index c11b47e..1c28a9a 100644 --- a/internal/dashboard/settings.go +++ b/internal/dashboard/settings.go @@ -5,25 +5,29 @@ import ( "errors" "os" "path/filepath" + "strings" ) type settings struct { - SettingsVersion int `json:"settings_version,omitempty"` - Views []savedView `json:"saved_views,omitempty"` - SystemdActiveOnly bool `json:"systemd_active_only"` - SSHConnections map[string]sshPreference `json:"ssh_connections,omitempty"` - Theme string `json:"theme"` - RefreshSeconds int `json:"refresh_seconds"` - Projects []project `json:"projects,omitempty"` - Favorites []string `json:"favorites,omitempty"` - Layout string `json:"layout,omitempty"` - PaneRatio int `json:"pane_ratio,omitempty"` - WrapLogs bool `json:"wrap_logs"` - Splash bool `json:"splash"` - NerdIcons bool `json:"nerd_icons"` - Accent string `json:"accent,omitempty"` - Background string `json:"background,omitempty"` - GraphMode string `json:"graph_mode,omitempty"` + SettingsVersion int `json:"settings_version,omitempty"` + Views []savedView `json:"saved_views,omitempty"` + SystemdActiveOnly bool `json:"systemd_active_only"` + SSHConnections map[string]sshPreference `json:"ssh_connections,omitempty"` + Theme string `json:"theme"` + RefreshSeconds int `json:"refresh_seconds"` + Projects []project `json:"projects,omitempty"` + Favorites []string `json:"favorites,omitempty"` + Layout string `json:"layout,omitempty"` + PaneRatio int `json:"pane_ratio,omitempty"` + WrapLogs bool `json:"wrap_logs"` + Splash bool `json:"splash"` + NerdIcons bool `json:"nerd_icons"` + Accent string `json:"accent,omitempty"` + Background string `json:"background,omitempty"` + GraphMode string `json:"graph_mode,omitempty"` + PrometheusURL string `json:"prometheus_url,omitempty"` + PrometheusMetrics []pinnedMetric `json:"prometheus_metrics,omitempty"` + PrometheusInstances map[string]string `json:"prometheus_instances,omitempty"` } func settingsPath() (string, error) { @@ -68,6 +72,10 @@ func readSettings() (settings, error) { } s.Theme = themes[themeIndex(s.Theme)].name s.GraphMode = graphMode(s.GraphMode) + s.PrometheusURL = strings.TrimSpace(s.PrometheusURL) + if len(s.PrometheusMetrics) > maxPinnedMetrics { + s.PrometheusMetrics = s.PrometheusMetrics[:maxPinnedMetrics] + } return s, nil } diff --git a/internal/dashboard/ui.go b/internal/dashboard/ui.go index 65bc98c..3107cae 100644 --- a/internal/dashboard/ui.go +++ b/internal/dashboard/ui.go @@ -100,6 +100,7 @@ type workspace struct { commandHistory []string workloadHistory map[string][]workloadSignal hostLabel string + promSelected promEndpoint // Prometheus chosen with "Open in Metrics"; wins over settings } type Options struct { @@ -108,6 +109,7 @@ type Options struct { Filter string ActiveOnly bool Kubectl string + Prometheus string } func Run(options Options) error { @@ -122,6 +124,9 @@ func Run(options Options) error { if strings.TrimSpace(options.Kubectl) != "" { os.Setenv("SYSTEMDOC_KUBECTL", strings.TrimSpace(options.Kubectl)) } + if strings.TrimSpace(options.Prometheus) != "" { + os.Setenv("SYSTEMDOC_PROMETHEUS", strings.TrimSpace(options.Prometheus)) + } w := newWorkspace(ctx, options.User) if options.Docker { w.mode = 1 @@ -747,8 +752,8 @@ func (w *workspace) input(event *tcell.EventKey) *tcell.EventKey { } case '3': w.networkPage() - case '4', '5': - w.hostPage(int(event.Rune() - '1')) + case '4', '5', '6': + w.openHostTab(int(event.Rune() - '1')) case '1', '2': w.switchMode(int(event.Rune() - '1')) case 'z': @@ -836,7 +841,7 @@ func (w *workspace) input(event *tcell.EventKey) *tcell.EventKey { case 'a': w.actions() case '?': - w.message("Welcome to Systemdoc", "Five areas. One workspace.\n\n0 opens the Control Deck · 1 / 2 services and containers · 3 network · 4 processes · 5 storage\n/ filters names, states and projects · Enter inspects · Tab switches panes\nx maps the selected workload constellation · I opens the incident storyline · C opens CPU cores · G cycles block/braille/ASCII graphics\nl logs · h retained log history · L live log drawer · c configuration · r resources · o overview\nR reviews restart · t previews themes · a searches actions · q quits\nS sort · i active only · A cycle state filter · z expand focused pane · [ / ] resize split · Escape restores\nClick Active / Attention cards to filter; click again to clear\n\nLive data refreshes automatically · , changes the polling interval.\nV saved views · T systemd timers · E troubleshooting snapshot\ns richer log search · p Compose projects · H operation history\nf favorite · F favorites only · u system/user scope\n: supported commands · Actions includes lifecycle and native tools.") + w.message("Welcome to Systemdoc", "Six areas. One workspace.\n\n0 opens the Control Deck · 1 / 2 services and containers · 3 network · 4 processes · 5 storage · 6 metrics\n/ filters names, states and projects · Enter inspects · Tab switches panes\nx maps the selected workload constellation · I opens the incident storyline · C opens CPU cores · G cycles block/braille/ASCII graphics\nl logs · h retained log history · L live log drawer · c configuration · r resources · o overview\nR reviews restart · t previews themes · a searches actions · q quits\nS sort · i active only · A cycle state filter · z expand focused pane · [ / ] resize split · Escape restores\nClick Active / Attention cards to filter; click again to clear\n\nLive data refreshes automatically · , changes the polling interval.\nV saved views · T systemd timers · E troubleshooting snapshot\ns richer log search · p Compose projects · H operation history\nf favorite · F favorites only · u system/user scope\n: supported commands · Actions includes lifecycle and native tools.") case 'j': return tcell.NewEventKey(tcell.KeyDown, 0, tcell.ModNone) case 'k': diff --git a/internal/dashboard/visual_check_test.go b/internal/dashboard/visual_check_test.go index 4ef4182..6de3649 100644 --- a/internal/dashboard/visual_check_test.go +++ b/internal/dashboard/visual_check_test.go @@ -11,6 +11,7 @@ import ( "systemdoc/internal/dockerinspect" "systemdoc/internal/hostnet" + "systemdoc/internal/promapi" ) // This opt-in test exports terminal cells for visual review, without adding any @@ -119,6 +120,19 @@ func TestVisualReview(t *testing.T) { h.at[0] = time.Now() h.render() } + if mode := os.Getenv("SYSTEMDOC_VISUAL_MODE"); mode == "metrics" || mode == "metrics-setup" { + w.settings.PrometheusURL = "http://localhost:9090" + m := w.metricsPage() + m.endpoint = w.prometheusEndpoint() + if mode == "metrics" { + d := sampleDetection() + m.apply(true, "", &d, nil, nil) + m.apply(true, "", nil, nil, sampleResults(m.visiblePanels(), 48)) + } else { + m.install = promapi.InstallSteps(promapi.InstallContext{GOOS: "linux", DistroID: "debian", PackageManager: "apt-get", Docker: true}) + m.apply(false, "connect: connection refused", nil, nil, nil) + } + } if os.Getenv("SYSTEMDOC_VISUAL_MODE") == "constellation" { body := constellationText(w.current(), 0, "nginx.service\nnetwork-online.target\npostgresql.service\nredis.service\nsystem.slice\n", w.settings.NerdIcons, w.inventoryStates()) view := textView().SetDynamicColors(true).SetScrollable(true).SetWrap(false) diff --git a/internal/dashboard/workflows.go b/internal/dashboard/workflows.go index 432470e..9c000b2 100644 --- a/internal/dashboard/workflows.go +++ b/internal/dashboard/workflows.go @@ -146,6 +146,7 @@ func (w *workspace) actions() { add("Ports and networking", "Find port owners, connections and host interfaces", w.networkPage) add("Process explorer", "CPU, memory, parent trees and process ports", func() { w.hostPage(3) }) add("Disk and storage", "Mount usage, inode pressure and deleted open files", func() { w.hostPage(4) }) + add("Metrics", "Prometheus trends, pinned metrics, scrape targets and setup", func() { w.openMetrics(promEndpoint{}, "") }) add("Saved views", "Save and restore filters, sorting and layout", w.savedViews) if !usesLaunchd() { add("Systemd timers", "Browse next and last runs in the current system/user scope", w.timers) @@ -211,6 +212,11 @@ func (w *workspace) actions() { add("Journal: since this boot · "+bootState, "Toggle journalctl --boot so older boots are left out", func() { w.toggleLogFilter(false) }) } } + if endpoint, ok := prometheusWorkload(w.mode, item); ok { + add("Open in Metrics · "+item.Name, "Prometheus at "+endpoint.URL+" · suite 6", func() { w.openMetrics(endpoint, "targets") }) + } else if family, ok := metricsFamilyFor(item); ok && item.ID != "" { + add("Open in Metrics · "+item.Name, "Jump to this exporter's panels in suite 6", func() { w.openMetrics(promEndpoint{}, family) }) + } if w.mode == 1 && item.ID != "" { add("Trace with sysdig", "Syscalls, files, connections and errors of this workload · needs sudo", func() { w.sysdigForWorkload(item) }) } diff --git a/internal/promapi/README.md b/internal/promapi/README.md new file mode 100644 index 0000000..fd2db13 --- /dev/null +++ b/internal/promapi/README.md @@ -0,0 +1,20 @@ +# Prometheus API client and panel catalog + +`New(raw, token)` validates a Prometheus base URL and returns a client whose +requests are bounded: eight seconds per request, 4 MiB per body, no redirects, +system TLS roots. `Ready`, `BuildInfo`, `RuntimeInfo`, `Query`, `QueryRange`, +`MetricNames`, `LabelValues` and `Targets` decode `/api/v1` envelopes into +plain values. Label values and server messages pass through `textutil.Clean`; +the UI still escapes terminal markup. NaN and infinite samples are reported as +unknown. A bearer token is sent as a header and never appears in an error or +in `String()`. A username or password inside the URL is rejected. + +`Catalog` lists the built-in exporter families (node_exporter, cAdvisor, +kubelet, Proxmox VE exporter, Prometheus itself) with their PromQL templates; +`Expand` fills the `$sel`, `$win`, `$node` and `$top` placeholders. +`InstallSteps` returns per-platform install guidance with runnable steps +marked separately from steps that only show a command or a file. + +Run `go test ./internal/promapi`. Fixtures live in `testdata/`. The package +depends only on the standard library and `internal/textutil`; it must not +import `dashboard`, tview or tcell. diff --git a/internal/promapi/catalog.go b/internal/promapi/catalog.go new file mode 100644 index 0000000..79510b8 --- /dev/null +++ b/internal/promapi/catalog.go @@ -0,0 +1,495 @@ +package promapi + +import ( + "strconv" + "strings" + "time" +) + +// PanelKind selects how the dashboard renders a panel. +type PanelKind int + +// Panel kinds. +const ( + // KindArea is a filled single-series chart over a range query. + KindArea PanelKind = iota + // KindSpark is a multi-series line chart over a range query. + KindSpark + // KindBars is a horizontal bar list from an instant query. + KindBars + // KindStat is a single value or short value list from an instant query. + KindStat +) + +// String returns the kind name used in logs and tests. +func (k PanelKind) String() string { + switch k { + case KindArea: + return "area" + case KindSpark: + return "spark" + case KindBars: + return "bars" + case KindStat: + return "stat" + } + return "unknown" +} + +// Scope describes which selector a family needs before its panels make sense. +type Scope int + +// Family scopes. +const ( + // ScopeNone means the family is server-wide and ignores selectors. + ScopeNone Scope = iota + // ScopeInstance means panels are filtered by the instance label. + ScopeInstance + // ScopePVENode means panels are filtered by instance and a PVE node name. + ScopePVENode +) + +// Units attached to panels. The dashboard formats values by unit. +const ( + UnitPercent = "%" + UnitBytes = "B" + UnitBytesPerSecond = "B/s" + UnitSeconds = "s" + UnitLoad = "load" + UnitCelsius = "°C" + UnitCount = "count" + UnitPerSecond = "/s" + UnitNone = "" +) + +// Family groups panels that depend on the same exporter. +type Family struct { + // ID is a short stable identifier such as "node". + ID string + // Title is shown as the section heading. + Title string + // Detect lists metric names; the family is offered when any is present. + Detect []string + // Scope says which selector fields the panels use. + Scope Scope + // Panels in display order. + Panels []Panel +} + +// Panel is one chart or statistic backed by a PromQL template. +type Panel struct { + // ID is unique across the catalog, e.g. "node.cpu". + ID string + // Title is the panel heading. + Title string + // Unit is one of the Unit constants. + Unit string + // Query is the PromQL template; see Expand for placeholders. + Query string + // Fallback is an alternate template using older metric names. It is + // used when Detect names a metric the server does not expose. + Fallback string + // Detect optionally names the metric Query depends on. When the server + // lacks it the dashboard uses Fallback if set, or hides the panel. + Detect string + // Ceiling fixes the axis maximum; 0 means autoscale. + Ceiling float64 + // CeilingQ is an instant query whose scalar result replaces Ceiling. + CeilingQ string + // Kind selects the renderer. + Kind PanelKind + // Legend names the label(s) identifying each series, joined with "/". + // Empty means a single unnamed series. + Legend string + // TopN is the default $top value; 0 when the template has no $top. + TopN int + // Range is true when the panel uses query_range rather than query. + Range bool + // Optional panels are hidden silently when they return no data. + Optional bool + // Note is a short hint shown next to the title. + Note string +} + +// Selector carries the values substituted into panel templates. +type Selector struct { + // Instance is the Prometheus instance label value, or empty for all. + Instance string + // Node is the Proxmox node name for ScopePVENode families. + Node string + // Window is the rate window such as "2m"; empty uses the default. + Window string + // Top is the topk limit; 0 uses the panel default. + Top int +} + +// DefaultRateWindow is used when a Selector carries no window. +const DefaultRateWindow = "2m" + +// DefaultTop is used when neither the selector nor the panel sets a limit. +const DefaultTop = 8 + +var labelValueEscaper = strings.NewReplacer(`\`, `\\`, `"`, `\"`, "\n", `\n`) + +// EscapeLabelValue quotes a value for use inside a PromQL label matcher. +func EscapeLabelValue(v string) string { + return labelValueEscaper.Replace(v) +} + +// Expand substitutes selector placeholders into a PromQL template. +// +// $sel -> instance="", (with trailing comma) or nothing +// $win -> rate window, default "2m" +// $node -> escaped PVE node name +// $top -> topk limit +// +// Templates place $sel first inside braces. Empty matcher sets left behind +// by an empty selector are removed so the result stays valid PromQL. +func Expand(query string, sel Selector) string { + selText := "" + if sel.Instance != "" { + selText = `instance="` + EscapeLabelValue(sel.Instance) + `",` + } + win := sel.Window + if win == "" { + win = DefaultRateWindow + } + top := sel.Top + if top <= 0 { + top = DefaultTop + } + r := strings.NewReplacer( + "$sel", selText, + "$win", win, + "$node", EscapeLabelValue(sel.Node), + "$top", strconv.Itoa(top), + ) + out := r.Replace(query) + out = strings.ReplaceAll(out, "{ ", "{") + out = strings.ReplaceAll(out, ",}", "}") + out = strings.ReplaceAll(out, "{}", "") + return out +} + +// Expr expands the panel query, using the panel's TopN when sel.Top is 0. +func (p Panel) Expr(sel Selector) string { + return Expand(p.Query, p.withTop(sel)) +} + +// FallbackExpr expands the panel's Fallback template the same way as Expr. +// It returns "" when the panel has no fallback. +func (p Panel) FallbackExpr(sel Selector) string { + if p.Fallback == "" { + return "" + } + return Expand(p.Fallback, p.withTop(sel)) +} + +// CeilingExpr expands CeilingQ, or returns "" when the panel has none. +func (p Panel) CeilingExpr(sel Selector) string { + if p.CeilingQ == "" { + return "" + } + return Expand(p.CeilingQ, p.withTop(sel)) +} + +func (p Panel) withTop(sel Selector) Selector { + if sel.Top == 0 && p.TopN > 0 { + sel.Top = p.TopN + } + return sel +} + +// Label matcher fragments shared by several templates. +const ( + guestSel = `id=~"(qemu|lxc)/.+"` + guestJoin = ` * on (id) group_left (name, type) pve_guest_info{$sel node="$node"}` + diskDevices = `device!~"loop.*|ram.*|zram.*|sr.*|fd.*"` + netDevices = `device!~"lo|veth.*|docker.*|br-.*|virbr.*|cali.*|flannel.*|cni.*|tap.*|fwbr.*|fwln.*|fwpr.*|vmbr.*"` + fsTypes = `fstype!~"tmpfs|overlay|squashfs|ramfs|devtmpfs|nsfs|autofs|fuse.*"` + containerNet = `interface!~"lo|veth.*"` +) + +// Catalog lists every family in display order. +var Catalog = []Family{ + { + ID: "node", + Title: "Host (node_exporter)", + Detect: []string{"node_cpu_seconds_total", "node_memory_MemTotal_bytes"}, + Scope: ScopeInstance, + Panels: []Panel{ + {ID: "node.cpu", Title: "CPU busy", Unit: UnitPercent, Ceiling: 100, Kind: KindArea, Range: true, + Query: `100 * (1 - avg(rate(node_cpu_seconds_total{$sel mode="idle"}[$win])))`}, + {ID: "node.cpu.percpu", Title: "Per-CPU busy", Unit: UnitPercent, Ceiling: 100, Kind: KindBars, Legend: "cpu", + Query: `100 * (1 - rate(node_cpu_seconds_total{$sel mode="idle"}[$win]))`}, + {ID: "node.cpu.iowait", Title: "I/O wait + steal", Unit: UnitPercent, Ceiling: 100, Kind: KindSpark, Legend: "mode", Range: true, Optional: true, + Query: `100 * sum by (mode) (rate(node_cpu_seconds_total{$sel mode=~"iowait|steal"}[$win])) / scalar(count(node_cpu_seconds_total{$sel mode="idle"}) > 0)`, + Note: "share of all CPUs"}, + {ID: "node.mem.pct", Title: "Memory used", Unit: UnitPercent, Ceiling: 100, Kind: KindArea, Range: true, + Detect: "node_memory_MemAvailable_bytes", + Query: `100 * (1 - node_memory_MemAvailable_bytes{$sel} / (node_memory_MemTotal_bytes{$sel} > 0))`, + Fallback: `100 * (node_memory_active_bytes{$sel} + node_memory_wired_bytes{$sel} + node_memory_compressed_bytes{$sel}) / (node_memory_total_bytes{$sel} > 0)`, + Note: "macOS: active + wired + compressed"}, + {ID: "node.swap", Title: "Swap used", Unit: UnitPercent, Ceiling: 100, Kind: KindArea, Range: true, Optional: true, + Detect: "node_memory_SwapTotal_bytes", + Query: `100 * (1 - node_memory_SwapFree_bytes{$sel} / (node_memory_SwapTotal_bytes{$sel} > 0))`, + Fallback: `100 * node_memory_swap_used_bytes{$sel} / (node_memory_swap_total_bytes{$sel} > 0)`, + Note: "hidden without swap"}, + {ID: "node.load", Title: "Load 1/5/15", Unit: UnitLoad, Kind: KindSpark, Legend: "__name__", Range: true, + Query: `{$sel __name__=~"node_load(1|5|15)"}`, + CeilingQ: `count(node_cpu_seconds_total{$sel mode="idle"})`, + Note: "ceiling = CPU count"}, + {ID: "node.fs", Title: "Filesystem used", Unit: UnitPercent, Ceiling: 100, Kind: KindBars, Legend: "mountpoint", TopN: 12, + Query: `topk($top, 100 * (1 - node_filesystem_avail_bytes{$sel ` + fsTypes + `} / (node_filesystem_size_bytes{$sel ` + fsTypes + `} > 0)))`, + Note: "real filesystems only"}, + {ID: "node.disk.read", Title: "Disk read", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: "device", TopN: 4, Range: true, + Query: `topk($top, sum by (device) (rate(node_disk_read_bytes_total{$sel ` + diskDevices + `}[$win])))`}, + {ID: "node.disk.write", Title: "Disk write", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: "device", TopN: 4, Range: true, + Query: `topk($top, sum by (device) (rate(node_disk_written_bytes_total{$sel ` + diskDevices + `}[$win])))`}, + {ID: "node.disk.busy", Title: "Disk busy", Unit: UnitPercent, Ceiling: 100, Kind: KindBars, Legend: "device", TopN: 6, + Query: `topk($top, 100 * rate(node_disk_io_time_seconds_total{$sel ` + diskDevices + `}[$win]))`}, + {ID: "node.net.rx", Title: "Network receive", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: "device", TopN: 4, Range: true, + Query: `topk($top, rate(node_network_receive_bytes_total{$sel ` + netDevices + `}[$win]))`, + Note: "physical interfaces; bridges and veths excluded"}, + {ID: "node.net.tx", Title: "Network transmit", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: "device", TopN: 4, Range: true, + Query: `topk($top, rate(node_network_transmit_bytes_total{$sel ` + netDevices + `}[$win]))`, + Note: "physical interfaces; bridges and veths excluded"}, + {ID: "node.uptime", Title: "Up since", Unit: UnitSeconds, Kind: KindStat, + Query: `node_time_seconds{$sel} - node_boot_time_seconds{$sel}`}, + {ID: "node.psi", Title: "Pressure stall", Unit: UnitPercent, Ceiling: 100, Kind: KindSpark, Legend: "__name__", Range: true, Optional: true, + Query: `100 * rate({$sel __name__=~"node_pressure_(cpu|memory|io)_waiting_seconds_total"}[$win])`, + Note: "needs kernel PSI"}, + {ID: "node.temp", Title: "Temperatures", Unit: UnitCelsius, Kind: KindBars, Legend: "label", TopN: 8, Optional: true, + Query: `topk($top, (node_hwmon_temp_celsius{$sel} * on (chip, sensor) group_left (label) node_hwmon_sensor_label{$sel}) or node_hwmon_temp_celsius{$sel})`, + CeilingQ: `max(node_hwmon_temp_crit_celsius{$sel})`, + Note: "legend falls back to chip/sensor; ceiling = critical"}, + }, + }, + { + ID: "cadvisor", + Title: "Containers (cAdvisor)", + Detect: []string{"container_cpu_usage_seconds_total"}, + Scope: ScopeInstance, + Panels: cadvisorPanels("cadvisor", `name!=""`, "name"), + }, + { + ID: "kubelet", + Title: "Pods (kubelet cAdvisor)", + Detect: []string{"container_cpu_usage_seconds_total"}, + Scope: ScopeInstance, + Panels: append(cadvisorPanels("kubelet", `container!="", container!="POD"`, "namespace/pod"), + Panel{ID: "kubelet.restarts", Title: "Container restarts", Unit: UnitCount, Kind: KindBars, Legend: "namespace/pod", TopN: 8, Optional: true, + Detect: "kube_pod_container_status_restarts_total", + Query: `topk($top, sum by (namespace, pod) (kube_pod_container_status_restarts_total))`, + Note: "needs kube-state-metrics"}, + ), + }, + { + ID: "pve", + Title: "Proxmox VE (pve-exporter)", + Detect: []string{"pve_up"}, + Scope: ScopePVENode, + Panels: []Panel{ + {ID: "pve.node.cpu", Title: "Node CPU", Unit: UnitPercent, Ceiling: 100, Kind: KindArea, Range: true, + Query: `100 * pve_cpu_usage_ratio{$sel id="node/$node"}`}, + {ID: "pve.node.mem", Title: "Node memory", Unit: UnitPercent, Ceiling: 100, Kind: KindArea, Range: true, + Query: `100 * pve_memory_usage_bytes{$sel id="node/$node"} / (pve_memory_size_bytes{$sel id="node/$node"} > 0)`}, + {ID: "pve.node.up", Title: "Node status", Unit: UnitNone, Kind: KindStat, + Query: `pve_up{$sel id="node/$node"}`, Note: "1 = online"}, + {ID: "pve.node.uptime", Title: "Node uptime", Unit: UnitSeconds, Kind: KindStat, + Query: `pve_uptime_seconds{$sel id="node/$node"}`}, + {ID: "pve.node.version", Title: "PVE version", Unit: UnitNone, Kind: KindStat, Legend: "version", Optional: true, + Query: `pve_version_info{$sel}`}, + {ID: "pve.guest.cpu", Title: "VM & CT CPU", Unit: UnitPercent, Ceiling: 100, Kind: KindBars, Legend: "name", TopN: 12, + Query: `topk($top, 100 * pve_cpu_usage_ratio{$sel ` + guestSel + `}` + guestJoin + `)`}, + {ID: "pve.guest.mem", Title: "VM & CT memory", Unit: UnitPercent, Ceiling: 100, Kind: KindBars, Legend: "name", TopN: 12, + Query: `topk($top, 100 * pve_memory_usage_bytes{$sel ` + guestSel + `} / (pve_memory_size_bytes{$sel ` + guestSel + `} > 0)` + guestJoin + `)`}, + {ID: "pve.guest.net.rx", Title: "Guest receive", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: "name", TopN: 6, Range: true, + Detect: "pve_network_receive_bytes_total", + Query: `topk($top, rate(pve_network_receive_bytes_total{$sel ` + guestSel + `}[$win])` + guestJoin + `)`, + Fallback: `topk($top, rate(pve_network_receive_bytes{$sel ` + guestSel + `}[$win])` + guestJoin + `)`, + Note: "older exporters use pve_network_receive_bytes"}, + {ID: "pve.guest.net.tx", Title: "Guest transmit", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: "name", TopN: 6, Range: true, + Detect: "pve_network_transmit_bytes_total", + Query: `topk($top, rate(pve_network_transmit_bytes_total{$sel ` + guestSel + `}[$win])` + guestJoin + `)`, + Fallback: `topk($top, rate(pve_network_transmit_bytes{$sel ` + guestSel + `}[$win])` + guestJoin + `)`, + Note: "older exporters use pve_network_transmit_bytes"}, + {ID: "pve.guest.disk.read", Title: "Guest disk read", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: "name", TopN: 6, Range: true, Optional: true, + Detect: "pve_disk_read_bytes_total", + Query: `topk($top, rate(pve_disk_read_bytes_total{$sel ` + guestSel + `}[$win])` + guestJoin + `)`, + Fallback: `topk($top, rate(pve_disk_read_bytes{$sel ` + guestSel + `}[$win])` + guestJoin + `)`, + Note: "older exporters use pve_disk_read_bytes"}, + {ID: "pve.guest.disk.write", Title: "Guest disk write", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: "name", TopN: 6, Range: true, Optional: true, + Detect: "pve_disk_written_bytes_total", + Query: `topk($top, rate(pve_disk_written_bytes_total{$sel ` + guestSel + `}[$win])` + guestJoin + `)`, + Fallback: `topk($top, rate(pve_disk_write_bytes{$sel ` + guestSel + `}[$win])` + guestJoin + `)`, + Note: "older exporters use pve_disk_write_bytes"}, + {ID: "pve.guests.running", Title: "Guests running", Unit: UnitCount, Kind: KindStat, + Query: `count(pve_up{$sel ` + guestSel + `} * on (id) group_left pve_guest_info{$sel node="$node"} == 1) or vector(0)`}, + {ID: "pve.guests.stopped", Title: "Guests stopped", Unit: UnitCount, Kind: KindStat, + Query: `count(pve_up{$sel ` + guestSel + `} * on (id) group_left pve_guest_info{$sel node="$node"} == 0) or vector(0)`}, + {ID: "pve.storage", Title: "Storage used", Unit: UnitPercent, Ceiling: 100, Kind: KindBars, Legend: "storage", TopN: 12, + Query: `topk($top, 100 * pve_disk_usage_bytes{$sel id=~"storage/.+"} / (pve_disk_size_bytes{$sel id=~"storage/.+"} > 0) * on (id) group_left (storage, plugintype) pve_storage_info{$sel node="$node"})`}, + {ID: "pve.ha", Title: "HA state", Unit: UnitNone, Kind: KindStat, Legend: "id/state", Optional: true, + Detect: "pve_ha_state", + Query: `pve_ha_state{$sel ` + guestSel + `, state!~"started|stopped"} == 1`, + Note: "guests in transitional HA states"}, + }, + }, + { + ID: "prometheus", + Title: "Prometheus server", + Detect: []string{"prometheus_tsdb_head_series"}, + Scope: ScopeNone, + Panels: []Panel{ + {ID: "prom.series", Title: "Head series", Unit: UnitCount, Kind: KindArea, Range: true, + Query: `max(prometheus_tsdb_head_series)`}, + {ID: "prom.samples", Title: "Samples appended", Unit: UnitPerSecond, Kind: KindArea, Range: true, + Query: `sum(rate(prometheus_tsdb_head_samples_appended_total[$win]))`}, + {ID: "prom.targets", Title: "Targets up", Unit: UnitCount, Kind: KindStat, + Query: `sum(up)`, Note: "of count(up)"}, + {ID: "prom.targets.total", Title: "Targets total", Unit: UnitCount, Kind: KindStat, + Query: `count(up)`}, + {ID: "prom.scrape", Title: "Scrape duration", Unit: UnitSeconds, Kind: KindBars, Legend: "job/instance", TopN: 8, + Query: `topk($top, scrape_duration_seconds)`}, + }, + }, +} + +// cadvisorPanels builds the shared container panels for the cadvisor and +// kubelet families. prefix is the panel ID prefix, guard the label matcher +// that excludes pause and cgroup-only series, and legend the labels that +// name a series ("name" or "namespace/pod"). Network metrics on kubelet are +// pod-level, so they use a pod guard instead. +func cadvisorPanels(prefix, guard, legend string) []Panel { + by := strings.ReplaceAll(legend, "/", ", ") + netGuard := guard + if prefix == "kubelet" { + netGuard = `pod!=""` + } + sel := `{$sel ` + guard + `}` + netSel := func(metric string) string { + return metric + `{$sel ` + netGuard + `, ` + containerNet + `}` + } + return []Panel{ + {ID: prefix + ".cpu", Title: "Container CPU", Unit: UnitPercent, Kind: KindSpark, Legend: legend, TopN: 8, Range: true, + Query: `topk($top, 100 * sum by (` + by + `) (rate(container_cpu_usage_seconds_total` + sel + `[$win])))`, + Note: "percent of one CPU"}, + {ID: prefix + ".mem", Title: "Working set", Unit: UnitBytes, Kind: KindBars, Legend: legend, TopN: 8, + Query: `topk($top, sum by (` + by + `) (container_memory_working_set_bytes` + sel + `))`}, + {ID: prefix + ".mem.pct", Title: "Memory vs limit", Unit: UnitPercent, Ceiling: 100, Kind: KindBars, Legend: legend, TopN: 8, Optional: true, + Query: `topk($top, 100 * sum by (` + by + `) (container_memory_working_set_bytes` + sel + `) / (sum by (` + by + `) (container_spec_memory_limit_bytes` + sel + ` > 0 < 1e18) > 0))`, + Note: "only containers with a memory limit"}, + {ID: prefix + ".net.rx", Title: "Container receive", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: legend, TopN: 6, Range: true, + Query: `topk($top, sum by (` + by + `) (rate(` + netSel("container_network_receive_bytes_total") + `[$win])))`}, + {ID: prefix + ".net.tx", Title: "Container transmit", Unit: UnitBytesPerSecond, Kind: KindSpark, Legend: legend, TopN: 6, Range: true, + Query: `topk($top, sum by (` + by + `) (rate(` + netSel("container_network_transmit_bytes_total") + `[$win])))`}, + {ID: prefix + ".age", Title: "Running since", Unit: UnitSeconds, Kind: KindStat, Legend: legend, + Query: `time() - max by (` + by + `) (container_start_time_seconds` + sel + `)`, + Note: "list"}, + {ID: prefix + ".count", Title: "Containers seen", Unit: UnitCount, Kind: KindStat, + Query: `count(container_last_seen` + sel + ` > time() - 60)`, + Note: "seen in the last minute"}, + } +} + +// FamilyByID returns the catalog family with the given ID. +func FamilyByID(id string) (Family, bool) { + for _, f := range Catalog { + if f.ID == id { + return f, true + } + } + return Family{}, false +} + +// DetectFamilies returns the catalog families whose Detect metrics appear in +// names, in catalog order. container_cpu_usage_seconds_total is attributed to +// the kubelet family when kubeletHint is true and to cadvisor otherwise. +func DetectFamilies(names []string, kubeletHint bool) []Family { + present := make(map[string]bool, len(names)) + for _, n := range names { + present[n] = true + } + var out []Family + for _, f := range Catalog { + if f.ID == "cadvisor" && kubeletHint || f.ID == "kubelet" && !kubeletHint { + continue + } + for _, d := range f.Detect { + if present[d] { + out = append(out, f) + break + } + } + } + return out +} + +// DetectNames returns every metric name the catalog probes for, without +// duplicates and in catalog order. It feeds the match[] parameter of the +// metric-name lookup. +func DetectNames() []string { + seen := make(map[string]bool) + var out []string + add := func(n string) { + if n != "" && !seen[n] { + seen[n] = true + out = append(out, n) + } + } + for _, f := range Catalog { + for _, d := range f.Detect { + add(d) + } + for _, p := range f.Panels { + add(p.Detect) + } + } + return out +} + +// RateWindow picks the rate() window for a scrape interval: four scrapes, +// clamped to 1m..5m and rounded up to "1m", "2m" or "5m". Unknown or +// non-positive intervals yield the default "2m". +func RateWindow(scrapeInterval time.Duration) string { + if scrapeInterval <= 0 { + return DefaultRateWindow + } + w := 4 * scrapeInterval + switch { + case w <= time.Minute: + return "1m" + case w <= 2*time.Minute: + return "2m" + default: + return "5m" + } +} + +// StepFor returns the query_range step for drawing window across columns: +// at least one scrape interval, at least window/columns, and rounded up to +// a whole number of scrape intervals. A non-positive scrape interval is +// treated as 15s and columns below 1 as 1. +func StepFor(window time.Duration, columns int, scrapeInterval time.Duration) time.Duration { + if scrapeInterval <= 0 { + scrapeInterval = 15 * time.Second + } + if columns < 1 { + columns = 1 + } + if window < 0 { + window = 0 + } + raw := window / time.Duration(columns) + if window%time.Duration(columns) != 0 { + raw++ + } + if raw < scrapeInterval { + return scrapeInterval + } + if rem := raw % scrapeInterval; rem != 0 { + raw += scrapeInterval - rem + } + return raw +} diff --git a/internal/promapi/catalog_test.go b/internal/promapi/catalog_test.go new file mode 100644 index 0000000..6eb2a9d --- /dev/null +++ b/internal/promapi/catalog_test.go @@ -0,0 +1,285 @@ +package promapi + +import ( + "regexp" + "strings" + "testing" + "time" +) + +var identRE = regexp.MustCompile(`[A-Za-z_:][A-Za-z0-9_:]*`) + +// familyPrefixes lists metric prefixes (or exact names) that count as a +// family reference when a panel does not name a Detect metric directly. +var familyPrefixes = map[string][]string{ + "node": {"node_"}, + "cadvisor": {"container_"}, + "kubelet": {"container_"}, + "pve": {"pve_"}, + "prometheus": {"prometheus_", "scrape_", "up"}, +} + +func checkExpanded(t *testing.T, id, label, q string) { + t.Helper() + if strings.Count(q, "(") != strings.Count(q, ")") { + t.Errorf("%s %s: unbalanced parentheses: %s", id, label, q) + } + if strings.Count(q, "{") != strings.Count(q, "}") { + t.Errorf("%s %s: unbalanced braces: %s", id, label, q) + } + for _, bad := range []string{"{,", ",,", ",}", "{}", "$"} { + if strings.Contains(q, bad) { + t.Errorf("%s %s: contains %q: %s", id, label, bad, q) + } + } +} + +func allTemplates(p Panel) map[string]string { + out := map[string]string{"query": p.Query} + if p.Fallback != "" { + out["fallback"] = p.Fallback + } + if p.CeilingQ != "" { + out["ceiling"] = p.CeilingQ + } + return out +} + +func TestCatalogExpandsCleanly(t *testing.T) { + selectors := map[string]Selector{ + "empty": {}, + "instance": {Instance: "host:9100", Node: "pve1", Window: "5m", Top: 3}, + "quoted": {Instance: `we"ird\host`, Node: `n"1`}, + } + for _, f := range Catalog { + for _, p := range f.Panels { + if p.Query == "" { + t.Errorf("%s: empty query", p.ID) + } + for label, tpl := range allTemplates(p) { + for name, sel := range selectors { + checkExpanded(t, p.ID, label+"/"+name, Expand(tpl, sel)) + } + } + if p.TopN > 0 != strings.Contains(p.Query, "$top") { + t.Errorf("%s: TopN=%d but query $top presence is %v", p.ID, p.TopN, strings.Contains(p.Query, "$top")) + } + if p.Range != (p.Kind == KindArea || p.Kind == KindSpark) { + t.Errorf("%s: Range=%v does not match kind %s", p.ID, p.Range, p.Kind) + } + if f.Scope == ScopePVENode && strings.Contains(p.Query, "$node") && !strings.Contains(p.Query, "$sel") { + t.Errorf("%s: uses $node without $sel", p.ID) + } + } + } +} + +func TestExpandSelector(t *testing.T) { + got := Expand(`node_memory_MemTotal_bytes{$sel}`, Selector{}) + if got != "node_memory_MemTotal_bytes" { + t.Errorf("empty selector: got %q", got) + } + got = Expand(`node_memory_MemTotal_bytes{$sel}`, Selector{Instance: "a:9100"}) + if got != `node_memory_MemTotal_bytes{instance="a:9100"}` { + t.Errorf("instance selector: got %q", got) + } + got = Expand(`rate(x{$sel mode="idle"}[$win])`, Selector{Instance: `q"\`}) + if got != `rate(x{instance="q\"\\", mode="idle"}[2m])` { + t.Errorf("escaping: got %q", got) + } + got = Expand(`topk($top, y{$sel id="node/$node"})`, Selector{Node: "pve1", Top: 2, Window: "5m"}) + if got != `topk(2, y{id="node/pve1"})` { + t.Errorf("node/top: got %q", got) + } + p := Panel{Query: `topk($top, z)`, TopN: 12} + if got := p.Expr(Selector{}); got != "topk(12, z)" { + t.Errorf("Expr default TopN: got %q", got) + } + if got := p.Expr(Selector{Top: 2}); got != "topk(2, z)" { + t.Errorf("Expr explicit Top: got %q", got) + } + if got := Expand(`topk($top, z)`, Selector{}); got != "topk(8, z)" { + t.Errorf("Expand default top: got %q", got) + } + if got := (Panel{}).FallbackExpr(Selector{}); got != "" { + t.Errorf("FallbackExpr without fallback: got %q", got) + } +} + +func TestCatalogContainerGuards(t *testing.T) { + for _, f := range Catalog { + for _, p := range f.Panels { + for label, tpl := range allTemplates(p) { + container := false + for _, ident := range identRE.FindAllString(tpl, -1) { + if strings.HasPrefix(ident, "container_") { + container = true + } + } + if !container { + continue + } + if !strings.Contains(tpl, `name!=""`) && !strings.Contains(tpl, `container!=""`) && !strings.Contains(tpl, `pod!=""`) { + t.Errorf("%s %s: container_ query without a name/container/pod guard: %s", p.ID, label, tpl) + } + } + } + } +} + +func TestCatalogPanelsReferenceFamily(t *testing.T) { + for _, f := range Catalog { + prefixes := familyPrefixes[f.ID] + if prefixes == nil { + t.Fatalf("family %s has no prefix entry in the test table", f.ID) + } + for _, p := range f.Panels { + if p.Optional { + continue + } + text := p.Query + " " + p.CeilingQ + found := false + for _, ident := range identRE.FindAllString(text, -1) { + for _, d := range f.Detect { + if ident == d { + found = true + } + } + for _, pre := range prefixes { + if ident == pre || strings.HasPrefix(ident, pre) && strings.HasSuffix(pre, "_") { + found = true + } + } + } + if !found { + t.Errorf("%s: references neither a Detect metric nor a %s metric", p.ID, f.ID) + } + } + } +} + +func TestCatalogIDsUnique(t *testing.T) { + seen := map[string]string{} + famSeen := map[string]bool{} + for _, f := range Catalog { + if famSeen[f.ID] { + t.Errorf("duplicate family %s", f.ID) + } + famSeen[f.ID] = true + if len(f.Detect) == 0 { + t.Errorf("family %s has no Detect metrics", f.ID) + } + for _, p := range f.Panels { + if prev, ok := seen[p.ID]; ok { + t.Errorf("panel %s in %s duplicates one in %s", p.ID, f.ID, prev) + } + seen[p.ID] = f.ID + if p.Fallback != "" && p.Detect == "" { + t.Errorf("panel %s has a Fallback but no Detect", p.ID) + } + } + } + if _, ok := FamilyByID("node"); !ok { + t.Error("FamilyByID(node) missing") + } + if _, ok := FamilyByID("nope"); ok { + t.Error("FamilyByID(nope) found") + } +} + +func TestDetectNames(t *testing.T) { + names := DetectNames() + seen := map[string]bool{} + for _, n := range names { + if seen[n] { + t.Errorf("duplicate detect name %s", n) + } + seen[n] = true + } + for _, want := range []string{"node_cpu_seconds_total", "container_cpu_usage_seconds_total", "pve_up", "prometheus_tsdb_head_series", "pve_network_receive_bytes_total", "kube_pod_container_status_restarts_total"} { + if !seen[want] { + t.Errorf("DetectNames missing %s", want) + } + } +} + +func TestDetectFamilies(t *testing.T) { + ids := func(fs []Family) string { + var out []string + for _, f := range fs { + out = append(out, f.ID) + } + return strings.Join(out, ",") + } + all := []string{"pve_up", "prometheus_tsdb_head_series", "node_memory_MemTotal_bytes", "container_cpu_usage_seconds_total", "unrelated"} + cases := []struct { + name string + names []string + kube bool + want string + }{ + {"none", nil, false, ""}, + {"unrelated", []string{"foo", "bar"}, false, ""}, + {"node only", []string{"node_cpu_seconds_total"}, false, "node"}, + {"node by second detect", []string{"node_memory_MemTotal_bytes"}, true, "node"}, + {"cadvisor", []string{"container_cpu_usage_seconds_total"}, false, "cadvisor"}, + {"kubelet hint", []string{"container_cpu_usage_seconds_total"}, true, "kubelet"}, + {"all", all, false, "node,cadvisor,pve,prometheus"}, + {"all kube", all, true, "node,kubelet,pve,prometheus"}, + } + for _, c := range cases { + if got := ids(DetectFamilies(c.names, c.kube)); got != c.want { + t.Errorf("%s: got %q want %q", c.name, got, c.want) + } + } +} + +func TestRateWindow(t *testing.T) { + cases := []struct { + in time.Duration + want string + }{ + {0, "2m"}, + {-time.Second, "2m"}, + {5 * time.Second, "1m"}, + {15 * time.Second, "1m"}, + {20 * time.Second, "2m"}, + {30 * time.Second, "2m"}, + {45 * time.Second, "5m"}, + {60 * time.Second, "5m"}, + {10 * time.Minute, "5m"}, + } + for _, c := range cases { + if got := RateWindow(c.in); got != c.want { + t.Errorf("RateWindow(%s) = %s want %s", c.in, got, c.want) + } + } +} + +func TestStepFor(t *testing.T) { + cases := []struct { + window time.Duration + columns int + interval time.Duration + want time.Duration + }{ + {time.Hour, 60, 15 * time.Second, time.Minute}, + {time.Hour, 120, 15 * time.Second, 30 * time.Second}, + {time.Hour, 1000, 15 * time.Second, 15 * time.Second}, + {time.Hour, 100, 15 * time.Second, 45 * time.Second}, + {time.Hour, 0, 15 * time.Second, time.Hour}, + {time.Hour, 60, 0, time.Minute}, + {0, 60, 30 * time.Second, 30 * time.Second}, + {61 * time.Second, 60, time.Second, 2 * time.Second}, + {7 * time.Second, 2, 2 * time.Second, 4 * time.Second}, + } + for _, c := range cases { + got := StepFor(c.window, c.columns, c.interval) + if got != c.want { + t.Errorf("StepFor(%s, %d, %s) = %s want %s", c.window, c.columns, c.interval, got, c.want) + } + if iv := c.interval; iv > 0 && got%iv != 0 { + t.Errorf("StepFor(%s, %d, %s) = %s is not a multiple of the interval", c.window, c.columns, c.interval, got) + } + } +} diff --git a/internal/promapi/client.go b/internal/promapi/client.go new file mode 100644 index 0000000..b2191f2 --- /dev/null +++ b/internal/promapi/client.go @@ -0,0 +1,178 @@ +package promapi + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "path" + "strings" + "time" + + "systemdoc/internal/textutil" +) + +const ( + // requestTimeout matches the ceiling commandcapture applies to CLI snapshots. + requestTimeout = 8 * time.Second + // bodyLimit matches the per-stream cap of commandcapture. + bodyLimit = 4 << 20 + userAgent = "systemdoc" +) + +// ErrUnauthorized reports a 401 or 403 from the server. +var ErrUnauthorized = errors.New("prometheus rejected the request; set SYSTEMDOC_PROMETHEUS_TOKEN or check the token") + +// Client issues bounded requests to one Prometheus server. +type Client struct { + base *url.URL + token string + http *http.Client +} + +// New validates a base URL of the form http(s)://host[:port][/prefix] and +// returns a client. A missing scheme defaults to http. A username, password, +// query string or fragment is rejected so credentials cannot end up in +// settings or on screen; token is sent as a bearer header instead. +func New(raw, token string) (*Client, error) { + raw = strings.TrimSpace(raw) + if raw == "" { + return nil, fmt.Errorf("no Prometheus URL configured") + } + if !strings.Contains(raw, "://") { + raw = "http://" + raw + } + parsed, err := url.Parse(raw) + if err != nil { + return nil, fmt.Errorf("prometheus URL: %w", err) + } + if parsed.Scheme != "http" && parsed.Scheme != "https" { + return nil, fmt.Errorf("prometheus URL must use http or https, not %q", parsed.Scheme) + } + if parsed.User != nil { + return nil, fmt.Errorf("prometheus URL must not contain a username or password; use SYSTEMDOC_PROMETHEUS_TOKEN") + } + if parsed.RawQuery != "" || parsed.Fragment != "" { + return nil, fmt.Errorf("prometheus URL must not contain a query string or fragment") + } + if parsed.Host == "" { + return nil, fmt.Errorf("prometheus URL needs a host, for example http://localhost:9090") + } + parsed.Path = strings.TrimRight(parsed.Path, "/") + parsed.RawPath = "" + transport := http.DefaultTransport.(*http.Transport).Clone() + return &Client{ + base: parsed, + token: token, + http: &http.Client{ + Timeout: requestTimeout, + Transport: transport, + // A redirect could carry the bearer token to another host; the + // user configures the final URL instead. + CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }, + }, + }, nil +} + +// String is the base URL for status lines. It never carries credentials. +func (c *Client) String() string { return c.base.String() } + +// endpoint joins the API path under any configured prefix. +func (c *Client) endpoint(apiPath string) string { + u := *c.base + u.Path = path.Join(c.base.Path, apiPath) + return u.String() +} + +// fetch issues one bounded request and returns the body. Errors name the API +// path and the base URL only, never the token. +func (c *Client) fetch(ctx context.Context, method, apiPath string, form url.Values) (int, []byte, error) { + ctx, cancel := context.WithTimeout(ctx, requestTimeout) + defer cancel() + target := c.endpoint(apiPath) + var body io.Reader + if method == http.MethodGet && len(form) > 0 { + target += "?" + form.Encode() + } else if len(form) > 0 { + body = strings.NewReader(form.Encode()) + } + request, err := http.NewRequestWithContext(ctx, method, target, body) + if err != nil { + return 0, nil, fmt.Errorf("%s: %w", apiPath, err) + } + request.Header.Set("Accept", "application/json") + request.Header.Set("User-Agent", userAgent) + if body != nil { + request.Header.Set("Content-Type", "application/x-www-form-urlencoded") + } + if c.token != "" { + request.Header.Set("Authorization", "Bearer "+c.token) + } + response, err := c.http.Do(request) + if err != nil { + if ctx.Err() != nil { + return 0, nil, fmt.Errorf("%s: %w", apiPath, ctx.Err()) + } + return 0, nil, fmt.Errorf("%s at %s: %s", apiPath, c.String(), transportReason(err)) + } + defer response.Body.Close() + data, err := io.ReadAll(io.LimitReader(response.Body, bodyLimit+1)) + if err != nil { + if ctx.Err() != nil { + return 0, nil, fmt.Errorf("%s: %w", apiPath, ctx.Err()) + } + return 0, nil, fmt.Errorf("%s: reading response: %w", apiPath, err) + } + if len(data) > bodyLimit { + return 0, nil, fmt.Errorf("%s: response exceeded 4 MiB; it was not used", apiPath) + } + return response.StatusCode, data, nil +} + +// transportReason strips the request URL from a net/http error so a status +// line shows the cause rather than a long address. +func transportReason(err error) string { + var urlErr *url.Error + if errors.As(err, &urlErr) { + return textutil.Clean(urlErr.Err.Error()) + } + return textutil.Clean(err.Error()) +} + +// do fetches an /api/v1 endpoint and decodes its envelope into out. +func (c *Client) do(ctx context.Context, method, apiPath string, form url.Values, out any) error { + status, data, err := c.fetch(ctx, method, apiPath, form) + if err != nil { + return err + } + switch { + case status >= 300 && status < 400: + return fmt.Errorf("%s: redirected by %s; configure the final Prometheus URL", apiPath, c.String()) + case status == http.StatusUnauthorized || status == http.StatusForbidden: + return fmt.Errorf("%s: %w", apiPath, ErrUnauthorized) + } + var env envelope + if err := json.Unmarshal(data, &env); err != nil || env.Status == "" { + // The body is not echoed: an upstream proxy may reflect request headers. + if status >= 400 { + return fmt.Errorf("%s: HTTP %d from %s", apiPath, status, c.String()) + } + return fmt.Errorf("%s: not a Prometheus API response from %s", apiPath, c.String()) + } + if env.Status != "success" { + return fmt.Errorf("%s: %s: %s", apiPath, textutil.Clean(env.ErrorType), firstLine(textutil.Clean(env.Error))) + } + if status >= 400 { + return fmt.Errorf("%s: HTTP %d", apiPath, status) + } + if out == nil { + return nil + } + if err := json.Unmarshal(env.Data, out); err != nil { + return fmt.Errorf("%s: unexpected response shape: %w", apiPath, err) + } + return nil +} diff --git a/internal/promapi/client_test.go b/internal/promapi/client_test.go new file mode 100644 index 0000000..1ec5455 --- /dev/null +++ b/internal/promapi/client_test.go @@ -0,0 +1,360 @@ +package promapi + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func fixture(t *testing.T, name string) []byte { + t.Helper() + data, err := os.ReadFile(filepath.Join("testdata", name)) + if err != nil { + t.Fatal(err) + } + return data +} + +func serve(t *testing.T, handler http.HandlerFunc) *Client { + t.Helper() + server := httptest.NewServer(handler) + t.Cleanup(server.Close) + client, err := New(server.URL, "") + if err != nil { + t.Fatal(err) + } + return client +} + +func TestNewValidation(t *testing.T) { + cases := []struct{ raw, want, problem string }{ + {"", "", "no Prometheus URL"}, + {"localhost:9090", "http://localhost:9090", ""}, + {"http://localhost:9090/", "http://localhost:9090", ""}, + {"https://prom.example/prom/", "https://prom.example/prom", ""}, + {"ftp://host", "", "http or https"}, + {"http://user:pw@host:9090", "", "username or password"}, + {"http://host/?x=1", "", "query string"}, + {"http://host/#frag", "", "query string"}, + {"http://", "", "needs a host"}, + } + for _, c := range cases { + client, err := New(c.raw, "") + if c.problem != "" { + if err == nil || !strings.Contains(err.Error(), c.problem) { + t.Errorf("%q: want error containing %q, got %v", c.raw, c.problem, err) + } + continue + } + if err != nil { + t.Errorf("%q: %v", c.raw, err) + continue + } + if client.String() != c.want { + t.Errorf("%q: want %q, got %q", c.raw, c.want, client.String()) + } + } +} + +func TestQueryVectorScalarAndUnknownValues(t *testing.T) { + var method, contentType, body string + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + method, contentType = r.Method, r.Header.Get("Content-Type") + raw, _ := readAll(r) + body = raw + if strings.Contains(raw, "scalar") { + w.Write(fixture(t, "query-scalar.json")) + return + } + w.Write(fixture(t, "query-vector.json")) + }) + samples, err := client.Query(context.Background(), "node_load1", time.Unix(1700000000, 0)) + if err != nil { + t.Fatal(err) + } + if method != http.MethodPost || contentType != "application/x-www-form-urlencoded" || !strings.Contains(body, "query=node_load1") || !strings.Contains(body, "time=1700000000") { + t.Fatalf("query must be a POST form: %s %s %q", method, contentType, body) + } + if len(samples) != 3 { + t.Fatalf("want 3 samples, got %d", len(samples)) + } + if !samples[0].Known || samples[0].Value != 1.25 || samples[0].Labels["instance"] != "localhost:9100" || samples[0].At.Unix() != 1700000000 { + t.Errorf("first sample: %+v", samples[0]) + } + if samples[1].Known || samples[2].Known { + t.Errorf("NaN and Inf must be unknown: %+v %+v", samples[1], samples[2]) + } + if strings.Contains(samples[1].Labels["instance"], "\x1b") { + t.Errorf("label value was not cleaned: %q", samples[1].Labels["instance"]) + } + scalar, err := client.Query(context.Background(), "scalar(1)", time.Time{}) + if err != nil || len(scalar) != 1 || scalar[0].Value != 42 || len(scalar[0].Labels) != 0 { + t.Fatalf("scalar: %v %+v", err, scalar) + } +} + +func TestQueryRejectsMatrix(t *testing.T) { + client := serve(t, func(w http.ResponseWriter, r *http.Request) { w.Write(fixture(t, "query-matrix.json")) }) + if _, err := client.Query(context.Background(), "x", time.Time{}); err == nil || !strings.Contains(err.Error(), "matrix") { + t.Fatalf("want matrix rejection, got %v", err) + } +} + +func TestQueryRangeMatrix(t *testing.T) { + var body string + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + body, _ = readAll(r) + w.Write(fixture(t, "query-matrix.json")) + }) + start, end := time.Unix(1700000000, 0), time.Unix(1700000045, 0) + series, err := client.QueryRange(context.Background(), "rate(x[1m])", start, end, 15*time.Second) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(body, "step=15") || !strings.Contains(body, "start=1700000000") || !strings.Contains(body, "end=1700000045") { + t.Errorf("form: %q", body) + } + if len(series) != 2 || len(series[0].Points) != 4 || series[0].Labels["cpu"] != "0" { + t.Fatalf("series: %+v", series) + } + if series[0].Points[2].Known || series[0].Points[3].Value != 4 { + t.Errorf("points: %+v", series[0].Points) + } + if _, err := client.QueryRange(context.Background(), "x", start, end, 0); err == nil { + t.Error("zero step must be rejected") + } +} + +func TestMetricNamesSortedCleanedFiltered(t *testing.T) { + var query string + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + query = r.URL.RawQuery + if r.Method != http.MethodGet || r.URL.Path != "/api/v1/label/__name__/values" { + t.Errorf("unexpected request %s %s", r.Method, r.URL.Path) + } + w.Write(fixture(t, "label-values.json")) + }) + names, err := client.MetricNames(context.Background(), []string{"node_cpu_seconds_total", "pve_up"}, time.Unix(1700000000, 0)) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(query, "match%5B%5D=node_cpu_seconds_total") || !strings.Contains(query, "match%5B%5D=pve_up") || !strings.Contains(query, "start=1700000000") { + t.Errorf("query: %q", query) + } + want := []string{"alpha_metric", "node_cpu_seconds_total", "okname", "zeta_metric"} + if strings.Join(names, ",") != strings.Join(want, ",") { + t.Errorf("names: %v", names) + } + if _, err := client.LabelValues(context.Background(), "bad label", nil, time.Time{}); err == nil { + t.Error("invalid label name must be rejected before any request") + } +} + +func TestTargets(t *testing.T) { + var query string + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + query = r.URL.RawQuery + w.Write(fixture(t, "targets.json")) + }) + targets, err := client.Targets(context.Background()) + if err != nil { + t.Fatal(err) + } + if query != "state=active" { + t.Errorf("query: %q", query) + } + if len(targets) != 2 || targets[0].Job != "node" || targets[0].Health != "up" || targets[0].ScrapeInterval != 15*time.Second || targets[0].ScrapeDuration != 12300*time.Microsecond || targets[0].LastScrape.IsZero() { + t.Fatalf("targets: %+v", targets) + } + down := targets[1] + if down.Health != "down" || !down.LastScrape.IsZero() || strings.Contains(down.LastError, "\n") || !strings.Contains(down.LastError, "connection refused") { + t.Errorf("down target: %+v", down) + } +} + +func TestBuildInfoAndRuntimeInfo(t *testing.T) { + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/v1/status/buildinfo": + w.Write(fixture(t, "buildinfo.json")) + case "/api/v1/status/runtimeinfo": + w.Write(fixture(t, "runtimeinfo.json")) + default: + http.NotFound(w, r) + } + }) + build, err := client.BuildInfo(context.Background()) + if err != nil || build.Version != "2.53.1" || build.Revision != "01234567" { + t.Fatalf("buildinfo: %v %+v", err, build) + } + runtime, err := client.RuntimeInfo(context.Background()) + if err != nil || !runtime.ReloadConfigSuccess || runtime.TimeSeriesCount != 48213 || runtime.StorageRetention != "15d" || runtime.StartTime.IsZero() { + t.Fatalf("runtimeinfo: %v %+v", err, runtime) + } +} + +func TestReady(t *testing.T) { + status := http.StatusOK + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/-/ready" { + t.Errorf("path %s", r.URL.Path) + } + w.WriteHeader(status) + w.Write([]byte("Prometheus Server is Ready.\n")) + }) + if err := client.Ready(context.Background()); err != nil { + t.Fatal(err) + } + status = http.StatusServiceUnavailable + if err := client.Ready(context.Background()); err == nil || !strings.Contains(err.Error(), "503") { + t.Fatalf("want 503 error, got %v", err) + } +} + +func TestBearerHeaderSentAndNeverInErrors(t *testing.T) { + const token = "s3cret-token-value" + var seen string + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + seen = r.Header.Get("Authorization") + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte("boom " + seen)) + })) + defer server.Close() + client, err := New(server.URL+"/prom", token) + if err != nil { + t.Fatal(err) + } + _, err = client.BuildInfo(context.Background()) + if seen != "Bearer "+token { + t.Fatalf("authorization header: %q", seen) + } + if err == nil || strings.Contains(err.Error(), token) || strings.Contains(client.String(), token) { + t.Fatalf("token leaked: %v", err) + } + if !strings.Contains(err.Error(), "HTTP 500") { + t.Errorf("want HTTP 500 in error, got %v", err) + } +} + +func TestEnvelopeErrorAndUnauthorized(t *testing.T) { + status := http.StatusBadRequest + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(status) + w.Write(fixture(t, "error-bad-data.json")) + }) + _, err := client.Query(context.Background(), "x(", time.Time{}) + if err == nil || !strings.Contains(err.Error(), "bad_data") || !strings.Contains(err.Error(), "parse error") || strings.Contains(err.Error(), "more") { + t.Fatalf("envelope error: %v", err) + } + status = http.StatusUnauthorized + _, err = client.Query(context.Background(), "x", time.Time{}) + if !errors.Is(err, ErrUnauthorized) { + t.Fatalf("want ErrUnauthorized, got %v", err) + } +} + +func TestNonJSONBodyAndRedirect(t *testing.T) { + hits := 0 + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + hits++ + switch r.URL.Path { + case "/api/v1/status/buildinfo": + w.Header().Set("Location", "/elsewhere") + w.WriteHeader(http.StatusFound) + case "/elsewhere": + t.Error("redirect must not be followed") + default: + w.Write([]byte("grafana")) + } + }) + _, err := client.Query(context.Background(), "x", time.Time{}) + if err == nil || !strings.Contains(err.Error(), "not a Prometheus API response") { + t.Fatalf("html body: %v", err) + } + _, err = client.BuildInfo(context.Background()) + if err == nil || !strings.Contains(err.Error(), "redirected") || hits != 2 { + t.Fatalf("redirect: %v (hits %d)", err, hits) + } +} + +func TestBodyLimit(t *testing.T) { + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + w.Write([]byte(`{"status":"success","data":"`)) + chunk := strings.Repeat("x", 64*1024) + for i := 0; i < 80; i++ { + w.Write([]byte(chunk)) + } + w.Write([]byte(`"}`)) + }) + _, err := client.BuildInfo(context.Background()) + if err == nil || !strings.Contains(err.Error(), "4 MiB") { + t.Fatalf("want size error, got %v", err) + } +} + +func TestContextCancel(t *testing.T) { + release := make(chan struct{}) + client := serve(t, func(w http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + }) + defer close(release) + ctx, cancel := context.WithCancel(context.Background()) + go func() { time.Sleep(50 * time.Millisecond); cancel() }() + started := time.Now() + _, err := client.BuildInfo(ctx) + if !errors.Is(err, context.Canceled) || time.Since(started) > time.Second { + t.Fatalf("want prompt cancellation, got %v after %s", err, time.Since(started)) + } +} + +func TestPrefixPathJoined(t *testing.T) { + var path string + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + path = r.URL.Path + w.Write(fixture(t, "buildinfo.json")) + })) + defer server.Close() + client, err := New(server.URL+"/prom/", "") + if err != nil { + t.Fatal(err) + } + if _, err := client.BuildInfo(context.Background()); err != nil { + t.Fatal(err) + } + if path != "/prom/api/v1/status/buildinfo" { + t.Errorf("path: %q", path) + } +} + +func TestTLSUsesSystemRootsOnly(t *testing.T) { + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write(fixture(t, "buildinfo.json")) })) + defer server.Close() + client, err := New(server.URL, "") + if err != nil { + t.Fatal(err) + } + if _, err := client.BuildInfo(context.Background()); err == nil || !strings.Contains(strings.ToLower(err.Error()), "certificate") { + t.Fatalf("self-signed certificate must fail with system roots: %v", err) + } +} + +func readAll(r *http.Request) (string, error) { + var b strings.Builder + buf := make([]byte, 4096) + for { + n, err := r.Body.Read(buf) + b.Write(buf[:n]) + if err != nil { + return b.String(), nil + } + } +} diff --git a/internal/promapi/decode.go b/internal/promapi/decode.go new file mode 100644 index 0000000..621689a --- /dev/null +++ b/internal/promapi/decode.go @@ -0,0 +1,87 @@ +package promapi + +import ( + "encoding/json" + "fmt" + "math" + "strconv" + "time" + + "systemdoc/internal/textutil" +) + +// envelope is the common /api/v1 response shape. +type envelope struct { + Status string `json:"status"` + Data json.RawMessage `json:"data"` + ErrorType string `json:"errorType"` + Error string `json:"error"` + Warnings []string `json:"warnings"` +} + +// parseValue decodes the API's string-encoded sample value. NaN and the +// infinities are reported as unknown rather than as numbers a chart would draw. +func parseValue(raw json.RawMessage) (float64, bool, error) { + var text string + if err := json.Unmarshal(raw, &text); err != nil { + return 0, false, fmt.Errorf("sample value is not a string: %w", err) + } + value, err := strconv.ParseFloat(text, 64) + if err != nil { + return 0, false, fmt.Errorf("sample value %q: %w", textutil.Clean(text), err) + } + if math.IsNaN(value) || math.IsInf(value, 0) { + return 0, false, nil + } + return value, true, nil +} + +// parsePair decodes a [unixSeconds, "value"] pair. +func parsePair(raw json.RawMessage) (Point, error) { + var pair []json.RawMessage + if err := json.Unmarshal(raw, &pair); err != nil || len(pair) != 2 { + return Point{}, fmt.Errorf("sample is not a [time, value] pair") + } + var seconds float64 + if err := json.Unmarshal(pair[0], &seconds); err != nil { + return Point{}, fmt.Errorf("sample time: %w", err) + } + value, known, err := parseValue(pair[1]) + if err != nil { + return Point{}, err + } + whole, fraction := math.Modf(seconds) + return Point{At: time.Unix(int64(whole), int64(fraction*1e9)).UTC(), Value: value, Known: known}, nil +} + +func cleanLabels(in map[string]string) Labels { + out := make(Labels, len(in)) + for key, value := range in { + out[textutil.Clean(key)] = textutil.Clean(value) + } + return out +} + +// parseTime accepts RFC 3339 with or without fractional seconds; Prometheus +// reports an unset time as the zero RFC 3339 instant, which stays zero here. +func parseTime(text string) time.Time { + for _, layout := range []string{time.RFC3339Nano, time.RFC3339} { + if parsed, err := time.Parse(layout, text); err == nil { + if parsed.Year() <= 1 { + return time.Time{} + } + return parsed + } + } + return time.Time{} +} + +// firstLine keeps server messages to one line for status rows. +func firstLine(text string) string { + for i, r := range text { + if r == '\n' || r == '\r' { + return text[:i] + } + } + return text +} diff --git a/internal/promapi/install.go b/internal/promapi/install.go new file mode 100644 index 0000000..065344d --- /dev/null +++ b/internal/promapi/install.go @@ -0,0 +1,317 @@ +package promapi + +// InstallContext describes the host so InstallSteps can pick the right +// commands. PackageManager is one of "apt-get", "dnf", "pacman", "zypper", +// "brew" or "" when none was found. DistroID is the ID from /etc/os-release +// ("debian", "ubuntu", "fedora", "rhel", "rocky", "almalinux", "arch", ...). +type InstallContext struct { + GOOS string + DistroID string + PackageManager string + Docker bool + Proxmox bool +} + +// InstallStep is one command or snippet in the setup guide. Runnable steps +// carry an Argv the dashboard may execute after review; show-only steps +// carry an Argv for display or a Note with file contents. +type InstallStep struct { + // Component groups steps: "prometheus", "node_exporter", "cadvisor", + // "pve-exporter" or "scrape-config". + Component string + // Title is a one-line description. + Title string + // Argv is the command without a shell; empty for pure notes. + Argv []string + // Sudo marks commands that need root. + Sudo bool + // Runnable marks commands the dashboard may execute after review. + Runnable bool + // Note holds context, a file path or file contents to show. + Note string +} + +// Component names used by InstallSteps. +const ( + ComponentPrometheus = "prometheus" + ComponentNodeExporter = "node_exporter" + ComponentCadvisor = "cadvisor" + ComponentPVEExporter = "pve-exporter" + ComponentScrapeConfig = "scrape-config" +) + +// CadvisorImage is the pinned cAdvisor image used by the Docker step. +const CadvisorImage = "ghcr.io/google/cadvisor:v0.52.1" + +// ScrapeSnippetNode is the prometheus.yml scrape job for node_exporter. +const ScrapeSnippetNode = ` - job_name: node + static_configs: + - targets: ["localhost:9100"] +` + +// ScrapeSnippetCadvisor is the prometheus.yml scrape job for cAdvisor. +const ScrapeSnippetCadvisor = ` - job_name: cadvisor + static_configs: + - targets: ["localhost:8080"] +` + +// ScrapeSnippetPVE is the prometheus.yml scrape job for pve-exporter. The +// target is the Proxmox node to query; the exporter itself listens on 9221. +const ScrapeSnippetPVE = ` - job_name: pve + metrics_path: /pve + params: + module: [default] + cluster: ["1"] + node: ["1"] + static_configs: + - targets: ["localhost"] + relabel_configs: + - source_labels: [__address__] + target_label: __param_target + - source_labels: [__param_target] + target_label: instance + - target_label: __address__ + replacement: 127.0.0.1:9221 +` + +// PVEConfigSnippet is /etc/prometheus/pve.yml for pve-exporter (mode 0600). +const PVEConfigSnippet = `default: + user: prometheus@pve + token_name: monitoring + token_value: "" + verify_ssl: false +` + +// PVEExporterUnit is a systemd unit for a pipx-installed pve-exporter. +const PVEExporterUnit = `[Unit] +Description=Prometheus Proxmox VE exporter +After=network-online.target + +[Service] +ExecStart=/root/.local/bin/pve_exporter --config.file /etc/prometheus/pve.yml --web.listen-address 127.0.0.1:9221 +Restart=on-failure + +[Install] +WantedBy=multi-user.target +` + +const ( + promConfigPath = "/etc/prometheus/prometheus.yml" + pveConfigPath = "/etc/prometheus/pve.yml" + dockerDesktopNote = "Docker Desktop containers cannot see the host PID and network namespaces, so node_exporter and cAdvisor in Docker report the Linux VM, not this Mac. Install them with Homebrew instead." +) + +func run(component, title string, sudo bool, note string, argv ...string) InstallStep { + return InstallStep{Component: component, Title: title, Argv: argv, Sudo: sudo, Runnable: true, Note: note} +} + +func show(component, title, note string, argv ...string) InstallStep { + return InstallStep{Component: component, Title: title, Argv: argv, Note: note} +} + +// InstallSteps returns the setup guide for the host described by c. Steps +// are ordered Prometheus server, node_exporter, cAdvisor (Docker hosts), +// pve-exporter, then scrape configuration. Runnable steps never need a +// shell; show-only steps are documentation. +func InstallSteps(c InstallContext) []InstallStep { + linux := c.GOOS == "linux" + var steps []InstallStep + steps = append(steps, serverSteps(c)...) + steps = append(steps, nodeExporterSteps(c)...) + if c.Docker { + if linux { + steps = append(steps, run(ComponentCadvisor, "Run cAdvisor in Docker", false, + "Exposes container metrics on 127.0.0.1:8080. Newer image tags exist; bump the pin after checking the release notes.", + "docker", "run", "-d", "--name", "cadvisor", "--restart", "unless-stopped", + "-p", "127.0.0.1:8080:8080", + "-v", "/:/rootfs:ro", "-v", "/var/run:/var/run:rw", "-v", "/sys:/sys:ro", "-v", "/var/lib/docker/:/var/lib/docker:ro", + CadvisorImage)) + } else { + steps = append(steps, show(ComponentCadvisor, "cAdvisor in Docker (Linux hosts)", dockerDesktopNote, + "docker", "run", "-d", "--name", "cadvisor", "--restart", "unless-stopped", + "-p", "127.0.0.1:8080:8080", + "-v", "/:/rootfs:ro", "-v", "/var/run:/var/run:rw", "-v", "/sys:/sys:ro", "-v", "/var/lib/docker/:/var/lib/docker:ro", + CadvisorImage)) + } + } + steps = append(steps, pveSteps(c)...) + steps = append(steps, scrapeSteps(c)...) + return steps +} + +func serverSteps(c InstallContext) []InstallStep { + const comp = ComponentPrometheus + var steps []InstallStep + switch c.PackageManager { + case "apt-get": + steps = append(steps, + run(comp, "Install Prometheus", true, "Debian's package ships a config that already scrapes localhost:9100.", + "apt-get", "install", "-y", "prometheus"), + run(comp, "Enable and start Prometheus", true, "", "systemctl", "enable", "--now", "prometheus"), + show(comp, "Configuration files", promConfigPath+" holds scrape jobs; /etc/default/prometheus holds command-line flags such as retention.")) + case "dnf": + if c.DistroID == "fedora" { + steps = append(steps, + run(comp, "Install Prometheus", true, "", "dnf", "install", "-y", "golang-github-prometheus"), + run(comp, "Enable and start Prometheus", true, "", "systemctl", "enable", "--now", "prometheus"), + show(comp, "Configuration file", promConfigPath)) + } else { + steps = append(steps, show(comp, "Prometheus server on RHEL-family systems", + "No packaged server is available. Run the upstream container (podman run -d --name prometheus -p 127.0.0.1:9090:9090 -v "+promConfigPath+":"+promConfigPath+":ro -v prometheus-data:/prometheus docker.io/prom/prometheus) or install the release tarball from prometheus.io/download.")) + } + case "pacman": + steps = append(steps, + run(comp, "Install Prometheus", true, "", "pacman", "-S", "--noconfirm", "--needed", "prometheus"), + run(comp, "Enable and start Prometheus", true, "", "systemctl", "enable", "--now", "prometheus"), + show(comp, "Configuration files", promConfigPath+" holds scrape jobs; /etc/conf.d/prometheus holds command-line flags.")) + case "zypper": + steps = append(steps, + run(comp, "Install Prometheus", true, "Package names follow the openSUSE server:monitoring repository.", + "zypper", "install", "-y", "golang-github-prometheus-prometheus"), + run(comp, "Enable and start Prometheus", true, "", "systemctl", "enable", "--now", "prometheus"), + show(comp, "Configuration file", promConfigPath)) + case "brew": + steps = append(steps, + run(comp, "Install Prometheus", false, "", "brew", "install", "prometheus"), + run(comp, "Start Prometheus as a service", false, "", "brew", "services", "start", "prometheus"), + show(comp, "Configuration files", "$(brew --prefix)/etc/prometheus.yml holds scrape jobs; $(brew --prefix)/etc/prometheus.args holds command-line flags.")) + } + if c.Docker && dockerServer(c) { + dockerNote := "Create " + promConfigPath + " first; the container reads it read-only." + if c.GOOS != "linux" { + dockerNote = "Create prometheus.yml first and adjust the bind mount to its path; Docker Desktop can only share directories listed in its file-sharing settings." + } + steps = append(steps, + run(comp, "Create a volume for Prometheus data", false, "", "docker", "volume", "create", "prometheus-data"), + run(comp, "Run Prometheus in Docker", false, dockerNote, + "docker", "run", "-d", "--name", "prometheus", "--restart", "unless-stopped", + "-p", "127.0.0.1:9090:9090", + "-v", promConfigPath+":"+promConfigPath+":ro", + "-v", "prometheus-data:/prometheus", + "prom/prometheus"), + show(comp, "Scrape configuration", "Minimal "+promConfigPath+":\n\nglobal:\n scrape_interval: 15s\nscrape_configs:\n"+ScrapeSnippetNode)) + } + if len(steps) == 0 { + steps = append(steps, show(comp, "Install Prometheus", + "No supported package manager or Docker was found. Download the release tarball from prometheus.io/download, unpack it, and run ./prometheus --config.file=prometheus.yml.")) + } + return steps +} + +// dockerServer reports whether the Docker server steps should be offered: +// when no package manager provides a runnable server. +func dockerServer(c InstallContext) bool { + switch c.PackageManager { + case "": + return true + case "dnf": + return c.DistroID != "fedora" + } + return false +} + +func nodeExporterSteps(c InstallContext) []InstallStep { + const comp = ComponentNodeExporter + var steps []InstallStep + switch c.PackageManager { + case "apt-get": + steps = append(steps, + run(comp, "Install node_exporter", true, "", "apt-get", "install", "-y", "prometheus-node-exporter"), + run(comp, "Enable and start node_exporter", true, "Listens on :9100.", "systemctl", "enable", "--now", "prometheus-node-exporter"), + show(comp, "Collector flags", "/etc/default/prometheus-node-exporter")) + case "dnf": + steps = append(steps, + run(comp, "Install node_exporter", true, "Fedora ships node-exporter; RHEL-family systems need EPEL 9 or 10.", + "dnf", "install", "-y", "node-exporter"), + run(comp, "Enable and start node_exporter", true, "Listens on :9100.", "systemctl", "enable", "--now", "node_exporter")) + case "pacman": + steps = append(steps, + run(comp, "Install node_exporter", true, "", "pacman", "-S", "--noconfirm", "--needed", "prometheus-node-exporter"), + run(comp, "Enable and start node_exporter", true, "Listens on :9100.", "systemctl", "enable", "--now", "prometheus-node-exporter"), + show(comp, "Collector flags", "/etc/conf.d/prometheus-node-exporter")) + case "zypper": + steps = append(steps, + run(comp, "Install node_exporter", true, "", "zypper", "install", "-y", "golang-github-prometheus-node_exporter"), + run(comp, "Enable and start node_exporter", true, "Listens on :9100.", "systemctl", "enable", "--now", "prometheus-node_exporter")) + case "brew": + steps = append(steps, + run(comp, "Install node_exporter", false, "", "brew", "install", "node_exporter"), + run(comp, "Start node_exporter as a service", false, "Listens on :9100.", "brew", "services", "start", "node_exporter"), + show(comp, "Collector flags", "$(brew --prefix)/etc/node_exporter.args")) + } + if c.Docker && (c.PackageManager == "" || c.GOOS != "linux") { + argv := []string{"docker", "run", "-d", "--name", "node-exporter", "--restart", "unless-stopped", + "--net=host", "--pid=host", "-v", "/:/host:ro,rslave", + "quay.io/prometheus/node-exporter:latest", "--path.rootfs=/host"} + if c.GOOS == "linux" { + steps = append(steps, run(comp, "Run node_exporter in Docker", false, "Uses the host network and PID namespaces so it sees the host, not the container.", argv...)) + } else { + steps = append(steps, show(comp, "node_exporter in Docker (Linux hosts)", dockerDesktopNote, argv...)) + } + } + if len(steps) == 0 { + steps = append(steps, show(comp, "Install node_exporter", + "Download the release tarball from prometheus.io/download and run ./node_exporter; it listens on :9100.")) + } + return steps +} + +func pveSteps(c InstallContext) []InstallStep { + const comp = ComponentPVEExporter + var steps []InstallStep + if !c.Proxmox { + return []InstallStep{ + show(comp, "Proxmox VE (not detected here)", "Run pve-exporter on a Proxmox node or point it at the PVE API. It needs a read-only API token and "+pveConfigPath+":\n\n"+PVEConfigSnippet), + show(comp, "Scrape job for pve-exporter", ScrapeSnippetPVE), + } + } + steps = append(steps, + run(comp, "Create a read-only API user", true, "", "pveum", "user", "add", "prometheus@pve", "--comment", "Prometheus metrics"), + run(comp, "Grant PVEAuditor to the user", true, "", "pveum", "acl", "modify", "/", "--users", "prometheus@pve", "--roles", "PVEAuditor"), + show(comp, "Create an API token", "Prints the token secret once; copy it into "+pveConfigPath+".", + "pveum", "user", "token", "add", "prometheus@pve", "monitoring", "--privsep", "1"), + run(comp, "Grant PVEAuditor to the token", true, "", "pveum", "acl", "modify", "/", "--tokens", "prometheus@pve!monitoring", "--roles", "PVEAuditor"), + show(comp, "Write "+pveConfigPath, "Mode 0600, owned by the user running the exporter:\n\n"+PVEConfigSnippet), + ) + if c.Docker { + steps = append(steps, run(comp, "Run pve-exporter in Docker", false, "Start after "+pveConfigPath+" exists.", + "docker", "run", "-d", "--init", "--name", "prometheus-pve-exporter", "--restart", "unless-stopped", + "-p", "127.0.0.1:9221:9221", + "-v", pveConfigPath+":"+pveConfigPath+":ro", + "prompve/prometheus-pve-exporter")) + } else { + steps = append(steps, + run(comp, "Install pipx", true, "", "apt-get", "install", "-y", "pipx"), + run(comp, "Install pve-exporter", false, "Installs pve_exporter into ~/.local/bin for the current user.", "pipx", "install", "prometheus-pve-exporter"), + show(comp, "Systemd unit for pve-exporter", "Save as /etc/systemd/system/prometheus-pve-exporter.service, adjust ExecStart to the pipx path, then systemctl enable --now prometheus-pve-exporter:\n\n"+PVEExporterUnit)) + } + steps = append(steps, show(comp, "Scrape job for pve-exporter", "Add to scrape_configs in "+promConfigPath+":\n\n"+ScrapeSnippetPVE)) + return steps +} + +func scrapeSteps(c InstallContext) []InstallStep { + const comp = ComponentScrapeConfig + configPath := promConfigPath + if c.PackageManager == "brew" { + configPath = "$(brew --prefix)/etc/prometheus.yml" + } + steps := []InstallStep{ + show(comp, "Scrape node_exporter", "Add to scrape_configs in "+configPath+":\n\n"+ScrapeSnippetNode), + } + if c.Docker { + steps = append(steps, show(comp, "Scrape cAdvisor", "Add to scrape_configs in "+configPath+":\n\n"+ScrapeSnippetCadvisor)) + } + switch { + case c.GOOS == "linux" && c.PackageManager != "": + steps = append(steps, + run(comp, "Check the configuration", false, "", "promtool", "check", "config", promConfigPath), + run(comp, "Reload Prometheus", true, "", "systemctl", "reload", "prometheus")) + case c.PackageManager == "brew": + steps = append(steps, run(comp, "Restart Prometheus", false, "Homebrew's service picks up the edited "+configPath+".", "brew", "services", "restart", "prometheus")) + case c.Docker && c.PackageManager == "": + steps = append(steps, + run(comp, "Check the configuration", false, "", "docker", "exec", "prometheus", "promtool", "check", "config", promConfigPath), + run(comp, "Reload Prometheus", false, "", "docker", "kill", "--signal=HUP", "prometheus")) + } + return steps +} diff --git a/internal/promapi/install_test.go b/internal/promapi/install_test.go new file mode 100644 index 0000000..e78927e --- /dev/null +++ b/internal/promapi/install_test.go @@ -0,0 +1,251 @@ +package promapi + +import ( + "strings" + "testing" +) + +func findSteps(steps []InstallStep, component string, argvContains ...string) []InstallStep { + var out []InstallStep + for _, s := range steps { + if s.Component != component { + continue + } + joined := strings.Join(s.Argv, " ") + ok := true + for _, want := range argvContains { + if !strings.Contains(joined, want) { + ok = false + } + } + if ok { + out = append(out, s) + } + } + return out +} + +func anyRunnable(steps []InstallStep) bool { + for _, s := range steps { + if s.Runnable { + return true + } + } + return false +} + +func TestInstallStepsContexts(t *testing.T) { + cases := []struct { + name string + ctx InstallContext + check func(t *testing.T, steps []InstallStep) + }{ + {"apt", InstallContext{GOOS: "linux", DistroID: "debian", PackageManager: "apt-get"}, func(t *testing.T, steps []InstallStep) { + for _, want := range [][]string{ + {ComponentPrometheus, "apt-get install -y prometheus"}, + {ComponentPrometheus, "systemctl enable --now prometheus"}, + {ComponentNodeExporter, "apt-get install -y prometheus-node-exporter"}, + {ComponentNodeExporter, "systemctl enable --now prometheus-node-exporter"}, + {ComponentScrapeConfig, "systemctl reload prometheus"}, + } { + got := findSteps(steps, want[0], want[1]) + if len(got) != 1 || !got[0].Sudo || !got[0].Runnable { + t.Errorf("apt: want one sudo runnable %q step in %s, got %+v", want[1], want[0], got) + } + } + if got := findSteps(steps, ComponentScrapeConfig, "promtool check config"); len(got) != 1 || !got[0].Runnable { + t.Errorf("apt: promtool step missing or not runnable: %+v", got) + } + if got := findSteps(steps, ComponentCadvisor); len(got) != 0 { + t.Errorf("apt without docker: unexpected cadvisor steps %+v", got) + } + if anyRunnable(findSteps(steps, ComponentPVEExporter)) { + t.Error("apt without proxmox: pve steps should be show-only") + } + }}, + {"brew", InstallContext{GOOS: "darwin", PackageManager: "brew"}, func(t *testing.T, steps []InstallStep) { + for _, s := range steps { + if s.Sudo { + t.Errorf("brew: unexpected sudo step %+v", s) + } + } + if got := findSteps(steps, ComponentPrometheus, "brew install prometheus"); len(got) != 1 || !got[0].Runnable { + t.Errorf("brew: install step %+v", got) + } + if got := findSteps(steps, ComponentNodeExporter, "brew services start node_exporter"); len(got) != 1 || !got[0].Runnable { + t.Errorf("brew: node_exporter service step %+v", got) + } + if got := findSteps(steps, ComponentScrapeConfig, "systemctl"); len(got) != 0 { + t.Errorf("brew: unexpected systemctl steps %+v", got) + } + }}, + {"docker-only linux", InstallContext{GOOS: "linux", DistroID: "ubuntu", Docker: true}, func(t *testing.T, steps []InstallStep) { + got := findSteps(steps, ComponentNodeExporter, "docker run", "node-exporter") + if len(got) != 1 || !got[0].Runnable || got[0].Sudo { + t.Errorf("docker linux: node-exporter step %+v", got) + } + if got := findSteps(steps, ComponentCadvisor, CadvisorImage); len(got) != 1 || !got[0].Runnable { + t.Errorf("docker linux: cadvisor step %+v", got) + } + if got := findSteps(steps, ComponentPrometheus, "docker run", "prom/prometheus"); len(got) != 1 || !got[0].Runnable { + t.Errorf("docker linux: prometheus step %+v", got) + } + if got := findSteps(steps, ComponentScrapeConfig, "systemctl"); len(got) != 0 { + t.Errorf("docker linux: unexpected systemctl steps %+v", got) + } + }}, + {"darwin docker", InstallContext{GOOS: "darwin", Docker: true}, func(t *testing.T, steps []InstallStep) { + got := findSteps(steps, ComponentNodeExporter, "docker run", "node-exporter") + if len(got) != 1 || got[0].Runnable || !strings.Contains(got[0].Note, "Docker Desktop") { + t.Errorf("darwin docker: node-exporter step %+v", got) + } + if got := findSteps(steps, ComponentCadvisor); len(got) != 1 || got[0].Runnable { + t.Errorf("darwin docker: cadvisor step %+v", got) + } + }}, + {"dnf fedora", InstallContext{GOOS: "linux", DistroID: "fedora", PackageManager: "dnf"}, func(t *testing.T, steps []InstallStep) { + if got := findSteps(steps, ComponentPrometheus, "dnf install -y golang-github-prometheus"); len(got) != 1 || !got[0].Runnable || !got[0].Sudo { + t.Errorf("fedora: server step %+v", got) + } + if got := findSteps(steps, ComponentNodeExporter, "dnf install -y node-exporter"); len(got) != 1 || !got[0].Runnable { + t.Errorf("fedora: node-exporter step %+v", got) + } + }}, + {"dnf rhel", InstallContext{GOOS: "linux", DistroID: "rhel", PackageManager: "dnf"}, func(t *testing.T, steps []InstallStep) { + if anyRunnable(findSteps(steps, ComponentPrometheus)) { + t.Errorf("rhel: server steps should be show-only: %+v", findSteps(steps, ComponentPrometheus)) + } + if len(findSteps(steps, ComponentPrometheus)) == 0 { + t.Error("rhel: no server guidance") + } + if got := findSteps(steps, ComponentNodeExporter, "dnf install -y node-exporter"); len(got) != 1 || !got[0].Runnable { + t.Errorf("rhel: node-exporter step %+v", got) + } + }}, + {"dnf rhel docker", InstallContext{GOOS: "linux", DistroID: "rocky", PackageManager: "dnf", Docker: true}, func(t *testing.T, steps []InstallStep) { + if got := findSteps(steps, ComponentPrometheus, "docker run", "prom/prometheus"); len(got) != 1 || !got[0].Runnable { + t.Errorf("rocky docker: prometheus docker step %+v", got) + } + }}, + {"proxmox", InstallContext{GOOS: "linux", DistroID: "debian", PackageManager: "apt-get", Proxmox: true}, func(t *testing.T, steps []InstallStep) { + pve := findSteps(steps, ComponentPVEExporter) + if len(pve) == 0 { + t.Fatal("proxmox: no pve-exporter steps") + } + if got := findSteps(steps, ComponentPVEExporter, "pveum user add prometheus@pve"); len(got) != 1 || !got[0].Runnable || !got[0].Sudo { + t.Errorf("proxmox: user add step %+v", got) + } + if got := findSteps(steps, ComponentPVEExporter, "pveum user token add"); len(got) != 1 || got[0].Runnable || got[0].Sudo { + t.Errorf("proxmox: token step must be show-only: %+v", got) + } + if got := findSteps(steps, ComponentPVEExporter, "--tokens", "prometheus@pve!monitoring"); len(got) != 1 || !got[0].Runnable { + t.Errorf("proxmox: token acl step %+v", got) + } + if got := findSteps(steps, ComponentPVEExporter, "pipx install prometheus-pve-exporter"); len(got) != 1 || !got[0].Runnable { + t.Errorf("proxmox: pipx step %+v", got) + } + found := false + for _, s := range pve { + if strings.Contains(s.Note, PVEConfigSnippet) { + found = true + } + } + if !found { + t.Error("proxmox: no step shows pve.yml") + } + last := steps[len(steps)-1] + if last.Component != ComponentScrapeConfig { + t.Errorf("proxmox: last step is %s, want scrape-config", last.Component) + } + }}, + {"proxmox docker", InstallContext{GOOS: "linux", DistroID: "debian", PackageManager: "apt-get", Proxmox: true, Docker: true}, func(t *testing.T, steps []InstallStep) { + if got := findSteps(steps, ComponentPVEExporter, "prompve/prometheus-pve-exporter"); len(got) != 1 || !got[0].Runnable { + t.Errorf("proxmox docker: exporter container step %+v", got) + } + if got := findSteps(steps, ComponentPVEExporter, "pipx"); len(got) != 0 { + t.Errorf("proxmox docker: unexpected pipx steps %+v", got) + } + }}, + {"nothing", InstallContext{GOOS: "linux", DistroID: "alpine"}, func(t *testing.T, steps []InstallStep) { + if len(steps) == 0 { + t.Fatal("nothing: no guidance at all") + } + for _, s := range steps { + if s.Runnable || s.Sudo { + t.Errorf("nothing: unexpected runnable step %+v", s) + } + } + }}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + c.check(t, InstallSteps(c.ctx)) + }) + } +} + +func TestInstallStepsHygiene(t *testing.T) { + contexts := []InstallContext{ + {GOOS: "linux", DistroID: "debian", PackageManager: "apt-get", Docker: true, Proxmox: true}, + {GOOS: "linux", DistroID: "fedora", PackageManager: "dnf", Docker: true}, + {GOOS: "linux", DistroID: "rhel", PackageManager: "dnf"}, + {GOOS: "linux", DistroID: "arch", PackageManager: "pacman", Docker: true}, + {GOOS: "linux", DistroID: "opensuse-tumbleweed", PackageManager: "zypper"}, + {GOOS: "linux", Docker: true}, + {GOOS: "linux"}, + {GOOS: "darwin", PackageManager: "brew", Docker: true}, + {GOOS: "darwin", Docker: true}, + {GOOS: "linux", DistroID: "debian", PackageManager: "apt-get", Proxmox: true}, + } + order := map[string]int{ComponentPrometheus: 0, ComponentNodeExporter: 1, ComponentCadvisor: 2, ComponentPVEExporter: 3, ComponentScrapeConfig: 4} + for _, ctx := range contexts { + steps := InstallSteps(ctx) + if len(steps) == 0 { + t.Errorf("%+v: no steps", ctx) + } + prev := -1 + for _, s := range steps { + rank, ok := order[s.Component] + if !ok { + t.Errorf("%+v: unknown component %q", ctx, s.Component) + } + if rank < prev { + t.Errorf("%+v: component %s out of order", ctx, s.Component) + } + prev = rank + if s.Title == "" { + t.Errorf("%+v: step without title: %+v", ctx, s) + } + if len(s.Argv) == 0 && s.Note == "" { + t.Errorf("%+v: step %q has neither argv nor note", ctx, s.Title) + } + if s.Sudo && !s.Runnable { + t.Errorf("%+v: sudo step %q is not runnable", ctx, s.Title) + } + if !s.Runnable { + continue + } + if len(s.Argv) == 0 { + t.Errorf("%+v: runnable step %q has empty argv", ctx, s.Title) + } + for _, a := range s.Argv { + if a == "" || strings.ContainsAny(a, "|;>") || strings.Contains(a, "&&") || strings.Contains(a, "$(") { + t.Errorf("%+v: runnable step %q has shell-like argv element %q", ctx, s.Title, a) + } + } + if s.Argv[0] == "sudo" { + t.Errorf("%+v: runnable step %q embeds sudo; use the Sudo flag", ctx, s.Title) + } + switch s.Argv[0] { + case "apt-get", "dnf", "pacman", "zypper", "systemctl", "pveum": + if !s.Sudo { + t.Errorf("%+v: step %q should be sudo", ctx, s.Title) + } + case "brew", "docker": + if s.Sudo { + t.Errorf("%+v: step %q should not be sudo", ctx, s.Title) + } + } + } + } +} diff --git a/internal/promapi/methods.go b/internal/promapi/methods.go new file mode 100644 index 0000000..4b835c5 --- /dev/null +++ b/internal/promapi/methods.go @@ -0,0 +1,245 @@ +package promapi + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/url" + "sort" + "strconv" + "time" + + "systemdoc/internal/textutil" +) + +// Ready reports whether the server answers /-/ready with success. The body is +// plain text and not stable, so only the status code is used. +func (c *Client) Ready(ctx context.Context) error { + status, _, err := c.fetch(ctx, http.MethodGet, "/-/ready", nil) + if err != nil { + return err + } + if status == http.StatusUnauthorized || status == http.StatusForbidden { + return fmt.Errorf("/-/ready: %w", ErrUnauthorized) + } + if status < 200 || status >= 300 { + return fmt.Errorf("/-/ready: HTTP %d from %s", status, c.String()) + } + return nil +} + +// BuildInfo returns the server version. It goes through the API prefix, so it +// also proves that the configured URL and token reach the API. +func (c *Client) BuildInfo(ctx context.Context) (BuildInfo, error) { + var raw struct { + Version string `json:"version"` + Revision string `json:"revision"` + GoVersion string `json:"goVersion"` + } + if err := c.do(ctx, http.MethodGet, "/api/v1/status/buildinfo", nil, &raw); err != nil { + return BuildInfo{}, err + } + return BuildInfo{Version: textutil.Clean(raw.Version), Revision: short(raw.Revision), GoVersion: raw.GoVersion}, nil +} + +func short(revision string) string { + if len(revision) > 8 { + return revision[:8] + } + return revision +} + +// RuntimeInfo returns the server's runtime state; missing fields stay zero. +func (c *Client) RuntimeInfo(ctx context.Context) (RuntimeInfo, error) { + var raw struct { + StartTime string `json:"startTime"` + ReloadConfigSuccess bool `json:"reloadConfigSuccess"` + LastConfigTime string `json:"lastConfigTime"` + TimeSeriesCount int64 `json:"timeSeriesCount"` + StorageRetention string `json:"storageRetention"` + } + if err := c.do(ctx, http.MethodGet, "/api/v1/status/runtimeinfo", nil, &raw); err != nil { + return RuntimeInfo{}, err + } + return RuntimeInfo{ + StartTime: parseTime(raw.StartTime), + ReloadConfigSuccess: raw.ReloadConfigSuccess, + LastConfigTime: parseTime(raw.LastConfigTime), + TimeSeriesCount: raw.TimeSeriesCount, + StorageRetention: textutil.Clean(raw.StorageRetention), + }, nil +} + +type queryResult struct { + ResultType string `json:"resultType"` + Result json.RawMessage `json:"result"` +} + +// Query evaluates expr at the given instant (zero means now). A scalar result +// becomes one sample with empty labels. +func (c *Client) Query(ctx context.Context, expr string, at time.Time) ([]Sample, error) { + form := url.Values{"query": {expr}} + if !at.IsZero() { + form.Set("time", strconv.FormatInt(at.Unix(), 10)) + } + var result queryResult + if err := c.do(ctx, http.MethodPost, "/api/v1/query", form, &result); err != nil { + return nil, err + } + switch result.ResultType { + case "vector": + var rows []struct { + Metric map[string]string `json:"metric"` + Value json.RawMessage `json:"value"` + } + if err := json.Unmarshal(result.Result, &rows); err != nil { + return nil, fmt.Errorf("/api/v1/query: vector result: %w", err) + } + samples := make([]Sample, 0, len(rows)) + for _, row := range rows { + point, err := parsePair(row.Value) + if err != nil { + return nil, fmt.Errorf("/api/v1/query: %w", err) + } + samples = append(samples, Sample{Labels: cleanLabels(row.Metric), At: point.At, Value: point.Value, Known: point.Known}) + } + return samples, nil + case "scalar": + point, err := parsePair(result.Result) + if err != nil { + return nil, fmt.Errorf("/api/v1/query: %w", err) + } + return []Sample{{Labels: Labels{}, At: point.At, Value: point.Value, Known: point.Known}}, nil + default: + return nil, fmt.Errorf("/api/v1/query: %s result is not supported here", result.ResultType) + } +} + +// QueryRange evaluates expr over [start, end] at the given step. The caller +// chooses the step so the point count stays proportional to the chart width. +func (c *Client) QueryRange(ctx context.Context, expr string, start, end time.Time, step time.Duration) ([]Series, error) { + if step <= 0 { + return nil, fmt.Errorf("/api/v1/query_range: step must be positive") + } + form := url.Values{ + "query": {expr}, + "start": {strconv.FormatInt(start.Unix(), 10)}, + "end": {strconv.FormatInt(end.Unix(), 10)}, + "step": {strconv.FormatFloat(step.Seconds(), 'f', -1, 64)}, + } + var result queryResult + if err := c.do(ctx, http.MethodPost, "/api/v1/query_range", form, &result); err != nil { + return nil, err + } + if result.ResultType != "matrix" { + return nil, fmt.Errorf("/api/v1/query_range: %s result is not a matrix", result.ResultType) + } + var rows []struct { + Metric map[string]string `json:"metric"` + Values []json.RawMessage `json:"values"` + } + if err := json.Unmarshal(result.Result, &rows); err != nil { + return nil, fmt.Errorf("/api/v1/query_range: matrix result: %w", err) + } + series := make([]Series, 0, len(rows)) + for _, row := range rows { + s := Series{Labels: cleanLabels(row.Metric), Points: make([]Point, 0, len(row.Values))} + for _, raw := range row.Values { + point, err := parsePair(raw) + if err != nil { + return nil, fmt.Errorf("/api/v1/query_range: %w", err) + } + s.Points = append(s.Points, point) + } + series = append(series, s) + } + return series, nil +} + +// MetricNames lists metric names with series matching any of match (bare +// metric names are valid selectors) since the given time. An empty match +// lists every name the server knows, which can be large; callers bound it. +func (c *Client) MetricNames(ctx context.Context, match []string, since time.Time) ([]string, error) { + values, err := c.LabelValues(ctx, "__name__", match, since) + if err != nil { + return nil, err + } + names := values[:0] + for _, name := range values { + if ValidMetricName(name) { + names = append(names, name) + } + } + return names, nil +} + +// LabelValues lists the values of one label, optionally restricted to series +// matching the selectors since the given time. Values are cleaned and sorted. +func (c *Client) LabelValues(ctx context.Context, label string, match []string, since time.Time) ([]string, error) { + if !ValidMetricName(label) { + return nil, fmt.Errorf("label name %q is not valid", label) + } + form := url.Values{} + for _, selector := range match { + form.Add("match[]", selector) + } + if !since.IsZero() { + form.Set("start", strconv.FormatInt(since.Unix(), 10)) + } + var values []string + if err := c.do(ctx, http.MethodGet, "/api/v1/label/"+label+"/values", form, &values); err != nil { + return nil, err + } + cleaned := make([]string, 0, len(values)) + for _, value := range values { + cleaned = append(cleaned, textutil.Clean(value)) + } + sort.Strings(cleaned) + return cleaned, nil +} + +// Targets lists active scrape targets. Dropped targets are not requested +// because a large service-discovery setup can report thousands of them. +func (c *Client) Targets(ctx context.Context) ([]Target, error) { + var raw struct { + Active []struct { + Labels map[string]string `json:"labels"` + ScrapePool string `json:"scrapePool"` + ScrapeURL string `json:"scrapeUrl"` + Health string `json:"health"` + LastError string `json:"lastError"` + LastScrape string `json:"lastScrape"` + LastScrapeDuration float64 `json:"lastScrapeDuration"` + ScrapeInterval string `json:"scrapeInterval"` + } `json:"activeTargets"` + } + if err := c.do(ctx, http.MethodGet, "/api/v1/targets", url.Values{"state": {"active"}}, &raw); err != nil { + return nil, err + } + targets := make([]Target, 0, len(raw.Active)) + for _, row := range raw.Active { + labels := cleanLabels(row.Labels) + + interval, _ := time.ParseDuration(row.ScrapeInterval) + targets = append(targets, Target{ + Job: labels["job"], + Instance: labels["instance"], + ScrapeURL: textutil.Clean(row.ScrapeURL), + Health: textutil.Clean(row.Health), + LastError: firstLine(textutil.Clean(row.LastError)), + Pool: textutil.Clean(row.ScrapePool), + LastScrape: parseTime(row.LastScrape), + ScrapeDuration: time.Duration(row.LastScrapeDuration * float64(time.Second)), + ScrapeInterval: interval, + Labels: labels, + }) + } + sort.Slice(targets, func(i, j int) bool { + if targets[i].Job != targets[j].Job { + return targets[i].Job < targets[j].Job + } + return targets[i].Instance < targets[j].Instance + }) + return targets, nil +} diff --git a/internal/promapi/testdata/buildinfo.json b/internal/promapi/testdata/buildinfo.json new file mode 100644 index 0000000..832d876 --- /dev/null +++ b/internal/promapi/testdata/buildinfo.json @@ -0,0 +1 @@ +{"status":"success","data":{"version":"2.53.1","revision":"0123456789abcdef","branch":"HEAD","buildUser":"root@x","buildDate":"20240701-12:00:00","goVersion":"go1.22.4"}} diff --git a/internal/promapi/testdata/error-bad-data.json b/internal/promapi/testdata/error-bad-data.json new file mode 100644 index 0000000..26c25e2 --- /dev/null +++ b/internal/promapi/testdata/error-bad-data.json @@ -0,0 +1 @@ +{"status":"error","errorType":"bad_data","error":"invalid parameter \"query\": 1:7: parse error: unexpected end of input\nmore"} diff --git a/internal/promapi/testdata/label-values.json b/internal/promapi/testdata/label-values.json new file mode 100644 index 0000000..e438467 --- /dev/null +++ b/internal/promapi/testdata/label-values.json @@ -0,0 +1 @@ +{"status":"success","data":["node_cpu_seconds_total","zeta_metric","1bad","ok\u001b[0mname","alpha_metric"]} diff --git a/internal/promapi/testdata/query-matrix.json b/internal/promapi/testdata/query-matrix.json new file mode 100644 index 0000000..199790b --- /dev/null +++ b/internal/promapi/testdata/query-matrix.json @@ -0,0 +1,4 @@ +{"status":"success","data":{"resultType":"matrix","result":[ + {"metric":{"cpu":"0"},"values":[[1700000000,"1"],[1700000015,"2"],[1700000030,"-Inf"],[1700000045,"4"]]}, + {"metric":{"cpu":"1"},"values":[[1700000000,"0.5"],[1700000015,"0.75"]]} +]}} diff --git a/internal/promapi/testdata/query-scalar.json b/internal/promapi/testdata/query-scalar.json new file mode 100644 index 0000000..c74cb93 --- /dev/null +++ b/internal/promapi/testdata/query-scalar.json @@ -0,0 +1 @@ +{"status":"success","data":{"resultType":"scalar","result":[1700000000,"42"]}} diff --git a/internal/promapi/testdata/query-vector.json b/internal/promapi/testdata/query-vector.json new file mode 100644 index 0000000..066a4dc --- /dev/null +++ b/internal/promapi/testdata/query-vector.json @@ -0,0 +1,5 @@ +{"status":"success","data":{"resultType":"vector","result":[ + {"metric":{"__name__":"node_load1","instance":"localhost:9100","job":"node"},"value":[1700000000.123,"1.25"]}, + {"metric":{"__name__":"node_load1","instance":"\u001b[31mevil\u001b[0m:9100","job":"node"},"value":[1700000000,"NaN"]}, + {"metric":{"__name__":"node_load1","instance":"third:9100","job":"node"},"value":[1700000000,"+Inf"]} +]}} diff --git a/internal/promapi/testdata/runtimeinfo.json b/internal/promapi/testdata/runtimeinfo.json new file mode 100644 index 0000000..463c599 --- /dev/null +++ b/internal/promapi/testdata/runtimeinfo.json @@ -0,0 +1 @@ +{"status":"success","data":{"startTime":"2024-05-01T08:00:00Z","CWD":"/prometheus","reloadConfigSuccess":true,"lastConfigTime":"2024-05-01T08:00:01Z","corruptionCount":0,"goroutineCount":40,"GOMAXPROCS":8,"GOMEMLIMIT":0,"GOGC":"75","GODEBUG":"","storageRetention":"15d","timeSeriesCount":48213}} diff --git a/internal/promapi/testdata/targets.json b/internal/promapi/testdata/targets.json new file mode 100644 index 0000000..100f68f --- /dev/null +++ b/internal/promapi/testdata/targets.json @@ -0,0 +1,4 @@ +{"status":"success","data":{"activeTargets":[ + {"discoveredLabels":{},"labels":{"instance":"localhost:9100","job":"node"},"scrapePool":"node","scrapeUrl":"http://localhost:9100/metrics","globalUrl":"http://localhost:9100/metrics","lastError":"","lastScrape":"2024-05-01T10:00:00.123456789Z","lastScrapeDuration":0.0123,"health":"up","scrapeInterval":"15s","scrapeTimeout":"10s"}, + {"discoveredLabels":{},"labels":{"instance":"pve:9221","job":"pve"},"scrapePool":"pve","scrapeUrl":"http://pve:9221/pve","globalUrl":"http://pve:9221/pve","lastError":"Get \"http://pve:9221/pve\": dial tcp: connection refused\nsecond line","lastScrape":"0001-01-01T00:00:00Z","lastScrapeDuration":0,"health":"down","scrapeInterval":"30s","scrapeTimeout":"10s"} +],"droppedTargets":[]}} diff --git a/internal/promapi/types.go b/internal/promapi/types.go new file mode 100644 index 0000000..786c537 --- /dev/null +++ b/internal/promapi/types.go @@ -0,0 +1,71 @@ +// Package promapi queries a Prometheus server's HTTP API with bounded requests +// and decodes the responses into plain values. PromQL construction belongs to +// the caller; this package owns transport, decoding and the panel catalog. +package promapi + +import ( + "regexp" + "time" +) + +// Labels is a metric's label set. Keys and values have been passed through +// textutil.Clean; callers still escape terminal markup before rendering. +type Labels map[string]string + +// Name returns the metric name carried in __name__. +func (l Labels) Name() string { return l["__name__"] } + +// Sample is one instant-query result. Known is false when the server reported +// NaN or an infinity, which are the API's encodings of undefined data. +type Sample struct { + Labels Labels + At time.Time + Value float64 + Known bool +} + +// Point is one value in a range-query series. +type Point struct { + At time.Time + Value float64 + Known bool +} + +// Series is one range-query result. +type Series struct { + Labels Labels + Points []Point +} + +// Target is one active scrape target. +type Target struct { + Job, Instance string + ScrapeURL string + Health string // up, down or unknown + LastError string + Pool string + LastScrape time.Time + ScrapeDuration time.Duration + ScrapeInterval time.Duration + Labels Labels +} + +// BuildInfo describes the server binary. +type BuildInfo struct { + Version, Revision, GoVersion string +} + +// RuntimeInfo is the server's self-reported runtime state. Prometheus warns +// that these fields may change between versions, so every field is optional. +type RuntimeInfo struct { + StartTime time.Time + ReloadConfigSuccess bool + LastConfigTime time.Time + TimeSeriesCount int64 + StorageRetention string +} + +var metricName = regexp.MustCompile(`^[a-zA-Z_:][a-zA-Z0-9_:]*$`) + +// ValidMetricName reports whether name is a syntactically valid metric name. +func ValidMetricName(name string) bool { return metricName.MatchString(name) } diff --git a/playground/README.md b/playground/README.md index a847181..598291a 100644 --- a/playground/README.md +++ b/playground/README.md @@ -69,3 +69,14 @@ New Docker group membership takes effect after logging out and back in. To use it immediately in your current terminal, run `newgrp docker`, then launch Systemdoc from that shell. Opening another terminal within an existing desktop session may still inherit the old groups. + +## Prometheus and exporters + +The compose project also starts Prometheus on `127.0.0.1:9090` with a two-hour +retention, a node-exporter and cAdvisor, scraped every 15 seconds from +`prometheus.yml`. Systemdoc detects the `prom/prometheus` container and opens +it in the Metrics suite (`6`) without configuration; **Actions → Open in +Metrics** on the container does the same. On macOS the node-exporter reports +the Docker Desktop virtual machine, not the Mac; the cAdvisor panels still +describe the playground containers. Stop Prometheus (`docker compose stop +prometheus`) to see the unreachable state and the setup view. diff --git a/playground/compose.yaml b/playground/compose.yaml index c22da75..6f2d11e 100644 --- a/playground/compose.yaml +++ b/playground/compose.yaml @@ -54,3 +54,33 @@ services: fi sleep 3 & wait $$! done + + prometheus: + <<: *defaults + image: prom/prometheus:v2.53.1 + ports: + - "127.0.0.1:9090:9090" + volumes: + - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro + command: + - --config.file=/etc/prometheus/prometheus.yml + - --storage.tsdb.retention.time=2h + + node-exporter: + <<: *defaults + image: quay.io/prometheus/node-exporter:v1.8.2 + pid: host + volumes: + - /:/host:ro,rslave + command: + - --path.rootfs=/host + + cadvisor: + <<: *defaults + image: ghcr.io/google/cadvisor:v0.52.1 + mem_limit: 160m + volumes: + - /:/rootfs:ro + - /var/run:/var/run:ro + - /sys:/sys:ro + - /var/lib/docker/:/var/lib/docker:ro diff --git a/playground/prometheus.yml b/playground/prometheus.yml new file mode 100644 index 0000000..831f82e --- /dev/null +++ b/playground/prometheus.yml @@ -0,0 +1,15 @@ +# Scrape configuration for the playground Prometheus. Targets are the +# node-exporter and cAdvisor containers on the compose network. +global: + scrape_interval: 15s + +scrape_configs: + - job_name: prometheus + static_configs: + - targets: ['localhost:9090'] + - job_name: node + static_configs: + - targets: ['node-exporter:9100'] + - job_name: cadvisor + static_configs: + - targets: ['cadvisor:8080']