Skip to content
worldlinePublic

About

Point starcubes at a SQL database and get a fast, filterable timeseries of what happens in it

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

245 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

starcubes

Point starcubes at a SQL database and get a fast, filterable timeseries of what happens in it — without exporting a single row to a spreadsheet.

  • Configure a source in a form, not in SQL. Base table, watermark column, a row per fact column — that is the whole model. Save it as a profile and the next install is one click.
  • Built for volume. The 3.9 million transactions in the screenshots below sit in 345 MB: smallint-keyed dimensions, range partitions, BRIN on the time column, COPY FROM on the load path. Optional pre-aggregation keeps charts over months of history fast — and outlives the raw rows once retention starts disposing of them.
  • One self-hosted container. One image, one Postgres, one login — and MIT-licensed, so there is no paid tier holding the feature you came for.

The chart

What it's for

You have an operational database — payments, orders, requests, events — and a question about it: how many payments per hour, by status, cards only, over the last thirty days? You get an answer, and it immediately raises the next question: what did the failures look like, which merchant, was yesterday like the week before?

Today each of those answers costs a small ritual. Someone writes the GROUP BY, exports the result to CSV, pivots it in a spreadsheet, draws a chart, and pastes a screenshot into a mail. It works exactly once. The spreadsheet gives up long before the 3.9 million rows in the screenshots below, and what lands in the mail is a still image of yesterday that answers no follow-up. Write Python, of course, and a notebook will draw you whatever you like — this is for everyone who doesn't.

starcubes turns that ritual into a page. You configure the source once; after that every question is a click, and every view you find is a URL you can send. It is the window Splunk or Kibana give you onto your log lines, pointed instead at the tables your business actually runs on. Underneath, it reads your source on a high watermark, keeps a compact star schema on Postgres, and serves a stacked-bar timeseries straight off it.

Five minutes to a chart

./docker/build.sh                                    # podman by default
podman run -p 8080:8080 \
  -e STARCUBES_POSTGRES_DSN=postgresql://user:pw@host:5432/starcubes \
  starcubes:latest

The first boot applies the schema and creates an admin / admin login — change it. Then, at http://localhost:8080/sources/:

  1. New Datasource — name, dialect, connection URL.
  2. Apply a profile, or fill in the base table, watermark column and fact columns yourself.
  3. Run once, and open the chart.

The datasource list

A profile is a whole datasource configuration — fact columns, dimensions, indices, measures, group dimensions — as one YAML file, deliberately without any credentials. Download one, hand it to a colleague, apply it to a fresh install.

The profile catalogue

Prefer the command line? manage.py seed_example, manage.py etl_run, manage.py create_indices do the same thing — see docs/internals.md.

What it's like to use

Slice it without writing SQL

Group by any dimension, filter on any other. Filters nest with AND/OR, take is in / is not in / contains / starts with, and live in the URL — so a view you found is a link you can send.

Filters

Click a bar, see what is in it

Every bar breaks down into its dimensions, and from there into the individual primary keys behind it. The same panel starts an alarm on exactly the slice you are looking at.

Drill-down

Set a threshold on the view you are in

The bell carries the chart's active filter into a new alarm as its context. Alarms are evaluated by the ETL itself, once after every load, over the trailing window at the data's own clock — not the wall clock. Breaches are recorded as episodes with a first-breach and a closed watermark, so a threshold that flapped for an hour is one row, not sixty.

Alarms

Ask the database directly

A read-only SQL console against the datasource's own schema, with a single-statement guard, a cancel button, per-user history and CSV export.

Query console

Watch the loads

Start an incremental run, override its window, run it forever at an interval, or kill it. Every run writes a row, and its output lands in one filterable log.

ETL commands

And see what it costs you on disk, per table and per partition.

Storage

Features

Ingest Oracle, Postgres, MySQL, SQL Server via SQLAlchemy URLs
Log files — measures and dimensions are named groups of a regex, over file://, sftp://, https:// or a command
Rotated generations read in order, .gz/.bz2/.xz unpacked on the fly
The columns offer themselves: probe the log, tick the groups, done
Incremental loads on a high watermark, resumable, killable
Late-arrival overlap for sources that publish out of order
Auto-dimensions — new values discovered and keyed during the load
Three-stage extract / transform / load with COPY FROM
Storage One Postgres schema per datasource, star-shaped
Smallint dimension keys, up to 32 767 values per dimension
Day / week / month range partitions, gap-filled automatically
BRIN on the time column; indices built after the first bulk load
Explore Stacked-bar timeseries with an interval ladder and follow mode
Nested filters, grouped dimensions, drill-down to primary keys
Quantile bands, error bars, candlesticks for non-count measures
Server-side plot cache for frozen, past-only windows
Optional pre-aggregation — charts over months of history stay fast, and outlive retention
Read-only SQL console with history and CSV export
Operate Threshold alarms with firing episodes, evaluated by the ETL
Gap audit: bucket counts, PK set-diff, and per-id detail
Datasource profiles as portable YAML
Password login, optional Keycloak / OIDC SSO

How it works

How starcubes works: a SQL database or a log file is read incrementally on a high watermark by a three-stage ETL engine, which loads a compact star schema into Postgres; the chart, alarms and query console read that schema, and filters live in the URL.

Each datasource gets its own Postgres schema, because two sources rarely have compatible shapes. The fact table is range-partitioned on its time column and carries smallint foreign keys into per-dimension tables — which is where the 93 bytes per row come from. Dimensions come in three flavours: imported from a source lookup table, inlined as CSV in the metadata, or discovered by the ETL as it goes.

The full picture — pipeline stages, watermark arithmetic, partition lifecycle, schema decisions — is in docs/internals.md.

Requirements

  • Postgres for the warehouse.
  • A SQL source: Oracle, Postgres, MySQL or SQL Server. Oracle is the dialect in production use today; the ETL's window literals are still Oracle-shaped and need the bind-parameter conversion tracked in docs/backlog.md before another dialect is loaded.
  • Podman or Docker, or Python 3.12 with requirements.txt.

Documentation

HOWTO.md Driving the GUI, and every configuration field explained
docs/internals.md Architecture, repo layout, CLI quick start
docs/star-schema.md Domain vocabulary and schema decisions
docs/etl-engine.md The three-stage pipeline in detail
docs/logfile-source.md Log files as a source: transports, the pattern, and what a log cannot give
docs/etl-audit.md The three audit modes
docs/alarm.md Alarm evaluation and firing episodes
docs/chart-app.md The chart UI
docs/chart-rendering.md Plot types and partial bins
docs/sources-app.md The sources GUI
docs/source-profiles.md The profile format
docs/deployment-config.md Container, env vars, SSO
docs/backlog.md Known unfinished work

Acknowledgements

License

MIT — Copyright © 2026 Worldline. See LICENSE.

starcubes ships one third-party component, @hgruber/timeseries.js, which is MIT — Copyright © 2022–2026 Holger Gruber. Its notice travels with the code: THIRD-PARTY-NOTICES reproduces it in full, and the container image carries the package's own LICENSE next to the bundle.

About

Point starcubes at a SQL database and get a fast, filterable timeseries of what happens in it

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages