Occupy isolated execution grants on one host.
Check readiness, ensure a grant, project a workspace journal, observe guest changes, run processes, control execution under a fence, and revoke. Isolation stays behind that contract.
This repository is two things that share one interface:
- A Go module your application imports (
Host,Env). - An installable Linux host process that serves that same interface.
Your application is the authority for files. The grant is a projection: if it disappears, you still have whatever you already stored.
Install the module into the project that will start grants, push files, and run shells and commands.
go get github.com/sudosylabs/execenv@v1.2.3Client and host need not share a release. The wire is the protobuf schema in
proto/execenv/v1, spoken with Connect over HTTP/2, so
callers in other languages can generate a client from it. When it connects,
a client names the features it needs; a host that lacks one is refused at
once with ErrFeature, before any grant is occupied. The host separately
verifies the guest helper and its native journal before admitting an
environment.
Import the occupancy API and, in production, the remote dialer:
import (
"github.com/sudosylabs/execenv"
"github.com/sudosylabs/execenv/remote"
)Do not import the operator tool, the daemon package, or the isolation adapter. Those are for the host process, not for grant callers.
The in-memory adapter implements the same Host as production. Use it in
unit tests and on a laptop. It is not isolated; production TLS refuses it.
package main
import (
"context"
"fmt"
"log"
"github.com/sudosylabs/execenv"
"github.com/sudosylabs/execenv/memory"
)
func main() {
ctx := context.Background()
host, err := memory.New(memory.Config{
Images: []execenv.Image{"python", "default"},
Capacity: execenv.DefaultGrant.Times(4), // room for four default grants
})
if err != nil {
log.Fatal(err)
}
run(ctx, host)
}
func run(ctx context.Context, host execenv.Host) {
rep, err := host.Ready(ctx)
if err != nil {
log.Fatal(err)
}
fmt.Printf("usable=%v images=%v networks=%v free memory=%d release=%s\n",
rep.Usable, rep.Images, rep.Networks, rep.MemoryBytes.Free, rep.Release)
}Every method below takes that host. Swap memory.New for remote.New
when you have a real host; the rest of the code does not change.
package runner
import (
"crypto/tls"
"crypto/x509"
"errors"
"os"
"github.com/sudosylabs/execenv"
"github.com/sudosylabs/execenv/remote"
)
func Dial() (execenv.Host, error) {
pem, err := os.ReadFile(os.Getenv("EXECENV_CA"))
if err != nil {
return nil, err
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, errors.New("ca pem")
}
return remote.New(remote.Config{
Address: os.Getenv("EXECENV_ADDR"), // "host.example:8443"
Security: remote.SecurityTLS,
Token: []byte(os.Getenv("EXECENV_TOKEN")),
TLS: &tls.Config{
RootCAs: pool,
ServerName: os.Getenv("EXECENV_SERVER_NAME"),
},
})
}remote.New dials, sends the token as a bearer credential, and checks that
the host offers the features this client needs. A verified client
certificate may replace the token when the host enables mutual TLS. Keep one
client per host for the life of the process: it holds two HTTP/2
connections, one for control, processes and logs, one for file content, so
a large upload never delays a keystroke. Timeout bounds dialing and the
first exchange; OperationTimeout caps every call and defaults to 30
seconds.
Cleartext is loopback-only:
host, err := remote.New(remote.Config{
Address: "127.0.0.1:8443",
Security: remote.SecurityInsecureLocal,
Token: []byte("dev-token"),
})A non-loopback insecure address is rejected.
execenv.RequireIsolated(host) fails closed when the adapter is not
isolated. Use it in production composition so the in-memory host cannot
be selected by accident.
rep, err := host.Ready(ctx)
if err != nil {
return err
}
if !rep.Usable {
return execenv.ErrUnavailable
}Images is the catalog ids on that machine. Networks is the network modes
the host offers. CPUMillis, MemoryBytes and DiskBytes are its
capacity, each with Total, Reserved and Free: every grant reserves its
whole size from the moment it starts until it is revoked or lost. CPU totals
already include the operator's overcommit; memory is never overcommitted.
Defaults is what a grant gets when its Spec leaves a size unset, and
GrantMax the most one grant may ask for. Grants is how many grants the
host holds now, and Lost how many it has lost since it started (see Lost
grants). Release is the host's release, for display. One client talks to
one host. If you have several machines, you choose among them, by the
capacity each reports.
Usable false means the host takes no new grants; Reason says why.
execenv.ReasonDraining means its operator is draining it before a
being taken out of service: it keeps serving the grants it holds, and Ensure of a new id
returns ErrDraining. Place new grants elsewhere until it is usable again.
Isolation is the boundary class the host provides:
| Value | Boundary |
|---|---|
execenv.IsolationVM ("vm") |
each grant is a hardware-virtualized microVM with its own guest kernel |
execenv.IsolationNone ("none") |
in-memory adapter; tests only |
You allocate the grant id and store it. It is not derived by execenv.
env, err := host.Ensure(ctx, execenv.Spec{
ID: "job-7f3a",
Image: "python",
Network: execenv.NetworkNone,
// Zero fields use the host's Defaults.
Resources: execenv.Resources{CPUMillis: 2000, MemoryBytes: 2 << 30, DiskBytes: 8 << 30},
// Revoke the grant two hours after it starts, and freeze it after
// fifteen idle minutes.
WallTimeout: 2 * time.Hour,
IdleTimeout: 15 * time.Minute,
IdleAction: execenv.IdleFreeze,
Labels: map[string]string{"queue": "nightly"},
Limits: execenv.Limits{
DiskBytes: execenv.TokenBucket{Rate: 64 << 20, Burst: 256 << 20},
},
})
var short *execenv.CapacityError
if errors.As(err, &short) {
// short.Resource is what this host lacks; try another host.
}
if err != nil {
return err
}| Situation | Result |
|---|---|
| New id | occupies a grant |
| Same id and spec (Limits aside) | returns the existing occupancy |
| Same id, any other field different | ErrConflict |
| Image not installed on the host | ErrUnknownImage |
Does not fit in what is free, or above GrantMax |
*CapacityError, matching ErrCapacity, naming the short resource |
| Size below the minimums (100 CPU millis, 128 MiB, 64 MiB disk) | ErrInvalid |
| Host draining, id not held | ErrDraining |
| Id was lost and not yet revoked | ErrLost |
| Allowlist requested, host has no dests | ErrNetwork |
| Guest helper cannot serve the journal | ErrUnavailable; the guest is destroyed |
| Guest kernel lacks process events | ErrUnavailable; the guest is destroyed |
Ids and image names are 1–64 characters in [A-Za-z0-9._-:]. Concurrent
identical Ensure calls for one new id coalesce into one launch. Ensure does
not download images; occupying a new grant starts its guest, and it never
queues: a grant that does not fit is refused at once.
The guest sees enough virtual CPUs to use its CPUMillis and is throttled
to exactly that; MemoryBytes is its memory; DiskBytes is its workspace
disk, shared by the projected tree and the shell's scratch home.
A grant is busy while anything is attached to one of its processes and
after any call on it other than Observe and ObserveProcesses: reading
what the grant did does not keep it awake. Once idle for IdleTimeout the host applies
IdleAction: IdleRevoke revokes it; IdleFreeze freezes it as a fenced
freeze would, keeping the caller's fence revision, so ControlStatus shows
frozen and the caller thaws it with its next revision. IdleSuspend
freezes it the same way, then saves its running machine to the host's disk
and stops it: only its disk stays reserved, so its CPU and memory go to
other grants. A grant you froze yourself is suspended the same way once it
has been idle for IdleTimeout (your freeze counts as its last call), and
ControlStatus still shows your own frozen receipt. The caller's thaw
restores the machine where it stopped, processes running and output
resuming at the same offsets; while the host has no room for it the thaw
fails with ErrCapacity and the grant stays suspended. The in-memory host
models a suspend: the grant stays frozen with its processes, reports
suspended, and reserves only its disk until a thaw finds room again. WallTimeout revokes the grant that long
after it started, busy or not. Labels are the caller's, up to 16 pairs and
512 bytes; the host never reads them.
Limits throttle the grant's workspace disk (operations and bytes per
second) and its network device (bytes per second each way) with token
buckets: a rate, and a burst allowed once. A zero rate is no limit. Change
them while the grant runs with UpdateLimits, under the current fence:
err := env.UpdateLimits(ctx, fence, execenv.Limits{
DiskOps: execenv.TokenBucket{Rate: 2000},
})Every Env has the whole contract below: fenced control, journal projection,
semantic observations, processes and revoke. There is nothing to discover
and no type assertion to make.
Epoch() identifies this exact occupancy and survives reconnect, re-Ensure
and a restart of the host daemon. Reconstruction generates a fresh epoch;
the host never recovers an old epoch without its highest fence.
Grants outlive the daemon. Each grant's machine runs on its own, and the
daemon records each grant's spec, epoch, control receipt and machine in a
state file it replaces atomically. A restarted daemon adopts every grant
whose machine still runs: same epoch, same fence, same processes. Each
process's output resumes from the offsets you held, and the process log
and observations you had not acknowledged are still there. A grant frozen
when the daemon went away is still frozen; it thaws under its next fence.
A grant whose machine ended while no daemon ran is lost after the restart
(ErrLost until you revoke it). The in-memory host's grants end with its
process.
fence := execenv.ProjectionFence{
ExecutionGrantID: env.ID(),
EnvironmentEpoch: env.Epoch(),
FenceRevision: 7,
}
receipt, err := env.Control(ctx, execenv.ControlRequest{Fence: fence, State: execenv.ControlFrozen})
if err != nil {
return err
}
// Match receipt.Fence to the caller's current durable authorization before
// acknowledging the effect or opening any caller-side gate.
_ = receiptThe fence contains only the opaque grant ID, epoch, and positive revision
(maximum 2^53−1). Its meaning belongs to the caller. The host records the highest
intent before attempting an effect. Older requests and reuse of a revision with
a different state conflict. Exact successful retries return the same receipt;
exact failed retries attempt the effect again. ControlStatus reports the highest
intent and whether its effect was confirmed, including after a lost response.
Before the first fenced command its revision is zero, so enter the protocol with
a running request before the first projection call.
Control accepts running, frozen, and revoked. frozen blocks new I/O and
preserves the existing processes, their output and the tree; a later running
continues the same processes. The isolated host acknowledges frozen only after every monitor
thread is confirmed stopped; a sent signal alone is insufficient. Agent I/O
gating is separate from host execution suspension. A failed thaw keeps the local
gate closed and pauses the instance again. Any error requires reconciliation; it
is never evidence that the requested state was reached.
Revocation is terminal even if destruction initially fails. Failed destruction
keeps the grant's resources reserved until a successful retry. Administrative
Host.Revoke(id) destroys the current grant; Env.Revoke targets only the
captured epoch and cannot destroy a replacement. Process reads bind the
captured epoch; starts, input, resize and signals carry the fence.
When many grants start from the same tree, send it to each host once, as a
seed, instead of to every grant. Build an archive with package archive;
its digest names the tree, whatever order or compression it was written in:
var buf bytes.Buffer
w, err := archive.NewWriter(&buf, archive.CodecZstd)
if err != nil {
return err
}
_ = w.Dir("src")
_ = w.File("src/main.py", false, int64(len(main)), bytes.NewReader(main))
digest, err := w.Close()
if err != nil {
return err
}
if _, err := host.PutSeed(ctx, digest, &buf); err != nil {
return err
}
env, err := host.Ensure(ctx, execenv.Spec{ID: "job-7f3a", Image: "python", Seed: digest})The host unpacks the archive once, refusing anything but directories and
regular files and any tree whose digest is not the one named, and keeps it as
a ready workspace disk. A grant created with the seed gets its own copy of
that disk: a reflink where the host's filesystem shares blocks, a local
block copy elsewhere, and nothing over the network. Putting a seed the host
already has returns at once. Seeds lists them and DeleteSeed removes one;
grants already created keep their copy. An unknown seed in a Spec is
ErrNotFound.
Initialize a seeded grant with its full manifest as usual. For a file the
seed already holds, give a reference with its size and digest and no
transfer id; upload only what differs. Paths the seed holds and the manifest
does not name are removed. A file neither seeded nor uploaded fails the whole
initialization with a *MissingContentError naming its paths, and changes
nothing.
Callers that keep no durable copy of their own can work on the workspace directly. Reads name only the grant; changes carry its fence:
info, err := env.WriteFile(ctx, fence, "out/report.txt", body,
execenv.WriteOptions{ExpectedSHA256: execenv.ExpectAbsent})
entries, err := env.List(ctx, "out", 2, true) // with digests
rc, err := env.ReadFile(ctx, "out/report.txt", 0)
err = env.Import(ctx, fence, "", archiveReader) // all entries or none
rc, err = env.Export(ctx, []string{"out"}, "zstd")WriteFile lands the file only once its whole body has arrived;
ExpectedSHA256 makes it conditional on what is there now (ExpectAbsent:
nothing). MakeDirectory, Move and Remove do what their names say;
Remove needs recursive for a directory that is not empty. These are
guest changes like any other: once a projection is initialized they appear
in Observe. Bodies travel on the content connection.
A large write can survive a broken connection. Give it a TransferID of
your choosing: the bytes of a stream that breaks are kept under it, and you
carry on from where the host stopped:
opts := execenv.WriteOptions{TransferID: "upload-42"}
_, err := env.WriteFile(ctx, fence, "data.bin", body, opts)
if err != nil {
opts.Offset, err = env.TransferOffset(ctx, "upload-42")
_, _ = body.Seek(opts.Offset, io.SeekStart)
_, err = env.WriteFile(ctx, fence, "data.bin", body, opts)
}A resume whose Offset is not what TransferOffset reports is
ErrConflict: after a broken connection, bytes still in flight may land
after you asked, so ask again and resume from there. When your own
source fails, WriteFile returns only once the host has kept everything
sent before the failure. The file still lands only once the whole body is there, and
then the transfer is used up. Kept bytes expire an hour after they last
grew.
Your application owns a journal of the workspace. The grant receives it as consecutive journal entries, each bound to the current fence.
body, err := env.UploadProjectionContent(ctx, fence, strings.NewReader("print('hello')\n"))
if err != nil {
return err
}
_, err = env.InitializeProjection(ctx, execenv.ProjectionInitialization{
Fence: fence,
MutationID: "init-job-7f3a",
WorkspaceCursor: 41, // your journal position this snapshot reflects
Entries: []execenv.ProjectionEntry{
{EntryID: "e-src", Kind: execenv.ProjectionDirectory, Path: "src"},
{EntryID: "e-main", Kind: execenv.ProjectionFile, Path: "src/main.py", ContentVersion: "v1", Content: &body},
},
// Optional: leave dependency installs and build output out of Observe.
// You choose the file; the host hardcodes none.
Ignore: &execenv.IgnoreRules{
File: "your-ignore-file",
Fallback: []string{"node_modules/", "__pycache__/", "*.pyc"},
},
})
if err != nil {
return err
}
body, err = env.UploadProjectionContent(ctx, fence, strings.NewReader("print('hello world')\n"))
if err != nil {
return err
}
receipt, err := env.ApplyProjection(ctx, execenv.ProjectionApply{
Fence: fence,
MutationID: "save-42",
FromWorkspaceCursor: 41,
ThroughWorkspaceCursor: 42,
ExpectedHostCursor: hostCursor, // from your last Observe page
Mutations: []execenv.ProjectionMutation{{
JournalCursor: 42,
EntryID: "e-main",
Operation: execenv.ProjectionWrite,
Kind: execenv.ProjectionFile,
Path: "src/main.py",
ExpectedContentVersion: "v1",
ResultingContentVersion: "v2",
Content: &body,
}},
})Initialize once, before starting the first process, into an empty workspace.
Every Apply binds an exact grant/epoch/revision fence, mutation ID, from/through
journal cursors, expected host cursor, stable entry IDs, kinds, and opaque
content versions. The host compares versions for equality and never parses
them. Batches contain 1–128 consecutive mutations and at most 256 KiB of
metadata. Directory moves preserve all descendant identities and versions;
deleting a nonempty directory requires Recursive. Exact retries return their
original receipt; changing a used mutation ID conflicts. The epoch retains at
most 65,536 receipts and refuses additional mutations rather than forgetting
retry bindings.
An Apply whose expected host cursor is behind an unobserved guest change, or
that would overwrite a guest change on an affected path, fails with
ErrConflict before any path changes. Observe first, decide, then apply.
Freeze and thaw preserve the projection; neither requires a new snapshot.
To make your journal win over a guest change you will not accept, set
Overwrite on a create, write or delete. It replaces whatever the guest
left at that path and under it, unconfirmed changes included, and still
checks the mutation against your journal (entry, kind, expected version);
the Apply still names the current host cursor, so you have seen every
change it replaces. Every retained observation whose paths all lie in the
replaced subtree is discarded: the receipt's Discarded lists their host
sequences in order (an exact retry returns the same list), and later pages
carry each one's DiscardedBy, the journal cursor of the overwrite.
Acknowledge them without recording them; they cannot be confirmed, and
their content reads as ErrStale. An overwriting directory create replaces
an ignored directory at its path instead of adopting it. Overwrite is still
ErrConflict when the guest moved something across the path's boundary (a
projected path out from under it, or one into it), when a retained
observation names a path inside and one outside, or when the path's parent
is no longer a directory; resolve those first. Moves and renames cannot
overwrite.
receipt, err := env.ApplyProjection(ctx, execenv.ProjectionApply{
Fence: fence, MutationID: "restore-43",
FromWorkspaceCursor: 42, ThroughWorkspaceCursor: 43,
ExpectedHostCursor: page.HostCursor,
Mutations: []execenv.ProjectionMutation{{
JournalCursor: 43, EntryID: "e-main", Operation: execenv.ProjectionWrite,
Kind: execenv.ProjectionFile, Path: "src/main.py",
ExpectedContentVersion: "v2", ResultingContentVersion: "v3",
Content: &body, Overwrite: true,
}},
})
for _, sequence := range receipt.Discarded {
recordDiscarded(sequence) // then acknowledge as usual
}File contents travel separately as immutable, scoped SHA-256 references. A
body streams to the grant's workspace disk as it arrives and is never held in
memory, so no size limit applies to it but the disk itself: a body or a tree
that would not fit is ErrQuota, and nothing is left half-written. Unconsumed
content expires after five minutes, with at most 512 handles per environment. Over the remote wire each
upload is one stream of 64-KiB messages on the content connection; one that
breaks off leaves nothing behind. References cannot name
paths, URLs, credentials, another environment or observation content.
Paths are POSIX-relative: no leading /, no .., no NUL, UTF-8.
Observe is how the guest reports files it created, wrote, moved, or
deleted. Your own projection calls do not appear here.
With a Wait, Observe holds the call until the guest has produced an
observation after After, or the wait has passed, whichever comes first;
an empty page after the wait is not an error, so a caller keeps one call
open and never polls. Zero returns at once, and the wait is at most
MaxObserveWait (one minute). The host looks again in short passes, each
under the grant's control serializer, so a freeze, projection or revoke
never queues behind a wait; a freeze, revoke or loss during it ends it with
that error. Over the wire the wait is added to both sides' operation
timeout. Reading observations or the process log is not activity: a wait
kept open does not keep an idle grant awake.
page, err := env.Observe(ctx, execenv.ObserveRequest{
Fence: fence, After: acknowledged, Limit: 128, Wait: 30 * time.Second,
})
if err != nil {
return err // ErrLagged: resynchronize from your journal
}
for _, event := range page.Observations {
if event.Content != nil {
body, err := env.OpenObservationContent(ctx, fence, *event.Content)
if err == nil {
_, err = io.ReadAll(body) // the file as it is now
_ = body.Close()
}
if errors.Is(err, execenv.ErrStale) {
// Changed since: a later event of this entry reports it.
} else if err != nil {
return err
}
}
mutation, accepted := decideInYourStore(event) // you own durable files
if accepted {
if err := env.ConfirmObservationProjection(ctx, execenv.ObservationProjection{
Fence: fence, HostSequence: event.HostSequence, Mutation: mutation,
}); err != nil {
return err
}
}
acknowledged = event.HostSequence
}
if err := env.AcknowledgeObservations(ctx, fence, acknowledged); err != nil {
return err
}
hostCursor = page.HostCursorEach ordered event binds its original projected cursor and version, an
epoch-stable node identity, and the size and SHA-256 of a created or written
file as it was observed. ProcessTag names the process you started with
Start that the program which made the change descends from (the started
process itself, or anything it ran); a path several programs changed before
one capture names the last of them. It is empty when the host cannot say:
nothing you started made the change (the file API, for one), or the host
did not see who did. A guest learns the writer from the filesystem event
that reported the change and the guest kernel's process events, so a change
only the periodic walk found (a write through mmap, or after the event
queue overflowed) has none, and so may one whose writer ended long before
the capture. The in-memory host names the started process for what its
shell's redirections (> path, >> path) write. The body is not copied: OpenObservationContent
reads the file from the workspace as it is now, as one stream of 64-KiB
messages, hashing it as it goes. A file changed since then ends the stream
with ErrStale instead of its end, and a file gone or replaced fails with
ErrStale at once. That is safe to treat as "skip this one": every change
after an observation is a later observation of the same entry (a newer
write, a move or a delete), so reading the newest event of each entry
reaches the final state. Directory moves are single subtree operations,
parent creates precede children, and recursive directory deletion is
explicit.
After accepting a guest event durably, call ConfirmObservationProjection with
its host sequence and the exact resulting journal mutation before acknowledging
it. The host binds the original identity, topology and content digest to that
outcome. A later consecutive Apply of that mutation advances the baseline while
suppressing only that known echo, preserving any later guest writes. A
confirmation retry tolerates a new upload handle for the same content;
changing the outcome conflicts. Rejected or ignored outcomes need no
confirmation.
Acknowledge only after recording the outcome durably. Acknowledgement releases
the events, after which their content can no longer be opened; it does not
itself imply acceptance or advance projection versions. Unacknowledged
retention is bounded to 4096 events, and none of them holds a copy of a
body.
Overflow reports ErrLagged without discarding the retained prefix. A cursor
before the acknowledged prefix also returns ErrLagged.
Tool caches, virtual environments and dotfiles written through $HOME are not
in /workspace (see Run processes) and never appear here. A project-local venv or
node_modules inside /workspace is observed and counts toward the retention
bounds unless your IgnoreRules leave it out.
ProjectionInitialization.Ignore names a workspace-relative file of
gitignore patterns, and a fallback list used while that file is missing, is
not a regular file, is larger than 64 KiB, or does not parse. Patterns follow
gitignore: # comments, ! negation, a trailing / for directories only, a
leading or inner / anchors to the workspace root, ** spans any depth, and
the last matching pattern wins. A path under an ignored directory cannot be
re-included.
- Patterns suppress only paths the projection does not hold. A path you projected, or one Observe already reported, stays observed while it exists, including after a move to an ignored name. A held path moved into an ignored, unheld directory leaves the stream as a delete.
- The ignore file and its parent directories are always observed. The guest re-reads the file whenever its bytes change, whether you projected the edit or the guest wrote it, and the next Observe applies it: paths it un-ignores arrive as creates.
- An ignored directory that is not held is never walked. Its contents cost nothing, never lag the stream, and never count toward the observation or entry limits.
- Ignored paths stay in the environment; nothing is deleted for being
ignored. A projected file or move onto an ignored path replaces what the
guest left there, as a checkout does, and a projected directory adopts an
ignored directory already at its path unless it is an
Overwrite.
The rules last for the epoch: initialization happens once, and an exact retry is a replay.
The guest captures changes while the workload runs: Observe never freezes
it. Identities come from the filesystem itself (file handles that carry the
inode generation), not content hashes, so nothing is held open between
captures and a tree of any size costs no descriptors. One fanotify mark on
the workspace's filesystem names every path that changes, at any depth;
Observe reads again only those paths and what is under them, hashing only
files whose size or times changed, so its cost is the change's, not the
tree's, and an Observe with nothing new reads nothing. Every path that
changed before Observe was called is in what it reports. The whole
workspace is walked instead when the watcher cannot name what changed (its
queue overflowed), after the ignore file changed, and at least every 30
seconds, for writes through mmap that no event reports. A change still
under way when a capture passes it (a file being written, a directory being
moved or removed) is reported as far as it had got, and the next Observe
walks again to report the rest; a body read after it moved on is
ErrStale. Projection
calls (initialize and apply) still stop the workload behind a
kernel-acknowledged barrier while they change the workspace. The agent's
-contain-workload option requires a dedicated Linux guest with a usable
unified workload hierarchy. It launches unprivileged processes inside that
hierarchy before execution and prevents privilege escalation. Uncertain
filesystem commits remain contained. Baked images start a PID-1 child reaper,
require the kernel workload hierarchy, and run the contained agent; a guest
helper that cannot serve the journal is refused at Ensure. Run make certify
with the intended kernel/rootfs on real isolation hardware before production
use; it covers TLS remote journal projection, immutable guest events,
accepted-outcome echoes, semantic directory moves and preservation of the same
shell process through save and containment. Portable unit tests and Linux build
containers are not that certification.
ObserveProcesses is an ordered log of every program the environment's
workload starts and how each process ends. The guest helper records it from
the guest kernel, outside the workload, so the workload cannot see, alter or
suppress an event already recorded. Each event leaves the guest as soon as it
is taken; the host keeps it until you acknowledge it.
page, err := env.ObserveProcesses(ctx, fence, acknowledged, execenv.MaxProcessPage)
if err != nil {
return err // ErrLagged: that position was already acknowledged
}
for _, event := range page.Events {
switch event.Kind {
case execenv.ProcessStarted:
_ = event.Argv // bounded; Truncated says it was cut, Unread that it ended unread
_ = event.Tag // the Start tag for a process Start created, else ""
case execenv.ProcessExited:
_ = event.ExitCode // nil when event.Signal ended it
case execenv.ProcessDropped:
_ = event.Dropped // events lost before this one
}
record(event) // keep what you need before acknowledging
}
if err := env.AcknowledgeProcesses(ctx, fence, page.Next); err != nil {
return err
}
acknowledged = page.Next- The same fence as observations guards both calls; a stale epoch or
revision is
ErrConflict, a frozen grantErrFrozen, and revoke ends the log withErrRevoked. - Only the contained workload's processes are recorded, the login shell included; never the guest helper or init. A fork that never starts a program is part of its parent. A process that replaces its program reports another start under the same PID, and its one exit ends the latest.
- Sequences are per epoch and gapless. An exit never precedes its start, but PIDs are reused, so pair starts and exits by order.
Diris the working directory when it is inside/workspaceor the scratch home, else empty. A process that ended before its command line could be read is reportedUnread.- A process
Startcreated reports its start with itsTag; the processes it starts in turn have none, so a shell's commands are told apart from the shell. The helper records the tag before the process runs its program, so the start is never seen untagged, and since it ran that command itself, such a start is neverUnreadeither. - A full log never evicts and never blocks the workload. It counts what it
cannot keep, and one
ProcessDroppedevent carrying the count takes the next sequence once acknowledgement frees space. If the guest kernel's own event buffer overflows, the loss is reported the same way, counted as 1 because the kernel does not say how many. - A cursor before the acknowledged prefix is
ErrLagged; acknowledging or asking past the events produced isErrInvalid.
The guest kernel must have the process events connector
(CONFIG_CONNECTOR, CONFIG_PROC_EVENTS; scripts/check-kernel-config
requires both). A guest helper without it says so in its handshake and is
refused at Ensure, like one that cannot serve the journal. The in-memory
adapter's shell records itself and every command line it runs.
A grant runs processes by tag: up to 32 at once. A shell is a process with a terminal; a command is one without. Each tag is the caller's and unique among the grant's running processes; every other call names the process by it. A process lives in the grant, not on the connection that started it, and its output is kept on the host for any number of readers.
// A shell: an empty Argv with a terminal is the image's login shell.
_, err := env.Start(ctx, fence, execenv.ProcessSpec{
Tag: "sh",
PTY: &execenv.Window{Cols: 120, Rows: 40},
DetachGrace: time.Minute, // end it once nothing has been attached for a minute
})
// A command: separate stdout and stderr, its own exit code.
_, err = env.Start(ctx, fence, execenv.ProcessSpec{
Tag: "test",
Argv: []string{"go", "test", "./..."},
Dir: "src",
Env: map[string]string{"CGO_ENABLED": "0"},
Timeout: 10 * time.Minute, // TERM, then KILL five seconds later
})Read output by attaching. Each event carries its stream and absolute byte offset; remember the offsets you have read and attach from them after a lost connection to get exactly what followed:
a, err := env.AttachProcess(ctx, "sh", map[execenv.Stream]uint64{execenv.StreamPTY: offset})
defer a.Close() // detach; the process keeps running
for {
out, err := a.Next(ctx)
if errors.Is(err, io.EOF) {
break // after OutputExit
}
switch out.Kind {
case execenv.OutputData:
offset = out.Offset + uint64(len(out.Data))
case execenv.OutputTruncated:
// Bytes from out.Offset to out.To fell out of the history.
case execenv.OutputExit:
// out.Exit.Code, or out.Exit.Signal
}
}Input is by offset too. WriteInput appends at the offset you give and
returns the new one; bytes the process already took are dropped, so a
resend after a lost response types nothing twice, and an offset past what
it took is ErrConflict. ProcessInfo.Input says where input stands.
next, err := env.WriteInput(ctx, fence, "sh", input, []byte("make\n"))
err = env.Resize(ctx, fence, "sh", execenv.Window{Cols: 80, Rows: 24})
err = env.SignalProcess(ctx, fence, "sh", execenv.SignalInterrupt)
err = env.CloseInput(ctx, fence, "test") // a command started with Stdin- Streams are
ptyfor a terminal, elsestdoutandstderr. Each keepsHistoryBytes(1 MiB by default, at most 4 MiB). Attaching without an offset for a stream starts at its oldest byte still held; an offset past its end isErrInvalid. - A command's input is closed at once unless
Stdinis set. ProcessandListProcessesdescribe processes, running or exited. An exited process's output stays readable until its tag is started again. Starting a tag that is still running isErrConflict; a 33rd running process isErrBusy.- Signals are
INT,TERM,KILL,HUPandQUIT, sent to the process's whole session. - Freeze refuses input and starts (
ErrFrozen) and keeps every process. Revoke ends them (ErrRevoked); a lost grant ends them withErrLost. - Over a remote connection each attachment has its own 256 KiB flow control window: a stalled reader holds back only its own output, never calls or other attachments.
Zero Cols or Rows means 80×24.
Screen is a terminal process's screen as its terminal shows it now. The
guest helper keeps a headless terminal for each one, fed every byte the
process writes, so even a full-screen program reads back exactly:
s, err := env.Screen(ctx, "sh")
_, _ = os.Stdout.Write(s.Data) // redraws it on a blank terminal of s.Window
a, err := env.AttachProcess(ctx, "sh", map[execenv.Stream]uint64{execenv.StreamPTY: s.Offset})Data redraws the screen's text and styles, the cursor and whether the
program has the alternate screen; Offset is the output it reflects, so
attaching from it continues the screen with nothing missed or repeated.
Text is each row as plain text, and AltScreen, CursorRow, CursorCol
and CursorHidden say the rest. While a full-screen program has the
alternate screen, the main screen behind it is not included. A process
without a terminal is ErrInvalid; an exited one's screen stays readable
until its tag is started again.
A process starts in /workspace (execenv.GuestWorkspace), the projected
tree, or Dir within it. Its $HOME is /run/execenv/home
(execenv.GuestHome), guest-local scratch outside the tree: it is not
projected, observed, or harvested, and it is discarded with the grant. Both
live on the grant's writable disk, not in guest memory.
err := host.Revoke(ctx, "job-7f3a") // also env.Revoke(ctx); both are idempotentThe guest is destroyed. Keep anything you still need in your own store before this call.
A grant's environment can end without a revoke: its machine exits, its
guest kernel panics, or the host loses the guest link for good. The host
notices on its own. It frees the grant's resources and cleans up as a revoke would, and
every call on that occupancy fails with ErrLost: processes and their
attachments, Control, ControlStatus, projection, Observe and the process
log. Over the wire the same sentinel reaches the
client.
Ensure of the lost id also returns ErrLost, so a reconnecting caller
cannot mistake an empty new guest for the one it had. Acknowledge the loss
with Revoke (host or env; it returns nil), then Ensure the id again: it
gets a new Epoch, and you rebuild it from your journal as after any new
occupancy. Fences from the lost epoch never apply to the new one.
A host lists its grants and tells you when they change, so nothing polls:
grants, err := host.ListGrants(ctx) // held and lost grants, with state
g, err := host.WaitGrant(ctx, "grant-1") // once it has left GrantCreating
w, err := host.WatchGrant(ctx, "grant-1") // its state now, then each change
for {
g, err := w.Next(ctx)
if err == io.EOF {
break // after GrantRevoked
}
_ = g.State // creating, running, frozen, suspended, revoked or lost
}
hw, err := host.WatchHost(ctx) // every grant and the capacity, then changesGrantInfo carries the grant's epoch, image, network, resources, labels,
state, the fence revision of the control that put it there, and when it
started. A watch holds at most the latest state of each grant it has not
read yet: a burst of changes arrives as the last of them, so a slow reader
costs the host nothing more. An unknown grant is ErrNotFound.
The grant caller cannot add destinations.
env, err := host.Ensure(ctx, execenv.Spec{
ID: "job-7f3a",
Image: "python",
Network: execenv.NetworkAllowlist, // or NetworkNone
})NetworkNone gives the grant no NIC. NetworkAllowlist gives it the
IPv4 destinations configured on the host. If the host has none, Ensure
returns ErrNetwork. Changing network on a live grant is ErrConflict.
Match with errors.Is. Wrappers preserve the sentinel.
| Error | When |
|---|---|
ErrInvalid |
bad id, image, path, fence, or config |
ErrUnknownImage |
that catalog id is not on the host |
ErrCapacity |
the grant does not fit; errors.As a *CapacityError for the short resource |
ErrDraining |
host is draining; it serves held grants only |
ErrConflict |
live grant mismatch, stale fence, or unobserved guest change |
ErrUnavailable |
host unusable, or it refused the client's credentials |
ErrFeature |
the host does not offer a feature this client needs |
ErrFrozen |
I/O while frozen |
ErrRevoked |
grant is gone |
ErrLost |
grant's environment ended without a revoke; Revoke, then Ensure again |
ErrBusy |
the grant already runs 32 processes |
ErrClosed |
attachment or content reader closed, or input to a process that exited (not revoke) |
ErrNotFound |
path missing from the projection, or no process has that tag |
ErrLagged |
observations no longer retained, or a process log position already acknowledged |
ErrStale |
an observed file changed after it was observed; read its later event |
ErrTooLarge |
metadata over its bound, or too many content handles |
ErrQuota |
the grant's workspace disk is full; the call changed nothing |
ErrConnection |
remote session lost |
ErrNetwork |
allowlist requested, host offers none |
Do not log tokens, tree bodies, or process input and output.
| Cap | Value |
|---|---|
| Grant id / image id | 1–64 characters [A-Za-z0-9._-:] |
| File bodies, one or all | the grant's DiskBytes |
| Mutations per Apply | 128 |
| Apply metadata | 256 KiB |
| Initialization metadata | 64 MiB (a few hundred thousand entries) |
| Unacknowledged observations | 4096 events |
| Unacknowledged process events | 4096 events, 2 MiB |
| Process log page | 256 events, 256 KiB |
| Started command line | 64 arguments, 1 KiB |
| Reported working directory | 4096 bytes |
| Ignore file read | 64 KiB (larger uses the fallback) |
| Ignore fallback | 256 patterns, 16 KiB |
| Content and process output per message | 64 KiB |
| HTTP/2 stream window, client receiving | 256 KiB interactive, 4 MiB content |
| HTTP/2 stream window, host receiving | 4 MiB |
| Path length | 4096 bytes UTF-8 |
| Path segments | 64 |
| One segment | 255 bytes |
| Running processes per grant | 32 |
| Output history per stream | 1 MiB by default, 4 MiB at most |
| Process tag | 1–64 characters [A-Za-z0-9._-:] |
The workspace inside the guest is /workspace; $HOME is
/run/execenv/home. Both share the grant's workspace disk, its
Resources.DiskBytes.
This section is for the Linux machine that will occupy grants. It does not use Go. You install binaries and catalog disks, then give the application the listen address, token, and certificate.
Each grant is a microVM run by the runtime and supervisor pair bootstrap
installs: a guest that fully compromises its own kernel still has to escape a
hypervisor. A host without a usable /dev/kvm fails closed: bootstrap writes
nothing and Ready reports unusable. There is no container fallback.
- Linux on amd64
- Root, or equivalent, for the default layout
- A usable isolation device at
/dev/kvm(readable and writable by the host process, typically thekvmgroup) - The microVM runtime and supervisor pair (bootstrap installs them)
mkfs.ext4andresize2fs(e2fsprogs): every grant gets a sparse ext4 workspace disk, and every seed is kept as oneipandiptablesif you configure an allowlist- Network reachability from the application on port 8443 (or the listen address you choose)
Missing prerequisites fail closed. Docker is not used on this machine and is not a substitute.
Use a dedicated Linux machine, not a nested VM, when you can. The isolation device is itself a virtual machine monitor. Running that inside another hypervisor (a cloud VM with nested virtualization, QEMU-in-QEMU, WSL2, Docker Desktop’s Linux VM, many CI runners) often means:
/dev/kvmis missing or not writable- nested virt is advertised but disabled in the BIOS or by the provider
- extra latency, missing CPU features, or unstable guests
Bare metal, or a provider that exposes KVM to the guest as a first-class
device, is the production path. A “nested Linux” lab can work for
exploration if /dev/kvm opens, but it is the first thing to blame when
Ready is unusable or grants fail to start.
Grants ask for their own size; the Host configuration says how much the host gives in all and what a grant gets by default:
| Key | Default | Meaning |
|---|---|---|
capacity |
measured by bootstrap | cpu_millis, memory_bytes, disk_bytes the host gives grants in all |
cpu_overcommit |
1 | multiplies the CPU total; memory and disk are never overcommitted |
grant_defaults |
1000 millis, 512 MiB, 2 GiB | a grant's size where its Spec leaves one unset |
grant_max |
the capacity | the most one grant may ask for |
mkfs |
mkfs.ext4 on PATH |
formatter for workspace disks |
jail_uid_base |
2000000000 | first of 65536 host uids reserved for jailed microVM monitors |
A grant's machine gets enough virtual CPUs for its cpu_millis; its cgroup
throttles it to exactly that, with no swap. Its memory is held at its
memory_bytes plus the monitor's own overhead: past that the kernel
reclaims the host's cache of the grant's disk and slows its writes, and
only 256 MiB further does it stop the machine. Its workspace disk and
network device carry the grant's rate limits, changeable while it runs.
Each grant's monitor runs in its own jail under
<state>/work/jails, as its own unprivileged uid from the
jail_uid_base range, never as root. The kernel and rootfs are hard-linked
into the jail (read-only bind mounts when they sit on another filesystem),
so catalog files must stay readable by other users (the installed mode is
0644). Reserve the uid range for execenv: a restarted host kills every
process running under it except the machines of the grants it adopts.
The workspace disk is a sparse file in the grant's directory, so only written blocks use host storage. It is created when the grant starts and removed on revoke, and when a restarted daemon finds a grant's machine gone. The guest does not reclaim space from deleted files during a grant.
A grant can start from a saved machine instead of booting. List the machine sizes to save in Host configuration:
"shapes": [
{"name": "small", "vcpus": 1, "memory_bytes": 536870912},
{"name": "large", "vcpus": 4, "memory_bytes": 4294967296}
]For every catalog image and shape the daemon builds a template in the
background, one at a time: it boots the image at that size with a
placeholder disk, waits until the guest is idle before it has touched any
grant's disk, and saves it. A host that offers network access (allow)
saves each one a second time with a network device, on a placeholder link
of its own; the guest has no address on it yet. Ready lists each
template, whether it has a network device, and whether it is ready. A
grant whose memory is exactly a shape's, and whose CPU needs no more
virtual CPUs than the shape has, then starts from the template of its image
with the fewest such CPUs, with a network device exactly when the grant has
network access; its CPU is still throttled to what it reserved, and its
disk and network rate limits are its own. Any other grant, and every grant
until its template is ready, boots at its exact size.
A restored machine is made one grant's before anything in it runs. Guest
init waits for a start message from the host carrying fresh entropy, the
time, a machine id, the grant's epoch and, with network access, the
grant's address; it reseeds the random generator, sets the clock and the
machine id, gives the network device the grant's address and its default
route through the grant's own link, and only then mounts the grant's own
disk and starts the helper. A booted guest is configured by the same
message, so the two start alike. A restored network grant's device is
backed by the grant's own link from the moment it is loaded. So two grants from one template share no
randomness, identity or files. Each restored grant maps the template's
memory privately: pages it writes are its own. Templates live in
<state>/work/templates; a changed image or shape is built again.
The daemon watches every grant's machine. If a monitor exits, a guest
panics (guests boot with panic=1, so a panic resets and ends the
machine), or the guest link breaks, the daemon releases that grant's jail,
disk, tap and uid at once, as a revoke would, and its resources are free again.
Callers see ErrLost; Ready counts it in Lost.
Each microVM also gets a balloon device with free page
reporting and deflate_on_oom, so pages the guest frees go back to the
host and a grant's resident memory follows what it actually uses. This
needs the guest kernel to have CONFIG_VIRTIO_BALLOON,
CONFIG_MEMORY_BALLOON and CONFIG_PAGE_REPORTING built in; without
them the device is inert and grants behave as before. Restoring templates
and suspended grants needs CONFIG_VMGENID. CI checks a guest kernel's
.config with scripts/check-kernel-config.
Download execenvctl from the same tag your application pinned.
curl -fsSL -o execenvctl \
https://github.com/sudosylabs/execenv/releases/download/v1.2.3/execenvctl
chmod +x execenvctl
sudo mv execenvctl /usr/local/bin/
execenvctl --versionYou should see version=1.2.3 and tag=v1.2.3. Do not use a moving
latest URL.
sudo execenvctl bootstrapDefaults:
| Flag | Default |
|---|---|
--prefix |
/usr/local |
--sysconf |
/etc/execenv |
--state |
/var/lib/execenv |
--listen |
0.0.0.0:8443 |
--cpu-millis |
every CPU, 1000 each |
--memory-bytes |
the machine's memory, less 1 GiB or a tenth for the host |
--disk-bytes |
90% of the free space under --state |
--device |
/dev/kvm |
--mkfs |
mkfs.ext4 on PATH |
What bootstrap does, in order:
- Opens the isolation device. If that fails, it stops and writes nothing. The error says the device is missing or not usable, and that there is no container fallback.
- Finds
mkfs.ext4, or stops and writes nothing. - Creates the bin, state, and config directories.
- Installs the
execenvdaemon binary (from this tag, or--execenvfor an air-gapped file). - Installs the runtime and supervisor pair, fetching them if they are
not already on
PATH(unless--no-fetch). - Writes Host configuration and TLS files (see below).
- Writes
execenv.serviceand enables it (unless--no-start).
The catalog is still empty. The host cannot occupy grants until you install at least one Image.
sudo execenvctl status
# device=ok
# mkfs=ok
# unit=active
# drain=serving
# grants=0
# installed=
# release=v1.2.3drain is serving, draining, drained, or unreachable when the
daemon does not answer on its operator socket; grants is how many grants
it holds.
The capacity flags override what bootstrap measures; a re-run without them
keeps the capacity already written. Useful flags: --listen,
--execenv ./execenv, --no-start,
--no-fetch. --insecure writes cleartext for loopback labs only; do
not use it in production.
Re-running bootstrap is idempotent. It does not rotate the token. It
does not drop TLS files even if you pass --insecure the second time.
It does not fetch catalog disks.
On first production bootstrap (no --insecure), execenvctl generates a
self-signed server certificate:
- ECDSA P-256 key
- Subject CN
execenv - SAN: DNS name
execenv, IP127.0.0.1 - Not valid for more than about ten years
- Certificate
/etc/execenv/tls.crt(mode0644) - Private key
/etc/execenv/tls.key(mode0600)
Host configuration records those paths and security: tls. The
application must trust tls.crt and set TLS.ServerName to execenv,
or you replace the pair with a certificate that matches the hostname
clients actually dial (a public CA, an internal CA, Let’s Encrypt, and
so on). Leave the same paths in Host configuration after you replace the
files, then restart the unit.
A 32-byte random token is written into Host configuration (mode 0600).
It is never printed. Copy it once into the application’s secret store.
Ordinary logs must never contain it.
Mutual TLS is also supported. Set tls_client_ca in Host configuration to a
PEM CA that verifies caller certificates. With a token configured, either a
valid token or a verified certificate authenticates the caller. With an empty
token, a verified client certificate is required. TLS defaults to version 1.3.
For a public hostname, a typical client config is: listen
0.0.0.0:8443, replace tls.crt / tls.key with a cert whose SAN is
that hostname, set the application’s ServerName to the same name, and
open 8443 on the firewall only to the application network.
The daemon takes an exclusive lock on work_dir. A second daemon cannot use
the same state directory. work_dir must be on a filesystem that allows
device nodes (not one mounted nodev, as a tmpfs /tmp often is): each
jail holds its own isolation device, and the daemon refuses grants
otherwise. Grants' machines outlive the daemon (see Control), so
startup adopts the ones its state files name and stops only the jails
nothing names; revoke removes the grant directory.
Images are kernel plus root filesystem, with the toolchain and guest agent already baked in. They are not Docker tags and are not pulled when a grant starts.
sudo execenvctl install python
sudo execenvctl listinstall fetches that id from this binary’s tagged release, verifies the
kernel-then-rootfs hash, writes local paths into Host configuration, and
restarts the daemon. The restarted daemon adopts its grants, so they keep
running. remove refuses an image a live grant was started from; revoke
those grants first.
Ids you can install when the release includes them: python, node,
go, java. default is a large universal-class disk, often published
onto the same tag later. fixture is a tiny disk for live isolation
tests.
Prefer the slim per-language ids for high-concurrency use.
Every grant reads its image from the host page cache, and a slim image
keeps that cache, boot time and install size small. default is several
gigabytes of toolchains most grants never touch; install it only for
grants that genuinely need a multi-language environment.
Disks baked before the per-grant workspace disk keep the old guest init:
their /workspace stays in guest memory and doubles as $HOME. Reinstall
(or execenvctl upgrade) to get the current guest init.
sudo execenvctl remove python # does not touch the tokenUntil an id is installed, Ensure of that id is ErrUnknownImage.
install, remove and upgrade restart a running daemon, and the
restarted daemon adopts every grant it held. To take a host out of service
instead, drain it:
sudo execenvctl drain --timeout 30m
# state=draining grants=3
# state=draining grants=1
# state=drained grants=0While draining, the daemon keeps serving the grants it holds (including
Ensure of one of them) and refuses new ones with ErrDraining; Ready
reports Usable false with reason draining, so the application places
new grants elsewhere. drain prints the remaining count as it changes and
returns once none remain. If --timeout passes first it exits non-zero
and the daemon stays draining; --timeout 0 starts the drain and returns
at once. To cancel a drain without restarting:
sudo execenvctl undrainA daemon whose grants would not survive a restart (the in-memory adapter)
is not restarted while it holds any; --force restarts it anyway and
destroys them. A running unit whose daemon does not answer is refused too,
since then nothing can say what it holds.
execenvctl reaches the daemon on a Unix socket, operator_socket in Host
configuration (bootstrap writes /var/lib/execenv/operator.sock, mode
0600). It carries only drain requests, the live grant count and the
images those grants run from, never grant operations, so grant callers on
the network cannot take the host out of service. Drain state is not saved:
every daemon start begins serving.
- Address (
host.example:8443) - Token from Host configuration
- CA / server certificate the client should trust
ServerNamethat matches the certificate
The host does not call home. Placement of users onto this machine is entirely the application’s job.
Install a newer execenvctl for the target tag, then:
sudo execenvctl upgradeThat replaces execenv and execenvctl from this binary’s tag, then
reinstalls every catalog id already on the host so the guest agent
matches, then restarts the daemon once. Live grants keep running: the
upgraded daemon adopts them, and their machines keep the guest agent they
started with until they end.
Token, TLS files, listen address, capacity, and allowlist stay.
If an installed id is missing from the new index, upgrade fails before
binaries move. When everything already matches: upgraded=already current.
make check # generated code current, test, race, vet
make generate # rewrite internal/gen from proto/ (commit the result)
make build # stamps Release, Build, and Tag when HEAD is an exact git tagPull requests run make check. Catalog-related pull requests smoke-bake
fixture. Pushing v* publishes linux/amd64 binaries and language
disks. Isolation hardware tests are opt-in (make isolation,
make certify).
Bake (scripts/bake) is CI-only. It is not an execenv command and is
not installed on the grant host.
scripts/bake --out ./out --kernel ./vmlinux --agent ./execenv \
--id python --dockerfile ./catalog/python/Dockerfile