Prerequisites
You need:- a maintained host with at least 4 vCPUs and 8 GB of memory; allocate these resources to Docker Desktop’s virtual machine when using it
- Docker Desktop or Docker Engine with Docker Compose
- Git, curl, uv, and the full hash of the Sediment commit you want to deploy
Deploy the API
Before building, review release security evidence. On the deployment host, check out the commit you want to deploy:sediment server is running on this host, stop it with Ctrl+C before using
Compose’s default API port, 8000. To keep both deployments running, choose a
different Compose API port. Compose creates its own PostgreSQL volume; it
doesn’t reuse or import the local database under ~/.sediment/server.
Create .env with separate generated credentials and owner-only permissions
before any secret reaches the file:
--ingest-client for
each machine. The generator writes a distinct ingest-only token for each client
and a separate gateway token. It refuses existing files and directories writable
by another user.
Keep .env private. Don’t source it or distribute it to developers.
Edit these deployment settings in .env:
Open
.env in a private editor to retrieve each client’s entry from
SEDIMENT_INGEST_TOKENS. Distribute only that entry’s token through your
credential channel. Reserve SEDIMENT_OPERATOR_TOKEN for queries and reports.
The generator never prints secrets. To add a client after installation, follow
credential rotation.
Configure PostgreSQL
Keep the four generated database passwords in.env. The supplied
docker-compose.yml uses them as follows:
Compose sets the database name to
sediment and connects services to
postgres:5432 on the internal database network. You don’t need to install
PostgreSQL on the host or publish port 5432.
Compose builds SEDIMENT_BOOTSTRAP_DATABASE_URL for migrate and a separate
SEDIMENT_DATABASE_URL for each of api and operator from those passwords.
You don’t need to add a connection URL to .env. Keep the generated hexadecimal
passwords; they are safe to include in these URLs. For an existing deployment,
follow credential rotation instead of replacing
passwords before a restart.
The postgres service stores data in the sediment-postgres named volume at
/var/lib/postgresql/data. Keep this volume when restarting or rebuilding.
LiteLLM sends captured Inference calls to http://api:8000; the API writes
them to PostgreSQL. The gateway doesn’t receive database credentials.
Start the API and database
Build and start the deployment with its source identity:migrate to
provision the database roles and apply migrations. It starts the API only after
migrate exits successfully. The start command returns success after PostgreSQL
and the API pass health checks.
The 120-second readiness limit starts after image building. If startup fails,
inspect the containers and logs before retrying the same start command:
docker compose down.
Run a second local deployment
Before its first start, set these values in the second checkout’s private.env:
http://127.0.0.1:18080 for this stack’s API checks
and login. Ports remain bound to loopback. The defaults are sediment, 8000,
and 4000; volume names in this guide assume those defaults.
Expose a public HTTPS endpoint
If you operate external ingress, leave thehttps profile disabled.
Route your HTTPS API hostname to http://127.0.0.1:8000 through a reverse
proxy or tunnel on the host. Keep the Compose ports bound to loopback. Preserve
request bodies and authorization headers, and rate-limit authentication attempts
at ingress; the API doesn’t provide that limit.
If you use Cloudflare, follow its
local tunnel setup
and service installation.
Cloudflare carries requests through its network. If your data must stay within
your own network, use an internal proxy.
Remote clients use the HTTPS deployment root, such as
https://sediment-api.example.com. sediment login rejects embedded credentials,
query strings, fragments, and non-root paths. HTTP is allowed only on literal
loopback hosts.
Configure capture
- For private repositories, configure read-only mirror credentials.
- Configure GitHub webhooks for Pushes, pull requests, repository changes, and continuous integration (CI) outcomes.
- Optional: Enable bundled LiteLLM or connect an existing gateway for Inference calls. Keep the developer’s selected model unchanged.
Configure private mirrors
Before mounting private Git credentials, review the image and Git configuration requirements in Secure a deployment. The supplied release evidence doesn’t cover custom credential mounts. Then mount a deployment-local.netrc through docker-compose.override.yml:
0600. Make it readable by the API
container’s user without granting access to other host users:
docker compose up -d --no-deps api. After
enrollment, push a test commit with a Session note. In
docker compose logs --since 10m api, require
session_commit_observations_captured with a nonzero stored or duplicate count
for that repository. Verify the commit with the
forge check; a Push Fact alone
doesn’t verify private Git access.
Connect an existing LiteLLM gateway
Copylitellm/sediment_callback.py and
cli/sediment_cli/delivery.py next to the gateway configuration. Name the copied
helper sediment_delivery.py. Both files must be importable by the proxy. Register
the callback:
SEDIMENT_INGEST_TOKENS map.
The callback uses SEDIMENT_API_BEARER_TOKEN for ingest; don’t give it an
operator token. After changing the map in .env, recreate the API with
docker compose up -d --no-deps api. Removing an entry revokes that client.
Use HTTPS for remote callback destinations. Loopback HTTP is accepted for
localhost, 127.0.0.0/8, and [::1]. The callback rejects redirects. If the
callback and API share a trusted container network, set
SEDIMENT_GATEWAY_LOCAL_HTTP_ORIGIN to that one HTTP origin. The bundled Compose
profile uses http://api:8000. This exception matches the scheme, hostname, and
port exactly; it never authorizes OTLP delivery or another destination.
The callback uses five-second HTTP timeouts and preserves capture identity and
observation time across retries. Capture failures don’t fail the model request.
If you agree to store the prepared payload on disk, set
SEDIMENT_DELIVERY_DIR to a private directory on persistent storage. The payload
can contain unredacted prompts, code, or credentials before server redaction.
The callback starts a replay worker for its process lifetime. Without this
setting, it reports best_effort and attempts direct delivery.
Unsafe, unavailable, or busy buffer storage also triggers one direct attempt
with a best_effort diagnostic. Repair the volume to restore durable recovery.
See Preserve prepared payloads through outages
for limits, permissions, and recovery commands.
If you collect raw fixtures for integration debugging, set SEDIMENT_CAPTURE_DIR
to an absolute private directory owned by the gateway user. This opt-in writes
unredacted prompts, responses, code, and possibly credentials. The callback
creates the directory with mode 0700 and both JSON files with mode 0600.
It refuses permissive paths, foreign ownership, symlinks, and hardlinks.
An unsafe fixture destination logs fixture_write_failed without stopping
valid gateway delivery. Restrict access, use encrypted storage, and delete the
fixtures when the investigation ends. Basic redaction at the API doesn’t protect
these local raw files.
Clients must carry a real Session identifier through metadata or a supported
protocol carrier. The API skips unresolved calls and logs
gateway_ingest_skipped_no_session. Upgrade the server before clients when
identity parsing changes.
Verify the deployment
For remote clients, check the public health endpoint. For Docker Desktop evaluation, use the loopback health check from Start the API and database:sediment facts prints total and Derivation-visible rows. Zero counts are
expected before the first capture. Require a healthy API and PostgreSQL container
and a successful migration container (Exited (0)).
Inspect the PostgreSQL revision without changing it:
at_head. Before enrollment, create and restore a backup.
Record the revision, image identities, health result, database status, and
backup restore result. Then complete the
team enrollment. Health and empty Fact counts
don’t verify live capture; use the pilot’s Session and forge checks for that.
Operator commands connect through the separate operator database role and don’t
rerun provisioning. To stop and resume the same deployment, run
docker compose down, then docker compose up --wait --wait-timeout 120.
Keep .env and omit --volumes to retain credentials and Facts.
Enable bundled LiteLLM
If you need the bundled Anthropic gateway, addANTHROPIC_API_KEY to .env.
Keep the generated LITELLM_MASTER_KEY and gateway ingest token. The gateway
supports claude-* routing; other providers require a separate gateway
configuration. See the gateway boundary.
Agents authenticate with LITELLM_MASTER_KEY, which also administers the
gateway. Distribute gateway routing
defines how agents reach it.
If you agree to store unredacted capture payloads on disk, set
SEDIMENT_DELIVERY_DIR=/data/delivery/pending in .env. The gateway uses the
sediment-delivery named volume and owns a replay worker for its process
lifetime. Leave the setting empty for direct best-effort delivery.
From the deployment checkout, build and start the gateway with its source identity:
http://127.0.0.1:4000. The bundled proxy instead serves the gateway at /llm
on the API hostname. The gateway receives provider and ingest credentials, but no
database credentials. Its callback sends Inference calls to http://api:8000.
A capture failure doesn’t retract a successful model response.
If the volume is unsafe, unavailable, or busy, the callback attempts direct
delivery and logs the reason.
See Preserve prepared payloads through outages
for privacy, retention, capacity, and recovery limits. Upgrade the API before
the callback: a server without the capture envelope returns 422, which blocks
buffered entries until you upgrade and retry them.
To retry blocked entries after correcting the API or credentials, stop the
gateway so its callback releases the delivery lock. The one-off helper uses the
same configured volume, endpoint, and credentials:
docker compose logs gateway.
Use Configure inference-call
capture to
route clients and verify both the model request path and the capture path.
Enable agent-requested retrieval
Optional: set the retrieval settings from Enable agent-requested retrieval in.env, and recreate the API with docker compose up -d --no-deps api.
Compose passes these settings only to the API. Before you upgrade an old
deployment, rename any ingest client named retrieval.
Upgrade the deployment
Before upgrading, readCHANGELOG.md, review release security evidence,
back up the database, and test restoration. Measure migration time on a restored
copy. Constraint validation and index builds can block table reads and writes;
schedule a maintenance window for large datasets.
Stop the API and gateway before changing database roles or credentials. Preserve
POSTGRES_PASSWORD: changing it in .env doesn’t rotate an initialized server.
Choose the full hash of the commit you want to deploy. Pin the checkout to that
commit when upgrading, as you do for the first installation.
If your existing .env predates separate database roles and operator tokens,
prepare its replacement before running Compose against the updated checkout:
- From the existing checkout, stop the API and gateway. Save the encrypted database backup and its restore record.
-
Update the checkout. Restrict the existing credential file before opening it,
then generate a separate private candidate:
-
In a private editor, copy the existing
POSTGRES_PASSWORD,SEDIMENT_ORG_ID, webhook secret, clone-host policy, provider key, and gateway master key into.env.next. Keep the agreed delivery-retention settings. Keep the generated migrator, runtime, operator database passwords, operator HTTP token, gateway ingest token, and matching named ingest map. If existing clients useSEDIMENT_API_BEARER_TOKEN, retain that value as an ingest-only compatibility credential. KeepSEDIMENT_DEV_MODE=false. -
Replace the old file with
mv .env.next .env. The candidate is mode 0600 throughout. - Rerun the build and start commands from Deploy the API. If provisioning rejects unknown ownership or objects, investigate them instead of granting the API bootstrap authority. Do not start the old API against the confined roles.
- After the API is healthy, reenroll operator clients with the operator token. Old capture clients can retain the compatibility ingest token until you migrate them.
.env and use the same update sequence:
.env values, and rerun provisioning
before recreating their consumers. To rotate an ingest token, replace only that
entry in SEDIMENT_INGEST_TOKENS, update its enrolled client, and recreate the
API with docker compose up --wait --wait-timeout 120 api. To enroll another
client, generate a distinct token with
uv run --no-project python -c 'import secrets; print(secrets.token_hex(32))',
add it under a unique name in SEDIMENT_INGEST_TOKENS, then recreate the API
with the same command. Keep existing entries; don’t regenerate .env.
Client names operator, legacy, and retrieval are reserved.
The gateway’s token must match its named map entry. Rotate the operator
HTTP token independently. Remove SEDIMENT_API_BEARER_TOKEN after migrating
legacy capture clients; that token grants ingest authority only.
Upgrade the API before gateway callbacks and capture clients. Verify
sediment db status reports at_head before resuming capture.
For 0011_inference_call_aliases, stop every API replica and direct Fact writer
before provisioning. The migration copies provider and output tool-call
identifiers from every retained Inference call, including quarantined history,
and builds the physical lookup index in one transaction. It blocks parent reads and writes
and decodes one output row at a time; memory depends on the largest output and
its identifiers. Measure duration and disk growth on a restored database before
scheduling the maintenance window. Restart only the matching API build after
provisioning succeeds. Stale writers that omit the physical alias count fail
instead of creating unindexed Facts. See Indexed call identifiers.
Operating cadence
Each week, inspect storage usage, review failed scans and upstream fixes, and check Fact growth and outcomes:security-support.json and the review expiries in the
release evidence.
Back up and restore
On a separate recovery machine, generate an age identity. Keep its private key off the deployment host. Put only its public recipient inBACKUP_RECIPIENT on that host. In Bash, stream a PostgreSQL custom archive
directly into encryption:
pg_restore --exit-on-error --no-owner --no-privileges --dbname <disposable-database> through a pipe with pipefail.
Use that database’s private credential file instead of a password in arguments.
Restore into an empty database before running role provisioning; the latter
adopts restored known objects and reapplies the grants. Verify the schema
revision, Fact counts, quarantine log, and a representative export against the
backup record. Destroy only that disposable test database after verification.
Follow pg_restore for the
archive and connection options. Record the backup timestamp, restore result,
and recovery duration. Test restoration before enrollment and after
schema or backup-tool changes.
If you enable the bundled gateway, review unresolved completion identity:
Privacy and data handling
Secure a deployment describes what Sediment stores, how the credentials divide authority, and the network paths. This section covers what differs under Compose.Where data lives
Operator query routes return captured content. Optional retrieval access returns
content from its authorized Sessions. Capture credentials don’t authorize reads.
Host administrators can read container environments and mounted data.
Facts and mirrors have no automatic retention or expiry. Raw mirrors, optional
capture files, and encrypted backups remain sensitive even when Basic redaction
protects normalized Fact content. Authorize any raw capture retention explicitly.
Put Docker data and export storage on a dedicated filesystem or enforce a
volume-driver quota before deployment. Record its size and alert before
80% use. The mirror worker requires 1 GiB of free space, but doesn’t enforce a
quota. Budget for backups and incident recovery.
Quarantine and wholesale deletion
Quarantine excludes a Fact from every Derivation and export without changing the Fact row. An append-only audit log records each quarantine and release. Bulk inference-call quarantine dry-runs unless you pass--apply:
sediment facts shows the visible count. sediment quarantine-log shows the
audit trail and the quarantine-state Provenance value.
Quarantine captured data explains the
effect of a partial Edit observation quarantine.
To delete the deployment data, remove its Compose volumes. This deletes Facts,
mirrors, exports, staging, and buffered deliveries. You can’t undo it. Save
any required database backup and exports first:
Network exposure
An empty
SEDIMENT_ALLOWED_CLONE_HOSTS list admits public hosts only.
To disable mirror fetches in this deployment, remove SEDIMENT_MIRROR_PATH from
the API’s environment in docker-compose.yml and recreate the API. Compose
sets it directly; unsetting it in .env has no effect.
PostgreSQL has no published port. Only provisioning, API, and operator services
join its internal network. Local and TCP connections require password
authentication through SCRAM (Salted Challenge Response Authentication Mechanism).
The gateway has no database-network access.
Compose applies read-only roots, temporary-storage and resource limits, rotated
logs, and no-new-privileges. The API and gateway have no Linux capabilities.
PostgreSQL keeps only the capabilities required to initialize volume ownership
and drop privileges.
Keep the single API process unless you recalculate host and database capacity.
It permits two active reads, two mirror workers, and sixteen waiting mirror
jobs. Queries and reports share both read slots; a third read returns 503
without queueing. Read and mirror deadlines are 30 and 120 seconds. Memory
limits are 2 GiB for the API, 1 GiB for PostgreSQL, and 2 GiB for the gateway.
Reserve capacity for operator and migration jobs.
Review the disposition register before a
deployment. Database isolation and resource limits reduce exposure to
unresolved native parser vulnerabilities. They don’t remove vulnerable code or
protect stored Facts after a database-process compromise. A custom network, SQL
client, image, Git configuration, or credential distribution needs another
review.
For operation without public network access, provision software, container
images, and dependencies inside your network.
Troubleshooting
- Image builds or pulls hang: check Docker registry access and the configured credential helper.
/healthresponds but other requests reach the wrong process: inspectdocker ps. Another container may own port 8000.- Ingest returns
503 database_unavailable: restore PostgreSQL access, then retry the retained request bytes. The same running service can reconnect. Committed Facts retain their original stored identities on duplicate delivery. Each Fact-type batch commits its Facts and Session rows together; one ingest operation can contain several batches. Database deduplication doesn’t guarantee delivery of events that a sender never retained or sent. - The API doesn’t start after PostgreSQL becomes healthy: read
docker compose logs migrate api. Rundocker compose --profile operator run --rm operator sediment db statusto distinguish an absent, behind, at-head, or ahead revision without changing it. Startup diagnostics name only the sanitized database target and never credentials. - Compose recreates another checkout’s containers: both checkouts selected
the same project name. Keep the original deployment’s
.envintact. Set a distinct project name and ports in the evaluation checkout before starting it. - The bundled gateway exits during startup: run
docker compose logs gateway. Its entrypoint names a missingANTHROPIC_API_KEYorLITELLM_MASTER_KEYbefore LiteLLM starts. - An agent receives no model response through the bundled gateway: inspect
docker compose logs gatewayfor client authentication, model routing, or Anthropic errors. The Sediment callback doesn’t run until a model call succeeds. - An agent receives a response but no Inference call appears: inspect
docker compose logs gateway api. A callback authentication or ingest failure doesn’t retract the successful model response.
Teardown
- Use Uninstall capture on developer machines that received a local install.
- Follow Remove managed capture to remove fleet hooks, gateway callbacks, telemetry, and forge webhooks.
- If you need the dataset, create and extract the backup from Operating cadence.
- Run
docker compose --profile gateway --profile https --profile operator down --volumeson the host. - Remove the ingress routes and DNS records that served this deployment.