Managed application kernel
Maturity: evolving. This surface is still finding its shape and may be redefined in any minor release — always with a
### ⚠️ Breaking changesentry and a migration section, never silently. The server, contract and client surfaces it composes are stable.
Use stitchkit/application when several process-local resources must become
ready and shut down as one application. Keep using the lower-level server and
signal APIs when one managed server is already the complete lifecycle boundary.
The neutral entrypoint is server-only and works on Bun and Node ≥ 22. It does not import provider SDKs.
For complete database, poller, queue-consumer and operational-publisher cutovers, continue with the executable application migration recipes.
Minimal composition
import {
createApplication,
createManagedSchedule,
defineManagedResource,
managedServerResource,
} from 'stitchkit/application'
import { bindProcessSignals } from 'stitchkit/server'
const database = defineManagedResource({
id: 'database',
required: true,
start: async ({ signal, reportHealth }) => {
await db.connect({ signal })
reportHealth('healthy')
},
close: () => db.disconnect(),
})
const cleanup = createManagedSchedule({
id: 'cleanup',
dependsOn: ['database'],
everyMs: 60_000,
startAfterMs: 60_000,
overlap: { mode: 'skip' },
errorPolicy: 'continue',
run: ({ signal }) => removeExpiredRecords(signal),
})
const http = managedServerResource({
id: 'http',
dependsOn: [database],
// Called during `start`, after `database` is ready. The factory receives the
// same declared-dependency and startup-signal context as any resource.
server: (context) => {
const db = context.use(database)
return createServer({ port: env.PORT, services: createServices(db) })
},
})
const app = createApplication({
id: 'service',
resources: [database, http, cleanup],
// One number for how long this application may take to stop. `shutdown()`
// with no options and the signal path both spend it.
shutdown: { gracePeriodMs: 30_000, forceTimeoutMs: 5_000 },
})
const signals = bindProcessSignals(app, {
onComplete: (result) => {
process.exitCode = result.outcome === 'clean' ? 0 : 1
},
})
await app.start()
Bind before you start, and the order is load-bearing. Reading it the other way
round — start serving, then arrange for shutdown — leaves a window between the
first accepted request and the first listener, and a supervisor that sends
SIGTERM inside that window kills the process outright: exit 143, no shutdown
chain, no onComplete, nothing drained. The window is small and opens exactly
when the machine is busiest, which is when a supervisor is most likely to be
restarting something. bindProcessSignals does not start anything, so there is
no cost to binding first.
The exact resource callbacks are typed by their public configuration. The important ownership rule is stable: the application decides what the database/server/provider objects are and how they are configured; Stitchkit orders their process-local lifecycle and decides when they come into existence.
Resource authoring
Give every resource a stable, bounded ID and list its dependencies explicitly. The graph is validated before any start. Required resources may depend only on required resources; an optional integration cannot silently become a required dependency.
Start and readiness are different. A connection can finish setup and return a
ready handle. A poller can return a handle immediately, settle ready later,
and keep a long-lived completion promise. Stitchkit observes both.
Once Stitchkit invokes start/activation, cleanup is guaranteed to be attempted.
Write close so it is safe when setup was partial:
let cacheClient: CacheClient | undefined
const cache = defineManagedResource({
id: 'cache',
required: false,
start: async ({ signal, reportHealth }) => {
cacheClient = new CacheClient()
await cacheClient.connect(signal)
reportHealth('healthy')
},
close: async () => {
await cacheClient?.close()
cacheClient = undefined
},
})
Invoking start makes the descriptor rollback-eligible immediately. Its
close callback must therefore clean side effects even when start rejects
before returning a runtime handle. The kernel calls cleanup once and continues
cleaning other attempted resources even if one close fails.
Handing a resource to the resources that depend on it
dependsOn says when. To say what, return a value from start and read
it with context.use(...):
const database = defineManagedResource({
id: 'database',
start: async ({ signal }) => ({ value: await connect(env.DATABASE_URL, signal) }),
})
const worker = defineManagedResource({
id: 'worker',
dependsOn: [database],
start: (context) => {
const db = context.use(database) // Connection — not Connection | null
return { completion: consume(db, context.signal) }
},
})
Declare the dependency with the resource, not its id, whenever you intend to
read from it: that is the form use can type, and it keeps the declaration and
the read from drifting apart. A string still works when all you need is order.
managedServerResource follows the same rule. Its factory receives this context
during start, so constructing routes from a database/service/socket value does
not need an outer mutable handoff or a custom start override. A zero-argument
factory remains valid when it needs only ordering, and an already-running handle
can still be passed directly. → ADR 0121.
The value is published when start resolves and stays readable for the rest of
the application’s life — from activate and from the shutdown phases too, where
a dependant may still need the handle it was given.
use refuses two things, both loudly: a resource that was never declared in
dependsOn (it happens to work whenever declaration order is lucky), and a
resource that published nothing. The second is refused by the compiler as well —
reading a value off a resource with no value in its start does not type.
Readiness and health
await app.start() resolves only after every required resource is ready and
every required post-ready activation has succeeded. Lifecycle ready is
published before activation so schedules never arm during starting;
application admission opens only after successful activation. Optional failure
may retain lifecycle ready, but snapshot health is degraded, never
healthy.
Readiness is not hidden polling. A resource reports health changes through its lifecycle context; the application decides when a database/provider probe runs.
A reportHealth call inside start is kept. A resource that says nothing
is assumed healthy once it is ready; one that reports its own health has already
answered the question, and the answer stands. The readiness default never
overwrites it.
Reporting the same health again is not a change. A resource may confirm its
health on a timer; a report with the value it already reported publishes
nothing — no new revision, no changedAt, no subscriber call. A change of value
publishes once, as does the first report that confirms the healthy the kernel
assumed at readiness.
A resource is required unless you write required: false. That default is
what makes the next sentence bite.
Readiness requires every required resource to be healthy, so “ready but
degraded” is unreachable for a required resource by construction: an
application whose required resource reports anything but healthy refuses to
start, and says which resource and in what state. The refusal distinguishes the
two ways to get there — a resource that was never healthy is pointed at
required: false; one that was healthy and stopped is pointed at
onResourceFailure. A resource that is expected to start
degraded — up, but still dialling something external — belongs behind
required: false, where it keeps its own health and does not gate the
application:
defineManagedResource({
id: 'dialling',
required: false,
start: ({ reportHealth }) => { reportHealth('degraded') },
})
An optional resource reporting non-healthy does not gate readiness, but it does
move the application aggregate to degraded — which a readiness endpoint
mapping degraded to non-200 will notice.
If startup fails, every resource that was already started is closed in reverse
order. The rollback runs one phase — close — not the five a real shutdown
runs, and close receives the same deadlines a shutdown would give it, so a
server drains what is in flight instead of aborting it.
Those deadlines come from the application’s declared budget, and it is worth
knowing what they cost. With nothing in flight the rollback returns at once: a
grace period is a ceiling, not a sleep. With something in flight it waits for
it — that is the point — and with something that never finishes, a hung
upstream or a client that ignores a close frame, it waits out the whole budget
before forcing. Under the default 30s+5s that turns a failed start() that used
to reject in milliseconds into one that can take 35 seconds to reject.
An application that would rather hear about a broken start immediately says so, in the one place both stopping paths read:
createApplication({
id: 'app',
resources,
// Applies to `shutdown()` called with no options AND to the rollback of a
// failed `start()`, which has no call site of its own to be told.
shutdown: { gracePeriodMs: 5_000, forceTimeoutMs: 1_000 },
})
The budget is a real bound, not just a number handed to each resource: a close
that never returns is abandoned when the budget runs out, reported as a close
failure, and the startup error stays the cause of the AggregateError that
start() rejects with. Without that, one unresponsive resource could keep a
failed startup from ever reporting why it failed.
A required long-lived completion that rejects after startup makes readiness false and health unhealthy. Stitchkit records the failure but does not restart the resource or process.
ApplicationResourceShutdown.failures names the phase that failed — start,
ready, completion, admission, drain, close, force — and nothing
else. To learn why, pass onResourceFailure:
createApplication({
id: 'app',
resources,
onResourceFailure: ({ resourceId, phase, error }) => {
log.error({ resourceId, phase, err: error }, 'managed resource failed')
},
})
It fires for every phase, including the one place the cause would otherwise be lost entirely: an optional resource failing to start. A required one rethrows, so its cause reaches you anyway; an optional one is swallowed on purpose, because the application keeps running. A throwing observer cannot break the lifecycle it observes.
createApplicationHealthHandler publishes an ApplicationStatusProjection
through a Fetch-compatible handler, suitable for a raw route on Bun or Node.
The projection carries the verdict — the application’s own id, lifecycle,
health, ready, capturedAt and resource counts — and never the internal
topology: no per-resource ids, no dependsOn edges, no process epoch, no
admission counters.
Those stay in getSnapshot(), which never leaves the process, and in whatever
telemetry you wire to it. Product-specific probes may be composed beside it; do
not put secrets or raw provider failures in the response.
For the conventional three-route surface, reuse the same semantics instead of copying them into the application:
const operational = createApplicationOperationalHandlers(app)
const rawRoutes = [
{ method: 'GET', path: '/status', handler: operational.status },
{ method: 'GET', path: '/ready', handler: operational.readiness },
{ method: 'GET', path: '/live', handler: operational.liveness },
]
status always returns the current published projection with HTTP 200,
including while starting, draining or stopped. To read the full snapshot — every
resource, its dependency edges and the live counters — call app.getSnapshot()
in-process; there is no option that publishes it over HTTP, because these routes
are meant to be reachable. The two probes retain the existing
readiness/liveness status and Retry-After policy.
Applications that already own an OpenTelemetry SDK may inject its Meter into
the isolated adapter:
import { createApplicationOpenTelemetry } from 'stitchkit/application/opentelemetry'
const telemetry = createApplicationOpenTelemetry({
meter,
application: app,
activities: [activity],
schedules: [schedule],
})
// During application cleanup:
telemetry.close()
The adapter registers fixed observable gauges and pulls the latest canonical
snapshots on every collection. It owns no exporter, SDK lifecycle, cache,
subscription or polling loop. Attributes are limited to declared application,
resource, activity, stage and schedule IDs plus bounded framework states;
epoch, revision, timestamps, failures and product/provider identities are never
metric attributes. Install @opentelemetry/api only when this entrypoint is
used; the neutral stitchkit/application graph remains peer-free.
Every instrument uses unit 1 and reports an absolute current/lifetime value:
| Instruments | Meaning |
|---|---|
stitchkit.application.lifecycle, .ready |
current lifecycle/health fact and readiness |
stitchkit.application.admission.accepting, .accepted, .completed, .pending |
current gate and absolute lifetime admission counts |
stitchkit.application.resource.ready |
readiness for each declared resource with required/state/health attributes |
stitchkit.application.schedule.accepting, .active, .queued, .runs_started, .runs_completed, .runs_failed, .runs_woken, .wakes_coalesced, .ticks_skipped |
current schedule state and absolute run, wake and tick counts |
stitchkit.application.activity.active, .queued, .completed, .failed |
absolute stage projections for declared activity sources |
Replacing part of the graph without stopping the process
restart takes down one resource and everything that depends on it, then
brings that subtree back. Everything else keeps running, and the process epoch
does not move.
const result = await app.restart({ resourceId: 'database' })
result.outcome // 'restarted' | 'failed' | 'refused'
result.affected // ['database', 'repository', 'api'] — in start order
The dependants come down with it because they are holding what the resource published. A repository that kept running across a database replacement holds a pool that has been closed: it still typechecks, still has methods, and fails at whatever moment the first call happens. There is no version of this where a dependant keeps a live handle, so the subtree — not the resource — is the unit.
A leaf restart affects one resource, and an independent neighbour is never touched:
await app.restart({ resourceId: 'cache' })
// mailer and database were not stopped, not started, not activated
Refused is not failed. An unknown id, a restart during shutdown, and a
restart before the application is ready all return refused with a reason and
touch nothing. failed means the subtree came down and the new generation did
not come up — and the result agrees with the snapshot: if any affected resource
ends the restart in failed, so does the restart, including an optional one
whose failure the start loop does not re-throw.
The close phase is bounded, by the application’s shutdown budget or by one this call names:
await app.restart({ resourceId: 'database', gracePeriodMs: 5_000, forceTimeoutMs: 2_000 })
A resource has to be able to start twice. The kernel calls start again, so
a resource holding state across its own lifetime rebuilds it there. The ones this
framework ships do — a schedule re-arms, a keyspace opens a new generation and
re-loads, a managed server built from a factory gets a fresh server. A managed
server given a server instance cannot, and says so by name rather than
republishing something it has shut down. Give a keyspace a
backend: () => … factory when its backend cannot be re-opened after close().
→ ADR 0157.
Restarts of overlapping subtrees queue behind each other rather than being refused. Two callers asking at once is ordinary; two generations of one resource alive at once is the thing this must make impossible.
// Both succeed, one complete pass after the other.
await Promise.all([
app.restart({ resourceId: 'database' }),
app.restart({ resourceId: 'database' }),
])
What a restart is not is a process restart: it replaces resources, and does not re-read configuration the kernel captured when it was constructed.
→ ADR 0154.
Decisions a policy set makes together
createDecisionPipeline runs an ordered list of policies that each vote
allow, deny (with a reason) or defer. The first terminal verdict wins and
the rest do not run.
import { createDecisionPipeline } from 'stitchkit/application'
const pipeline = createDecisionPipeline<{ userId: string; scope: string }>([
{ id: 'banned', decide: (r) => (isBanned(r.userId)
? { outcome: 'deny', reason: 'account suspended' }
: { outcome: 'defer' }) },
{ id: 'scope', decide: (r) => (r.scope === 'admin'
? { outcome: 'allow' }
: { outcome: 'defer' }) },
{ id: 'default', decide: () => ({ outcome: 'deny', reason: 'no policy allowed this' }) },
], { policyTimeoutMs: 2_000 })
const result = await pipeline.decide(request)
result.outcome // 'allow' | 'deny' — never 'defer'
result.trace // the policies that actually ran, in order
Three things are deliberate.
A deny carries a reason, by schema. { outcome: 'deny' } alone does not
typecheck. A refusal whose cause exists only in the log of whoever refused is a
support ticket.
The trace is what ran, not what was declared. It stops at the terminal verdict. When something was denied the question is which policy denied it and what the ones before it said, and a trace listing policies that never ran would answer that wrongly while looking complete.
Every policy deferring raises. DecisionUndecidedError, not a default —
defaulting to allow turns an incomplete policy set into an open door, and
defaulting to deny turns it into an outage whose cause reads as a legitimate
refusal.
A policy that does not answer raises too. A non-decision, a throw, and running
past policyTimeoutMs are one error — DecisionPolicyError, naming the policy
and carrying the trace so far — because they are indistinguishable downstream and
handling one but not the others is how a broken policy becomes a skipped one. The
chain stops there rather than falling through to whatever the next policy would
have said.
policyTimeoutMs is required and has no default. Pick a number your slowest
policy comfortably beats. The framework will not choose it for you: a policy that
never settles hangs every caller of the operation it guards, and a default here
is a number nobody chose applied to code the framework has never seen.
The same allow/deny/defer type is what an event topic declared
mode: 'decision' uses in stitchkit/live — one vocabulary, because a listener
voting on an event and a policy voting on a request are answering the same
question.
→ ADR 0155.
Admission and graceful shutdown
Use the application operation lease for work that is not already counted by a managed resource:
const operation = app.admission.acquire()
if (!operation) return Response.json({ error: 'unavailable' }, { status: 503 })
try {
await performAcceptedWork()
} finally {
operation.release()
}
release() is idempotent. Admission and counter increment are atomic, so work
cannot slip between the shutdown check and drain accounting.
managedServerResource already owns contract/streaming-route response lifetimes. It cancels
their supplied signals at admission close and awaits source cleanup before dependent resources
close. Do not add application admission leases or a second cancellation registry for those
streams. A source that ignores cancellation or fails its cleanup prevents a clean shutdown;
the existing grace and force budgets still bound the result.
Bounded operation admission
Compose createBoundedAdmission when accepted work also competes for a finite
process-local resource:
const generations = createBoundedAdmission({
upstream: app.admission,
policy: {
global: { maxConcurrent: 8, rate: { limit: 120, intervalMs: 60_000 } },
perKey: { maxConcurrent: 1, maxKeys: 2_000 },
},
})
await generations.run(accountId, ({ signal }) => generate({ signal }), {
signal: request.signal,
timeoutMs: 30_000,
})
Acquisition is no-queue and atomic across every configured budget. Refusal names
the exact bound; only a rate refusal carries retryAfterMs. maxKeys keeps the
per-key registry finite, and expired idle entries are retired.
The caller timeout is a wait budget, not proof that the resource stopped. It
aborts the signal and settles the caller, but the lease remains active until the
underlying Promise actually settles. drain() therefore reports real work;
force() closes admission and reports remaining work without claiming to have
terminated it. → ADR 0118.
Bounded revision wake-up
Use createRevisionSignal when several operations need to wait for the same
process-local fact to change, but the change itself is not a queue item:
const connectionsChanged = createRevisionSignal({ maxWaiters: 256 })
const observed = connectionsChanged.getSnapshot().revision
const result = await connectionsChanged.wait(observed, {
signal: request.signal,
timeoutMs: 10_000,
})
// After changing the canonical connection state:
connectionsChanged.advance()
wait(after) resolves immediately when the signal is already newer than
after; this closes the race between reading a snapshot and registering the
wait. Waiting on the current revision parks one operation, up to maxWaiters.
Every terminal result carries the revision it observed and distinguishes
changed, timed-out, aborted, closed and capacity. A future after is
a caller error rather than a revision the signal could honestly promise to
reach.
One advance wakes every older waiter. It is not consumed by the first reader, does not retain event payloads and has no replay history. A bounded channel is for delivering items to one reader; a credit window grants a finite resource.
The signal is already its whole lifecycle handle. Call its idempotent close()
from the close phase of the managed resource that owns the changing fact;
there is no second managed-resource wrapper around the same state. → ADR 0163.
Bounded delivery and byte credit
createBoundedChannel is for one asynchronous reader when an event bus is not a
queue:
const output = createBoundedChannel<string>({
policy: 'ordered',
maxItems: 64,
maxBytes: 256 * 1024,
sizeOf: (value) => new TextEncoder().encode(value).byteLength,
})
const progress = createBoundedChannel<{ revision: number }>({
policy: 'latest',
maxItems: 1,
maxBytes: 128,
sizeOf: () => 128,
})
ordered never overwrites accepted values; overflow is a reasoned refusal.
latest retains exactly one pending replaceable value and reports
coalesced. Offers never create a hidden writer queue, and only one next() may
wait. Close chooses drain (default) or discard; abort discards; failure
rejects the parked and all later reads.
createCreditWindow({ capacityBytes }) is the smaller primitive for a protocol
that already owns its queue but needs exact byte permission. Each credit lease
replenishes once; it is flow-control credit, not a durable acknowledgement. The
application snapshot sink now shares the same latest-value mechanics without
changing its revision or status contract. → ADR 0119.
Bounded local diagnostic journal
Use createDiagnosticJournal when a process needs finite, ordered local metadata evidence and the
deployment log pipeline is not the right boundary:
It lives at stitchkit/application/diagnostic-journal, not in the main barrel:
it is the one part of the kernel that spawns, locks and writes files, and while it
was exported from stitchkit/application that single line made the whole
entrypoint unusable in a browser bundle. Its schemas stay in
stitchkit/application.
import { createDiagnosticJournal } from 'stitchkit/application/diagnostic-journal'
import { z } from 'zod'
const journal = await createDiagnosticJournal({
eventSchema: z.object({
kind: z.enum(['resource_failed', 'recovery_started']),
resource: z.string().max(80),
}).strict(),
path: '/var/lib/example/diagnostic.jsonl', // operator configuration, never request data
limits: {
maxEventBytes: 4 * 1024,
maxPendingItems: 128,
maxPendingBytes: 512 * 1024,
maxFileBytes: 8 * 1024 * 1024,
maxFiles: 4,
},
onFailure: (failure) => internalLogger.error(failure),
})
const result = journal.submit({ kind: 'recovery_started', resource: 'database' })
if (result.outcome === 'refused') internalCounter.add(1, { reason: result.reason })
// During managed-resource close. Timeout ends this wait, not the physical append.
await journal.close({ timeoutMs: 5_000 })
The owner schema and JSON serialization run synchronously before admission. Accepted frames carry a process epoch and contiguous sequence and retain their complete bytes inside both pending limits until their append attempt settles. Capacity, invalid, oversized, closed and terminal-failure refusals are explicit; accepted ordered frames are never evicted.
The absolute path’s parent must already exist and be operator-controlled. One manager owns it via
an exclusive .lock; new files use mode 0600 by default. maxFiles includes the active file,
and a non-newline startup tail is rotated intact rather than guessed or repaired. An abrupt process
death may leave a lock. The default refuse policy leaves recovery to the operator;
lock: 'reclaim-stale' compares machine, boot and process birth identity before
recovery. A reused PID cannot impersonate a modern owner. Legacy records with a
live PID, foreign namespaces and unavailable identity refuse automatic recovery;
see the upgrading guide for quiescent legacy recovery. The same mechanism owns the
short-lived reclaim guard.
flush() means every accepted append through that call’s boundary settled. It is not fsync, a
durable receipt, exactly-once execution or remote delivery. Timeout/cancellation bound only the
waiter; the writer retains physical capacity until settlement. There is no upload or managed replay API.
Use a durable application store or deployment-owned log collector when durable restart recovery, replay or
aggregation is required. → ADR 0134.
For finite inspection, use the shared reader with an explicit oldest-to-newest list of existing files. Keep anomaly reporting outside the journal being inspected:
import { readDiagnosticJournal } from 'stitchkit/application/diagnostic-journal'
import { z } from 'zod'
const eventSchema = z.object({
kind: z.enum(['resource_failed', 'recovery_started']),
resource: z.string().max(80),
}).strict()
for await (const row of readDiagnosticJournal({
paths: ['/var/lib/example/diagnostic.jsonl.1', '/var/lib/example/diagnostic.jsonl'],
eventSchema,
maxLineBytes: 4 * 1024,
})) {
if (row.type === 'anomaly') internalLogger.warn(row.anomaly)
else inspectEvent(row.frame.event)
}
The reader bounds memory by line size plus a fixed chunk, skips oversized/damaged rows with
explicit anomalies, and still returns valid rows before and after them. offset is zero-based
bytes; line is one-based. position distinguishes the file’s tail from interior damage.
A valid frame lacking its final LF yields a warning with skippedBytes: 0 and its frame.
Each file’s size is captured at open; concurrent appends are excluded. Missing files, permissions,
symlinks, non-regular files and mid-read truncation throw rather than masquerading as bad data.
The caller owns file selection and coordination with an active rotator.
Writer startup inspects retained and active files under its lock. On damage,
journal.getStatus().recovery contains counts and first/last anomaly locations, reconstructed
again on restart from the preserved bytes. This describes startup observations, not live
continuous inspection. Normal rotation retains the existing finite eviction policy.
→ ADR 0219.
A file opening cannot keep in place — a torn active file with maxFiles: 1, a generation name
that is not a regular file, a retained file it cannot read — is renamed to
<file>.quarantined-<epoch> beside it, and the journal starts (onStartupRefusal: 'quarantine',
the default). Nothing is deleted; quarantined files are outside retention and yours to remove.
Every open lists them in getStatus().recovery.quarantined until you do: the ones it moved
first, with their reason, then the ones earlier runs left, without one — at most 32, and
quarantinedUnlisted counts the rest.
A journal whose damage must stop the process — one that is a source of truth, not diagnostics —
declares onStartupRefusal: 'fail' and gets a typed DiagnosticJournalRecoveryError:
import { DiagnosticJournalRecoveryError } from 'stitchkit/application'
// A diagnostic journal: start, then say what is set aside.
const journal = await createDiagnosticJournal({ ...config })
const { quarantined = [], quarantinedUnlisted = 0 } = journal.getStatus().recovery ?? {}
for (const entry of quarantined) {
// `reason` is present only for a file this start moved; an earlier run's has none.
internalLogger.warn({ file: entry.file, movedTo: entry.quarantinedAs, reason: entry.reason })
}
if (quarantinedUnlisted > 0) internalLogger.warn({ quarantinedUnlisted })
// A journal that is a source of truth: refuse to start over damage.
try {
await createDiagnosticJournal({ ...config, onStartupRefusal: 'fail' })
} catch (error) {
if (error instanceof DiagnosticJournalRecoveryError) {
// error.reason: 'torn-without-retention-slot' | 'not-a-regular-file' | 'unreadable'
}
throw error
}
A quarantining journal writes, so the managed resource that owns it reports healthy, and its
dependants read recovery from the value they use(). There is no degraded journal: the kernel
degrades only optional resources, and a required resource cannot depend on one.
DiagnosticJournalRecoveryStatusSchema is evolving and .strict(); if you embed it in your own
protocol, every change to its shape is listed under breaking changes.
→ ADR 0240.
Shutdown performs one phase barrier at a time: stop admission everywhere,
cancel future schedules, drain admitted work, then close in reverse stable
topological order. Every hook shares the same grace deadline. Forced cleanup
shares one force deadline. Repeated shutdown() calls return the cached promise;
a repeated process signal forces that same chain.
Do not add a second process.on('SIGTERM') handler around the application.
bindProcessSignals(app) is the force/escalation owner. Exit code and hard-exit
policy remain application/supervisor choices.
Where the budget comes from on the signal path. bindProcessSignals(app)
with no shutdown forwards nothing, so the application spends the budget it
declared in createApplication({ shutdown }). Pass shutdown to the binding
only to override that budget for signals specifically; passing one key overrides
that key alone and leaves the other at the declaration. Declare the budget once,
on the application — repeating it in both places is how the two numbers start to
disagree, and the operator’s supervisor timeout is calculated from the one they
can read.
Managed schedules
Schedules activate only after top-level readiness:
createManagedSchedule({
id: 'reconcile',
everyMs: 5_000,
startAfterMs: 0,
overlap: { mode: 'queue-one' },
errorPolicy: 'continue',
onError: (error) => internalLogger.error(error),
run: ({ signal }) => reconcile(signal),
})
skipignores a due tick while one execution is active.queue-oneretains one successor; additional due ticks coalesce.parallelrequiresmaxConcurrent; overflow skips rather than creating a hidden queue.
The callback receives the application shutdown signal. Normal shutdown waits
for admitted executions; force aborts the shared signal. There are no retries,
cron/timezone rules, persisted cursors, backfill or cross-process locks. Put
those semantics in a durable application scheduler when they are required.
Schedule cadence stays monotonic, while status snapshots expose ISO wall-clock
capturedAt, changedAt and next/last-run timestamps for portable observation.
Wake a schedule when work appears
An owner that knows the moment work arrives keeps a long period as the check for a lost event and
calls wake() from the place that enqueues:
const drain = createManagedSchedule({
id: 'drain-queue',
everyMs: 60_000,
run: ({ trigger, signal }) => drainQueue(signal, trigger),
})
function enqueue(job: Job) {
queue.push(job)
drain.wake()
}
wake()asks for one run now and takes the timer’s path:dependsOn,overlap,errorPolicy,onError, the shutdown signal and the counters all apply.- Calls made in one tick, or while a run holds every slot, collapse into one following run, under
every overlap policy (a wake is an event, so
skipdefers it instead of dropping it). - Before the schedule is activated (its dependencies are not ready) and after it stops accepting
(shutdown,
stop-schedule),wake()does nothing and is not remembered. - The period keeps its own fixed-rate grid and is not moved by a wake, so it stays a single timer chain and never becomes a second queue.
runreceivestrigger('timer'or'wake'); status reportsrunsWoken,wakesCoalescedandlastTrigger.
Operational projection
Declare aggregate activity stages, then update them with anonymous handles:
const generations = createActivityProjection({
id: 'generation',
stages: ['queued', 'running', 'finalizing'],
})
const activity = generations.open('queued', 'queued')
generations.transition(activity, { stage: 'running', state: 'active' })
generations.transition(activity, { stage: 'finalizing', state: 'active' })
generations.complete(activity)
Item identity is deliberately absent from the snapshot. Observers receive aggregate counts with application identity, process epoch, revision and timestamps. Calling a terminal method twice does not double-count.
const sink = createApplicationSnapshotSink({
write: (snapshot) => publishOperationalSnapshot(snapshot),
})
const unsubscribe = generations.subscribe((snapshot) => {
sink.publish(snapshot)
})
// During application cleanup:
unsubscribe()
await sink.close()
The sink replays the current absolute value and coalesces obsolete pending revisions behind a slow write. Sink failure is observable but cannot fail the application operation that changed the projection.
Optional grammY adapter
Install grammY only in an application that imports the adapter:
import { Bot } from 'grammy'
import { grammyBotResources } from 'stitchkit/application/grammy'
A long-polling bot is two resources, and one call gives both:
const bot = new Bot(env.BOT_TOKEN)
bot.command('start', (ctx) => ctx.reply('Hello'))
bot.catch(({ error }) => logger.error('Telegram handler failed', { error }))
const telegram = grammyBotResources({
bot,
dependsOn: [database, http], // what handlers need before the first update
configure: () => bot.api.setMyCommands([{ command: 'start', description: 'Start' }]),
onEnded: ({ error }) => stopProcess(error), // see below
})
const app = createApplication({ id: 'bot', resources: [database, http, ...telegram.resources] })
The ids are stable — telegram-configuration, then telegram-polling — so a
monitor, a test or app.restart() can name them. configure runs on every
start, before polling, so a command menu that failed to publish fails the start
instead of hiding behind a bot that answers.
Updates are admitted by batch. grammY confirms a batch to Telegram by
asking for the next one, and it asks as soon as the batch went through
middleware — whatever the middleware did. An update that arrived while the
application was starting, degraded or stopping was refused by its handler and
then confirmed as handled. The polling resource now waits for admission before
asking for a batch, holds one admission lease for the batch it returns to
grammY, and releases it when grammY asks for the next. On shutdown the batch in
hand is finished before bot.stop() confirms the offset, so a stop neither
loses an update nor hands the next process one this process already handled; a
batch that arrives once the application will not admit it is never returned, and
Telegram keeps it for the next process.
When polling ends on its own — Telegram refused the token (401), another
process took the bot (409), or bot.catch rethrew — grammY’s poller does not
recover in-process. The resource becomes failed, the application stops being
ready, and onEnded is called once; it is never called for a stop the
application asked for. The framework does not exit processes (ADR 0074): the
entry point turns onEnded into its own shutdown and a non-zero exit so the
supervisor starts a fresh process. The Telegram bot template shows the one way.
429s are the application’s for its own replies: install
@grammyjs/auto-retry on bot.api.config with a bounded delay. Polling itself
already waits Telegram’s retry_after, and the broadcast and operator channel
in stitchkit/telegram handle theirs.
grammyPollingResource is the polling half on its own, for a bot that needs no
configuration step; it admits updates the same way.
For webhook ingress, createGrammyWebhookResource wraps provider-owned update
handling with admission and drain. The application still mounts the HTTP route,
verifies/configures its webhook and chooses the grammY framework adapter. An
update accepted before shutdown may finish; a later update is rejected.
The adapter never reads the token or the environment, changes bot.catch,
chooses retry plugins, drops pending updates, sends outbound messages or stores
updates. A durable inbox/retry worker remains an application component, not a
mode of this adapter.
A resource that fetches work
context.admission is the application’s admission as a resource sees it.
acquire() answers now; acquireWhenAccepting(signal) waits until the
application admits and then holds one operation — the right move for a poller or
a queue consumer, which should not fetch at all while the application is not
ready. It rejects with the signal’s reason on abort, and with
ApplicationAdmissionError once the application is shutting down or failed.
What disappears from an application
A typical small service can replace roughly 120–220 lines of generic signal, timer, admission and close bookkeeping with resource declarations. A larger multi-resource service commonly carries 250–450 generic lifecycle lines plus 15–30 lines for each periodic timer. These are directional review estimates, not an API guarantee.
The responsibility change is more important than line count:
| Before | After |
|---|---|
| hand-written phase enum and readiness waiters | createApplication snapshot |
process.on signal loop and escalation |
bindProcessSignals(app) |
| interval/timeout handles and overlap flag | createManagedSchedule |
| global in-flight counter and drain waiter set | application operation leases |
| manual stop/close fan-out | resource graph + bounded shutdown |
| progress delta events | absolute activity projection |
Durable job tables, lifecycle journals/outboxes, provider inboxes and business retry rules do not disappear. They were never process-local glue and remain application-owned.
Durable process facts and owner notifications
createProcessLifecycleLedger records a bounded versioned list of process runs.
Start, readiness, draining and shutdown target both runId and pid, so hot
reload and PID reuse cannot close the wrong generation. lifecycleLedgerResource
drives it from the kernel’s phases: start records the start, activate the
readiness, stopAdmission the drain — the run’s unavailableAt, the moment it
stops answering new work — and close / force the stop. A run that stops
without ever becoming ready (a failed startup rolls back with close only)
ends as startup-failed, not clean.
A start classifies what happened to the newest run before it, and the classification is decided by four facts — whether that run recorded its own exit, whether the pid is the same, whether the version changed and whether it ever became ready:
| newest run | same pid | version | previousExit |
predecessor’s termination |
|---|---|---|---|---|
| none | — | — | first-boot |
— |
recorded stoppedAt |
— | — | the recorded termination, including interrupted |
unchanged |
| still open | yes | — | hot-reload |
hot-reload, closed at the new start |
| still open | no | changed | handoff (default) |
stays open; it records its own stop later |
| still open | no | changed, versionChangeOverlap: 'abnormal' |
interrupted / abnormal / startup-failed |
closed at the new start, as for the same version |
| still draining | no | same or non-overlapping build | interrupted |
closed at the new start; its death time and kill signal are unknown |
| still active, was ready | no | same | abnormal (default) |
abnormal, closed at the new start — an upper bound, the crash time is unknown |
| still active, never ready | no | same | startup-failed (default) |
startup-failed, closed at the new start |
Every other open run the start finds was abandoned too and is closed the same
way — two crashes in a row leave two open runs, and both are closed, not only
the newest. Open means active or draining.
A version of unknown on either side is never a version change, so a dev build
after a crashed release reads abnormal, not handoff. forced means the
process itself acknowledged a kill; abnormal means a successor found it dead.
Downtime is the window nobody answered in
ReadyFact.downtimeMs is readyAt of the new run minus unavailableAt of the
last run that served — the predecessor’s drain plus the successor’s boot. A
110 s forced drain followed by a 4 s boot is 114 s of downtime; the gap between
the two processes alone would read 4 s before and after that defect was fixed.
Runs that never became ready answered nobody, so the count passes them: after a
failed start it still opens where the last serving run stopped answering. It is
0 when that run still answers (a handoff whose predecessor has not drained
yet) and null on first boot and when the end was never timed — a crash or a
hot reload records no unavailableAt. unavailableSince names the moment it is
counted from. The gap between two processes is still there, under its own
name: StartFact.processGapMs is the start minus the predecessor’s recorded
stop.
The ledger records what the process says while it can say it. stitchkit never
calls process.exit (ADR 0074), so a stop is recorded on the kernel’s close
or force path, both awaited. An application that exits the process on its
own before its shutdown settles loses that last write; the successor then
finds the unfinished drain and closes it as interrupted, keeping the unavailableAt
already recorded — which is why the drain moment is written first, at
stopAdmission, and the downtime survives a stop that was never recorded.
An abandoned draining run closes as interrupted, including before readiness. StartFact.previousDrainUpperBoundMs is the elapsed time from recorded admission stop to successor start, or null when unknown. This is an upper bound, not proof of a supervisor timeout or a particular kill signal. Live handoff predecessors remain open; same-pid restarts remain hot-reload.
The abnormal default is a choice, and it is the single-process deployment:
one process per build, a new pid of the same build means the old one stopped
answering. Where processes of one build overlap on purpose — a cluster, a
zero-downtime reload of the same build — pass sameVersionOverlap: 'handoff'
to the ledger, and every open run stays open until it records its own
shutdown. The cost of that setting is symmetric: a real crash under it is
reported as a handoff and the dead run stays open in the ledger until
retention drops it.
A version change is the other axis, and its default is the opposite:
handoff, because a release that starts the new build before stopping the
old one is the common zero-downtime shape. Where builds never overlap — one
process per deployment (a pm2 fork, a systemd unit, one container), the old
one stopped before the new one starts — pass versionChangeOverlap: 'abnormal'. An open predecessor of another build then died without
recording its exit, most often killed during the release itself: it is closed
as abnormal (or startup-failed) at the new start, downtimeMs is null
instead of 0, and previousVersion / versionChanged still describe the
release. A predecessor that did record its stop is read as before, and its
downtime is measured. The two settings compose: sameVersionOverlap: 'handoff' keeps open only the runs of a build allowed to overlap the new one.
The list is kept in the order the transitions wrote it —
every write goes through one atomic update, so that order is the causal one,
and a successor whose clock lags its predecessor still finds it at the head;
startedAt is data, not the sort key. Retention (retain, default 20) drops
finished runs first and an open one — a live handoff predecessor — only when
nothing finished is left. Facts are published through the ledger’s own
subscription and resource value; ApplicationEventSink remains the strict
application-state stream.
createNotificationOutbox is the transport-neutral durable delivery side. It
persists before send, claims an item with an expiring lease, carries one stable
idempotency key into the transport, retries under an injected clock and records
terminal drops. The guarantee is at-least-once: a crash after remote acceptance
but before the receipt is persisted may redeliver, so transports should use the
key when they support deduplication. The retry budget is sized for the outage
an owner notification has to outlive, not for a flaky call: the default backoff
doubles from one second and caps at sixty (backoffDelay, the one formula the
client’s resumable streams also use, with jitter 0), and the 99 waits between
the default maxAttempts of 100 sum to 1+2+4+8+16+32 s plus 93 × 60 s — about
94 minutes — before onDropped sees attempt-limit.
state() is a read and never fails on the bounds a transition enforces — a file
that grew past maxStateBytes under an older limit is inspectable and is
trimmed by the next transition. Two bounds do fail loudly: enqueue past
maxQueue throws rather than dropping silently, and a backoffMs that returns
NaN, a negative or an infinite delay rejects the flush. stop() lets the
send in flight finish and claims nothing more; a send without its own
deadline holds stop() — and so the resource’s force() — for that one call,
so give the transport a timeout. Superseding a key that is being sent right now
does not recall it: the send completes, and only its receipt is not written.
createDirectoryInbox receives entries from another program on the same host,
or directly from a transport adapter. Import the resource from
stitchkit/application/directory-inbox; its contracts live in
stitchkit/application. Its published handle has accept({ source, key, entry }),
flush() and state():
const inbox = createDirectoryInbox({
id: 'updates',
directory: env.INBOX_DIRECTORY,
schema: UpdateSchema,
handle: ({ identity, entry, signal }) => processUpdate(identity, entry, signal),
})
const ingress = defineManagedResource({
id: 'ingress',
dependsOn: [inbox],
start(context) {
const intake = context.use(inbox)
return { value: (key: string, entry: Update) =>
intake.accept({ source: 'client-updates', key, entry }) }
},
})
accept is available after the resource starts, including before activation.
It validates the entry and full producer identity, writes a lossless JSON envelope,
syncs its file and parent directory, then returns
{ status: 'accepted' | 'duplicate', filename }. Automatic delivery starts at
activate; an explicit flush() is available before activation. A producer
should acknowledge its provider only after acceptance resolves. Failure after
publication can mean the entry was accepted but its acknowledgement was lost:
retry the same identity, and duplicate keeps the original payload.
The identity is the pair source (1–128 characters) and key (1–1024 characters).
A hash of the pair names the file; the full pair is retained inside its envelope
and checked against existing files and records. Two sources may use the same key.
The handler’s legacy key remains the filename; programmatic entries additionally
carry identity. No Telegram numeric update id or MTProto event shape is invented.
The default maxPendingEntries: 1000 refuses new programmatic entries when the
pending directory is full, without publishing another file; duplicate acceptance
still succeeds. maxEntryBytes defaults to 1 MiB and counts the complete UTF-8
envelope. A completed identity is deduplicated while its receipt is retained
(retain, default 1000); a rejected envelope also deduplicates while its file
remains in rejected/. Choose the retention and rejected-file cleanup policy to
match the provider’s replay window. Provider cursors, identity extraction and
external-effect idempotency remain application responsibilities.
The original file-producer path also remains: atomically publish
<name>.json, using a temporary name followed by a rename. Names starting with
a dot are ignored; intake-<64 hex characters>.json is reserved for programmatic
envelopes. Entries are delivered in filename order. The reader refuses unsafe
leaf symlinks, multiple links and files changed during reading.
stopAdmission stops new acceptance and delivery, drain waits for accepted
storage operations and the handler in flight, and force aborts its handler
signal. A handler ignoring that signal may still finish an external effect;
the inbox leaves its entry unsettled rather than claiming it was cancelled.
Taken and done are separate durable records. Before handle, the entry is
claimed with a unique expiring lease (leaseMs, default five minutes). A
long handler renews that lease; an expired or replaced lease cannot complete,
reject, retry or remove the successor’s file. Success first commits a receipt,
then removes only that file generation under a new protected transaction.
A crash after receipt commit is recovered by cleanup without redelivery. A crash
before receipt commit may redeliver: the guarantee is at least once, so use the
full producer identity as the external idempotency key.
A throwing handler retries after a backoff from one second to five minutes.
Schema failures, oversize files and maxAttempts exhaustion (default 20) are
recorded before their files move to rejected/, with onRejected reporting the
reason. The state defaults to <directory>/.inbox-state.json, or uses the supplied
store. Manual file producers are responsible for atomic publication and capacity.
These primitives share the structural StateStore. update owns one atomic
read/modify/write transaction and passes a StateStoreUpdateContext as the
transition’s second argument. Its await context.assertHeld() checks that the
transaction is still active and its exclusive generation is still owned. A custom
store must implement both real serialization/atomic commit and this guard;
a no-op guard does not satisfy the contract. Ordinary one-argument transition
callbacks remain valid.
On a server, createFileStateStore uses the same process-instance lock as
withExclusiveLock, unique staging files and atomic directory-durable publication.
lockTimeoutMs (default 10 s, zero means one attempt) bounds acquisition only.
A live or unattributable owner is never replaced by age, including after an event
loop stall or SIGSTOP; only a provably dead local process instance is reclaimed.
Legacy readable pid/token records without a host are unknown owners and need
operator recovery after the old writer is verified stopped.
staleLockMs and retryMs are removed and are refused at runtime. The current
write removes its own staging file; unknown historical temporary files are
preserved, without an age sweep. Directory durability requires a filesystem
supporting directory sync (Linux and macOS); Windows is refused explicitly.
Ledger corruption may be declared reconstructable; an outbox must fail closed
rather than discard pending delivery.
Recovering change subscriptions
changeSubscriptionResource is an optional resource over a consumer-owned driver.
It uses a finite declared key set, subscribes before reading, reconciles after
reconnect and periodically repairs lost hints. reconcileIntervalMs is explicit;
notifications alone cannot prove freshness. Within a key, hints coalesce into one
trailing read and a failed read retries with backoff. Use the supplied
context.commit(() => publish(value)) after awaits to reject an obsolete connection.
Drivers implement connect(signal) and return subscribe({ signal, hint, disconnected }) plus close(). Subscribe must establish the notification boundary
before resolving. A database LISTEN driver owns its connection, channels, decoding
and transaction ordering; no SQL or database driver is included here. The resource
owns reconnect, deadlines and shutdown. Drivers must honor abort. If one ignores it,
its slot remains occupied until settlement; deadlines cannot safely manufacture
successful cleanup or permit an overlapping connection.
For watched reads, set reconcileIntervalMs on the existing createWatchHub rather
than building a second read loop. Its per-key machinery is shared with managed
subscriptions. Limits default to 1024 sources and 1024 subscribers per hub; set
explicit finite values for larger workloads. Supply a server-verified admission
scope to attach and abort it on logout, permission revocation or session replacement.
The read callback receives the trusted scope as its third argument. A client owner
ID in arguments is not authentication.
Multi-action notification delivery
Opt into delivery: { version, actions, project } on createNotificationOutbox.
The pure actions(payload) selector runs at enqueue; the versioned plan remains
fixed throughout retry. Each send receives its action and a stable idempotency
key; return a small JSON receipt such as a remote message or uploaded object ID.
An action that returned is confirmed and is never sent again: a result that is not
JSON, or does not fit the state’s byte budget, is kept as receipt: null with
unrecorded: 'not-json' | 'too-large' and reported to onError.
project(notification, receipts) applies all confirmed results locally and must
be idempotent. Checkpoints and the final projection timestamp live in the existing
atomic StateStore; completed receipts retain them within retention/byte limits.
This supports, for example, text plus attachment delivery with a local timeline projection, or upload plus publication with a local asset index. These are protocol examples, not claims of deployed consumer adoption. A failure in projection does not repeat confirmed sends. A crash after remote success but before receipt commit is ambiguous: the provider must deduplicate the action key or reconcile its result. A projection can also repeat if its completion checkpoint is lost.
A plan keeps its actions from enqueue, so an older version needs only its
projection to finish: a new executor lists it in retiredVersions: { v1: project }
until the queue has drained. A plan version the executor does not know is reported
to onError as NotificationPlanVersionError and waits — at least a poll interval,
spending no attempt, without classify — for a process that knows it: the other
side of a rolling deploy, the version a rollback returns to, or this one once it
lists the version. Whole-state file and memory stores
remain supported. Atomic enqueue with an unrelated business transaction is not
provided by this contract; retain the application’s transactional adapter when that
boundary is required. Send and projection callbacks own their external deadlines.