Vigil: monitoring that knows what is actually running

01Executive summary

Most monitoring answers one question: is it up? Vigil also answers the question that matters for a small estate of internet-facing applications: what vulnerable software is each app exposed to, judged on what is actually running right now.

It does this with a small set of ideas. Lightweight testers on each machine send signed results to a core. The core is the only place that decides anything, through a single alert lifecycle with stored deadlines. Everything it needs to say to a person goes through a durable outbox. Package lists from the images that really run are matched to public advisories. And one colour rule means a screen can never show green while something is wrong.

02Purpose and design principles

For a small estate of internet-facing apps, the bigger risk is vulnerable dependencies building up quietly. Vigil makes that visible per app and to clear alerts the moment the evidence says they are gone.

03Architecture: testers and a core

A small agent, the tester, runs on each machine and executes only the checks that machine's manifests declare. It initiates every conversation, pushing results outwards, so no machine needs an inbound path for monitoring. The core receives results, verifies them and hands them to the evaluator.

Signed, sequenced, manifest-bound

Every message between a tester and the core is a signed envelope (Ed25519 over the exact payload bytes). Each tester makes its own key on its own machine at enrolment, and the core keeps only public keys, so a copy of the core's database cannot impersonate a tester. Results carry a sequence number, so a replayed or older result is kept as history and never changes the current picture. Each result is also bound to the manifest version in force, and a manifest that does not conform to the contract is shown as non-conforming rather than silently ignored.

The core signs the checklists and actions it sends down. A forged or altered instruction fails at the tester, and a tester refuses anything that is not on its own declared checklist. The core never trusts the tester's clock: freshness is measured by when the core received a result.

04The one alert lifecycle

An earlier version spread decisions across several places, so a page, a screen and a digest could disagree. The redesign puts every decision in one pure function: given the current finding, an event, the policy and the time, it returns the new finding and the side effects to queue. The result and its side effects are written in a single database transaction, so a crash leaves all or nothing.

• A finding has one state (healthy, failing, alert, notice, unknown) and a few flags (stale, escalated, acknowledged, in maintenance, flapping, asleep, grouped, re-checking). Service and host states are worked out from their checks when read, worst first.
• Deadlines are stored, not remembered. Grace ends, escalation, staleness and re-check timeouts are columns in the database. One timer loop picks up whatever is due. After an outage it catches up once inside a restart grace, so a gap does not page everything as stale.
• Escalation. An alert left open past its escalation time is marked escalated and pages once.
• Reopen, don't duplicate. A recurrence shortly after a close reopens the same incident (same number), cancelling a pending all-clear rather than sending noise.
• Flapping holds opening, never closing. A noisy check waits for a stable period before opening, but a good result always closes.
• Check now. A signed request asks a tester to re-measure immediately, coalesced and rate-limited. It clears nothing itself; only the fresh result does.
• Acknowledgement. People only, with a reason and an end date. It stops paging and escalation but the finding never turns green. Agents, tokens, the hand-off and AI cannot acknowledge.

The transition table is generated into tests: every state and flag against every event, plus out-of-order evidence, restarts at each kind of deadline, two timer workers at once, and delivery failure and retry. Before each cut-over, every stored result to date was replayed offline through the new code and every difference from what was recorded was explained. The incident-opening replay covered over half a million results with no unexplained difference.

05Notice policy

Not everything wrong is broken. A pending software update or a published fix is a notice: something to act on within a grace period, not an emergency. One versioned policy file holds the rules, and each deadline records the policy version that set it, so a policy change affects new deadlines only unless an operator deliberately re-evaluates.

Each class has its own grace period, shortest for security fixes.

Testers report facts only: what is running, what is available, and the advisory. The core computes the deadline, and fetches publication dates from the public source itself rather than trusting the tester. The grace counts from the oldest unapplied release, so a weekly release train still goes overdue. An agent claiming a future date cannot push a deadline out. At the deadline, from the timer, a notice becomes an ordinary alert, and on a high-severity service it pages. Whether a vulnerability is on the public known-exploited list is also read by the core, and if that list is stale, "exploited" is treated as unknown rather than as no.

06The outbox and the hand-off courier

Every message Vigil sends (page, all-clear, digest, catch-up after alerts were switched off, work-item request) is a row in an outbox, written in the same transaction as the decision that caused it. Each row has an idempotency key. Workers deliver, record attempts, retry with backoff, and give up visibly; delivery is at least once and the keys make repeats harmless. If delivery keeps failing, the core's health endpoint fails so that an external watchdog pages. The system reports its own inability to speak.

Hand-off: the core decides, the courier carries out

When an alert has been open long enough, someone who owns the affected code should hear about it as a work item. Originally a separate script decided this for itself, a second decision-maker. Now the core decides, queues a hand-off row, and a small courier leases rows through a narrowly scoped API, creates, retitles or closes the work item, and reports back.

• Scoped credential. The courier's token works on its own routes only and every other credential is refused there. It cannot acknowledge, request a check or change anything but its own row.
• Safe to repeat. The courier finds an existing work item by a marker before creating one, so a lost report or replayed row never duplicates.
• Closes are delayed. A close is delayed and withdrawn if the incident reopens, so work items do not flap.
• Text is untrusted. A check's summary is written by a remote agent and ends up in a work item that people and tools read. The core cleans the text and quotes it in a block labelled as untrusted monitor output, not an instruction.
• Failures are visible. A failed hand-off shows on the incident and in the digest, and a stuck row turns a probe red.

07Dependency grouping and events

When a host goes down, every check on it fails. Without grouping that is a dozen pages for one cause. A check can declare depends_on another check. While that dependency has an open, measured failure, its dependents' failures are folded under it: one incident, one page. When the dependency recovers, a timer pass re-runs the ordinary rules for any dependent still bad, so nothing stays hidden.

Some facts are things that happened, not states that persist: a machine reset, a nightly job failed. A check of kind: event records a dated event that a person can dismiss. An event never opens an incident, never pages, and never holds a finding in alert. It appears in the daily digest. A dead event check is still caught by staleness.

08Vulnerability posture: packages matched to what runs

This is the purpose. Dependency scanners read lock files in a repository, which describes what was written, not what was shipped. Vigil judges the image that is actually running.

• Image identity from the runtime. Each tester reports the content hash of every running container's image, read from the container runtime, never from a label an image could set for itself.
• Binding. A package list counts as evidence only if a tester reports that exact image running under that service. Otherwise it is stored as "unclaimed" and shown as such. A credential can therefore only describe its own service's images, and can never hide another service's exposure.
• Public-registry-only queries. Only packages from public registries are sent to the public advisory database, and only their name and version. Private or internal packages are dropped, and the filter fails closed: if a package's origin is in doubt it is not sent. Requests are budgeted, size-capped and strictly parsed, with no redirects off the expected host.
• Unknown is not green. A running image with no package list, or a container no service declares, reads "coverage unknown". If the advisory service is unreachable the answer is unknown, not clear.
• Severity is taken from the record's own rating, else computed from its scoring vector, else shown as unknown. Known-exploited exposures are marked as such.
• Made from the image itself. The package list has to be made from the image that runs, once per new image and never on a schedule, with limits on CPU use.

09One colour rule

A pending update once showed green on one screen and amber on the one beside it. The fix was a single rule, written down as a rule: if there is an alert it cannot be green; if there is an update it cannot be green; alerts are not updates. One function returns exactly one of five colours, and every surface calls it: the map, host markers, chips, rows, check bars, tiles, headlines, filters and the digest headings.

• A failure that goes quiet stays red and is marked "gone quiet"; silence cannot turn it green.
• Flags such as acknowledged, escalated, folded under a dependency, in maintenance or being re-checked never change the colour, and neither do events.
• A service takes its worst check, a host its worst service. Labels count each kind ("2 alerts, 1 security update, 3 updates, 1 unknown") and read "all good" only when green.
• A probe fails if any surface disagrees, and was watched failing once by deliberately flipping one mapping. Paging is untouched by the colour rule.

10AI repair: built behind eligibility rules, switched off

An earlier version had model-driven triage and automatic repair. The upper layers were switched off and the foundation rebuilt. Repair has now come back as code with rules, but disabled, and it can only be enabled deliberately.

• Eligibility is re-asked at dispatch, at claim and before every action. Missing facts mean ineligible.
• The core verifies, not the job. After a job ends the core asks for a fresh check; only that result resolves anything. A job's own claim of success counts for nothing, and a result measured before the job ended does not count as verification.
• Agent-written text is untrusted. The model is told that check summaries are quoted input that may contain injected instructions.
• A failed repair goes to a person once. No automatic retry.

11Security principles

12How it was built

The first commit was on 13 October 2026. A foundation of tester, core, signed contract and manifests landed in the first days. A deliberate reset then removed unproven layers. The single alert lifecycle was designed, reviewed by two fresh AI reviewers with no prior context, and built in planned steps, most of them now live, each its own change with tests, probes and a review. About 380 commits in about two weeks, with AI assistants used for building and review.

• Design first, in writing. The lifecycle design states its promises, its exact rules and its build order before anything is built. Reviewers attack the design before the code exists.
• Replay before cut-over. Every recorded result is run through the new code offline, and each difference is explained before anything switches.
• One change, one owner. Each step is a single change with one owner, and a separate reviewer checks it against the design.
• Decisions are logged. Every significant choice has a numbered decision with the reason it was made.

13Status and what is next

Next: further rollout of exposure handling, then an optional "AI fix" action for eligible incidents.