homelab-helper — Roadmap¶

For the concrete, checkbox-tracked task list of what's actually left (remaining Phase 1 work + the Phase 6 trust-gradient build), see backlog.md. This doc is the narrative; the backlog is the punch list.

Principles¶

Three rules that shape every phase below.

1. Each phase produces a usable system on its own. Not a milestone in a partially-built product; a thing you'd actually run. If you stopped after Phase 1, what you have is a discovery-and-audit tool that's already valuable on its own. If you stopped after Phase 2, you have a continuous-monitoring tool with assertion verification. And so on. This forces clean abstractions and prevents the framework from being held hostage by future phases.

2. Acceptance criteria are demoable, not aspirational. Every phase has at least one criterion that ends with "...against the example.lan lab." Real artifacts, real tests, no "we'll know it's done when it feels right."

3. L2 (execution) is the final phase, gated by the trust gradient. Five phases get to a fully useful read-only system: discovery, continuous monitoring, multi-source coordination, conversational layer, planning. Phase 6 adds the executor — but every prior phase is independently complete, and L2 is opt-in: the framework never applies a change until the operator raises the trust floor for that specific class of action. The propose-only system (Phases 1–5) is the product for everyone who never opts in. L2 is committed to the roadmap, but deliberately last and deliberately gated.

Dependency graph¶

Phase 1 — Inventory & Discovery
    │
    ├─→ Phase 2 — Continuous Reconciliation
    │       │
    │       ▼
    │   Phase 3 — Multi-Source Coordination (adapters)
    │       │
    │       ▼
    │   Phase 4 — Conversational Layer (agents)
    │       │
    │       ▼
    │   Phase 5 — Planning & Recommendations
    │       │
    │       ▼
    │   Phase 6 — L2 Execution & Trust Gradient
    │       │
    │       ▼
    │   Phase 7 — Agentic Operations
    │       │
    │       ▼
    │   Phase 8 — Operate & Optimize
    │       │
    │       ▼
    │   Phase 9 — Road to 1.0
    │
    └── Cross-cutting: probe catalog growth, docs, test fixtures

Phases 2 and 3 are sequential as written but loosely coupled — could be parallelized once Phase 1 is solid. Phase 4 can begin once Phase 3 produces enough cross-source signal to be worth narrating; can ship narratively before all of Phase 3 is done. Phase 5 needs both inventory depth (P1-2) and external context (P3). Phase 6 needs the proposal stream and blast-radius/rollback metadata that the planner (P5) and reconciler (P1-2) produce — it executes what the read-only system has been proposing all along. Phase 8 turns the planners from snapshot reasoning into continuous, history-backed operations advice; its findings flow into the same proposal → trust gate → execution path Phase 7 built. Phase 9 depends on all of them being built, and on none of them being finished: it is the consolidation pass that turns eight phases of capability into something a second person can run.

Effort scale¶

Estimates assume Python-fluent developer at hobby pace — evenings and weekends, with normal life happening. Multiply by ~2 for realistic completion. Ranges reflect "fast track if you're focused" vs "comfortable pace with detours."

Phase 1 — Inventory & Discovery (MVP)¶

The foundation. After this phase, the framework can scan a network, deep-probe Linux hosts, populate NetBox, track parts across moves, run configuration assertions on a one-shot basis, and produce a meaningful audit. Everything later builds on this.

Goals¶

Implement the Slice 1 schema (11 tables).
Build the NetBox adapter with read+write for the modeled object types.
Build the Kernel-SSH adapter for warm host discovery.
Build cold and warm discovery probes as proper plugins.
Build the reconciler with current-state inventory maintenance.
Build the assertion engine for one-shot verifier runs.
Ship the CLI as the primary interface.
Produce the day-one audit against the example lab.

Deliverables¶

Deliverable	Notes
Harness DB models + Alembic migrations	Per `harness-schema-slice1.md`. SQLite default.
Probe plugin SDK	`Probe` base class, entry-point registration, output validation via Pydantic. Documented enough that external contributors can write probes.
Probes: `network.subnet-scan`, `network.fingerprint`	Asyncio-based; no nmap dependency required (nmap optional for deeper modes).
Probes: `host.identity`, `host.cpu`, `host.memory.dmidecode`, `host.storage`, `host.network`, `host.pci`, `host.gpu`, `host.services`	The warm probe suite. Each is a separate plugin.
NetBoxAdapter	Read+write Devices, Interfaces, IPs, VLANs, Prefixes, Clusters, VMs, Services, InventoryItems, custom fields.
KernelSSHAdapter	SSH session management, command execution, output parsing primitives shared across host probes.
Reconciler	Observation → inventory; current placement maintenance; finding generation.
AssertionEngine (one-shot mode)	Runs verifiers, generates AssertionRun rows and Findings. No scheduling yet.
FingerprintGenerator	Deterministic fingerprints for finding dedup.
CLI: `helper discover network`, `helper discover host`, `helper audit`, `helper findings`, `helper host show`, `helper config`	Primary interaction surface.
NetBox bootstrap routine	Creates the harness's custom fields in NetBox on first run.
Test fixtures: example lab as integration test	Replayable; CI-friendly.

Acceptance criteria¶

helper discover network 10.0.0.0/16 populates the harness DB and NetBox with discovered hosts, fingerprints recognized services (Proxmox, Cockpit, K8s API, web servers), and produces INVENTORY_GAP findings for unidentified hosts.
helper discover host node3 with valid SSH credentials completes the warm probe suite in under 30 seconds, populates Host.capabilities, creates PhysicalPart and Placement rows for every DIMM and storage device, and updates NetBox Device + InventoryItems accordingly.
helper audit against the populated example lab produces the eleven day-one findings from the modeling walkthrough, plus the new ones added in the gap-fill update (CONFIG-DRIFT-CONFIRMED for node0 e1000e, STRAY-CONFIG for WAN2, LEGACY-WORKLOAD for the esphome-worker LXCs, UNKNOWN-WORKLOAD for the stopped node2 VMs). At least eleven findings; ideally all sixteen.
Re-running discovery against the same lab does not duplicate inventory — fingerprint dedup works, placements close out cleanly when parts move, findings update last_seen rather than creating new rows.
Probe plugin SDK passes the "write a probe in 50 lines" test: writing a new probe (e.g. host.systemd-units) takes a single file under 50 lines of Python plus a Pydantic output schema.

Effort¶

6–10 weeks of focused evening/weekend work. The DB schema is already designed; the bulk of the time is probes, the NetBox adapter, the reconciler, and integration tests. The reconciler is the most subtle component — expect to rewrite parts of it twice.

Stop-here value¶

If you stopped after Phase 1, what you have is a network discovery + deep host inventory tool with audit output. That's already valuable — nothing existing combines those well for a homelab. You wouldn't have continuous monitoring, conversational interfaces, or planning, but you'd have something to run quarterly that finds the gaps in your lab.

Phase 2 — Continuous Reconciliation & Assertions¶

The shift from "discovery is something I trigger" to "the framework knows my lab in near-real-time." Adds the helper-agent and the temporal logic the reconciler needs to handle a continuously-updating world.

Goals¶

Build helper-agent as a deployable daemon for Linux hosts and Pi 4B.
Add scheduled probe execution (via Scheduler).
Add temporal awareness to the reconciler: confidence decay, stale-degradation findings.
Implement the OperationalIntent layer: "ask once, remember forever" pattern fully wired.
Schedule the AssertionEngine to run verifiers on cadence.
Build the configuration assertion library with your 16 supplied assertions as the starter corpus.

Deliverables¶

Deliverable	Notes
helper-agent daemon	Python, packaged as single binary. Systemd unit. Bearer-token auth. <50 MB RAM target. Works on Pi 4B.
`helper agent enroll <hostname>`	Token generation, agent install (via SSH push or pre-built deb/rpm).
Observation buffer/replay	Agent buffers locally when server unreachable; drains on reconnect.
Scheduler integration	APScheduler-based. Probe schedules per probe definition + per-host overrides.
Reconciler temporal logic	Confidence decay (VERIFIED → STALE after freshness window). Stale-degradation finding generation.
OperationalIntent reconciliation rules	`expected=off + observed=off = confirmation, not absence`. `STRAY` state suppresses findings forever for the WAN2 case.
AssertionEngine scheduling	Cron-style per assertion. Drift findings on transition pass→fail.
Seed assertion library	Your 16 ConfigurationAssertions translated into the framework's verifier_spec format. Shipped as a YAML bundle users can adopt.
`helper assert run <name>`	Manual trigger for ad-hoc verifier execution.
`helper intent set <target> <state>`	Operator command for declaring intent.

Acceptance criteria¶

helper-agent installs on node3 and three Pis without manual surgery, runs for 7 days unattended, posts observations every 5 minutes, never crashes.
Deliberately break one ConfigurationAssertion (e.g. disable the e1000e service on node1) — within 15 minutes, a CONFIG_DRIFT finding appears with the correct fingerprint and evidence.
operational_intent-tagged stopped VMs produce no finding. Stop claude-server on node2 with intent stopped-by-design; no UNKNOWN_WORKLOAD finding. Stop a different VM without intent; finding appears.
Power-policy-aware probing: node4 and node5 (WoL workers) probed during their expected-off windows produces no finding for unreachability. Probe runs are skipped or rerouted as designed.
The node0 e1000e drift gets confirmed automatically when the verifier runs against the gap-filled inventory — CONFIG_DRIFT_CONFIRMED finding generated without manual intervention.
Re-running Phase 1's helper audit after Phase 2's agents are deployed — same set of findings, but now with last_seen updating continuously and confidence reflecting real freshness.

Effort¶

4–6 weeks. The agent itself is small (1–2 weeks); the reconciler temporal logic is subtle (2 weeks); assertion verifier formats and the YAML corpus are non-trivial (1–2 weeks).

Stop-here value¶

With Phase 2 you have a continuous lab-monitoring tool with drift detection and intent-aware suppression. This is already more than most existing tools do — you'd be running it routinely and getting value daily. No conversational interface yet, no multi-source coordination, no recommendations, but the audit you ran quarterly is now alive.

Phase 3 — Multi-Source Coordination¶

The architecture's premise (multi-source SoT, not single SoT) only matters once you actually have multiple sources. This phase adds the adapters that make the example lab's full complexity visible.

Goals¶

Ship the ProxmoxAdapter, K8sAdapter, UniFiAdapter, CloudflareAdapter, GitArgoCDAdapter, OpenMediaVaultAdapter.
Wire each adapter into the engine so observations from each source flow through the same reconciler.
Implement synthesized views: service-to-host-to-pod resolution, DNS split-brain visibility, ArgoCD-vs-K8s drift, stray-config detection.
Add a query API that lets agents and the UI ask cross-source questions.

Deliverables¶

Deliverable	Notes
ProxmoxAdapter	Read cluster state, VMs, LXCs, Ceph health, OSD weights, replication status.
K8sAdapter	Read nodes, pods, services, labels, taints, events. Populates `local_context_data` on logical service VMs.
UniFiAdapter	Read DNS records, DHCP leases, switch port config, network/VLAN definitions, AP topology.
CloudflareAdapter	Read DNS records, ACME cert state, zones.
GitArgoCDAdapter	Read declared app state from Git; read ArgoCD application sync status; compare.
OpenMediaVaultAdapter	Read NAS state over OMV's JSON-RPC API: filesystems/pools, SMART/disk health, shared folders + NFS/SMB exports, service states. Read-only management-plane source for share/export inventory (and later stray-export detection).
Cross-source view builder	"Show me service X" returns the synthesized record across NetBox + K8s + UniFi + Cloudflare.
Stray-config detection	UniFi-configured networks/VLANs with no observed traffic in N days → `STRAY_CONFIG` finding.
Service endpoint reconciliation	ServiceEndpoint table populated; internal vs external endpoints visible.
New CLI verbs: `helper view service`, `helper view host`, `helper diff git-vs-cluster`	The query surface.

Acceptance criteria¶

Each adapter passes its smoke test against a real instance — helper adapter test proxmox succeeds against the example Proxmox cluster, returns expected observations.
helper view service home-assistant returns a complete record: NetBox Service entry, hosting VM (node0:105), K8s state (n/a — it's a Proxmox VM), UniFi internal DNS record, Cloudflare external DNS record, ArgoCD app status (n/a), recent observations.
DNS split-brain visible: querying any service produces both internal and external endpoint rows with the correct resolvers.
Git-vs-cluster drift produces findings: introduce a manual change to a K8s resource that ArgoCD manages, finding appears identifying the divergence.
Ceph health surfaced: helper view storage ceph shows OSD weights, near-full warnings, replication lag, and (still) the node0 1 GbE bottleneck finding now with confirming Ceph-side evidence.
The node0 Ceph bottleneck finding gets richer: instead of "node0 has 1 GbE and the others have 2.5 GbE," it's "node0 has 1 GbE and the others have 2.5 GbE and Ceph backend traffic is currently averaging X MB/s with peaks Y, capped near node0's link saturation."

Effort¶

5–7 weeks. Each adapter is roughly a week of careful work plus integration tests. Cross-source view builder is its own ~1 week of design. Stray-config detection requires temporal data so leans on Phase 2.

Stop-here value¶

With Phase 3 you have a unified view across all the systems that compose your lab. Findings get sharper, the audit gets more complete, and you have the data needed to drive the conversational and planning layers. Still no chat, still no recommendations, but the underlying data layer is now genuinely cross-cutting.

Phase 4 — Conversational Layer¶

The agent layer. Adds the LLM router, narration, conversational discovery, and the MCP server. The framework gains a voice; users can chat about their lab and get sensible answers.

Goals¶

Build the LLM router with Ollama default and BYOK cloud backends.
Wire the agent layer's read-only access to engine query methods.
Build the Narrator agent: findings → prose.
Build the Conversational Discovery Agent: interview-style host onboarding.
Build the Skill Inferer: passive skill profile updates from chat.
Ship the MCP server exposing engine and adapter capabilities.
(Maybe) ship a minimal Web UI for chat + finding browsing.

Deliverables¶

Deliverable	Notes
LLMRouter	Task-class → privacy → capability tier → backend selection. Refuses cleanly when policy can't be satisfied.
Ollama backend	First-run auto-detect; install prompt if missing.
Anthropic / OpenAI / OpenAI-compatible backends	BYOK; not active unless user configures keys.
Narrator agent	Converts findings into prose. Small/Mid tier.
Conversational Discovery Agent	Interviews user to fill inventory gaps. Bidirectional: agent asks, user answers, agent updates harness DB.
Skill Inferer	Passive signal; updates per-domain trust hints.
MCP server	Exposes engine capabilities as MCP tools. Compatible with Claude Desktop / Claude Code / Cursor.
`helper chat` CLI verb	Basic CLI chat for users without a UI.
Minimal Web UI (optional)	If shipped: SvelteKit, chat + findings browser. Can defer to Phase 4.5 if scope is tight.

Acceptance criteria¶

helper chat with default config invokes Ollama and answers a basic question ("what hosts do I have?") correctly.
"What's wrong with my Ceph cluster?" produces a narrated answer citing the CEPH_BOTTLENECK finding, explaining the 1 GbE/2.5 GbE asymmetry, and listing the four candidate mitigations from the day-one report.
Conversational Discovery Agent successfully populates a new host via chat: user says "I just added a new mini-PC," agent walks through identity + IP + role + credentials, populates harness DB, kicks off warm discovery.
MCP server: Claude Code can drive a discovery run by saying "discover the host at 10.0.6.27" — calls the right tools, results land in NetBox.
Strict-local privacy mode: configure privacy=strict-local, ask a question that needs Frontier capability — router refuses cleanly with a clear message.
Skill profile demonstrably updates: after chatting about ZFS and Ceph and Kubernetes, the skill profile reflects "storage: intermediate; container-orchestration: advanced" without the user setting these manually.

Effort¶

5–7 weeks. LLM router is small but tricky to get right. Each agent is 1–2 weeks. MCP server is 1 week if you've built one before, 2 if not. Web UI deferrable.

Stop-here value¶

With Phase 4 you have a conversational lab assistant that knows your specific infrastructure and surfaces real findings with context. This is the "demo to a friend" moment — the framework is genuinely impressive at this point even though it still hasn't recommended anything or fixed anything. Power users with Claude Code or Cursor get a fully MCP-driven workflow.

Phase 5 — Planning & Recommendations¶

The framework starts producing recommendations, not just observations. The planner takes inventory + workload catalog + user goals and emits multi-option plans with tradeoffs.

Goals¶

Build the WorkloadProfile schema and seed a starter library.
Build the NetworkPath abstraction (typed link graph the planner queries).
Build the solver (OR-Tools or similar) for placement reasoning.
Build the Planner agent: solver output + LLM narration.
Ship hardware reconfiguration recommendations: "if you move this DIMM..."
Ship the bottleneck analyzer for known patterns (Ceph, network, storage, scheduler).

Deliverables¶

Deliverable	Notes
WorkloadProfile schema	Fields: baseline resources, scaling shape, arch support, dependencies, data-gravity, common deployment artifacts.
Seed workload library	~50 common homelab services: Plex, Jellyfin, Home Assistant, Nextcloud, Immich, Pi-hole, Frigate, MinIO, Grafana, etc. Community contribution path.
NetworkPath abstraction	Typed link graph: cables, tunnels, wireless. Planner queries paths between hosts with characteristic inheritance.
Solver (OR-Tools-based)	Constraint problem: workloads × hosts × constraints (CPU, RAM, GPU, arch, storage, network, intent). Output: candidate plans.
Planner agent	Calls solver; LLM narrates.
Reconfiguration reasoner	"If you swap these DIMMs you can do X" — uses Placement history + current capability gaps.
Bottleneck analyzer	Pattern library: Ceph link asymmetry, K8s scheduler pressure, NFS saturation, etc.
New CLI: `helper plan add-workload <name>`, `helper plan rebalance`, `helper bottlenecks`	The recommender surface.

Acceptance criteria¶

WorkloadProfile library has ≥50 entries covering top homelab services.
"If I add Immich, where should it run?" returns a placement recommendation with reasoning — including arch constraints (amd64 only), RAM headroom, ML-tagging GPU optionality, storage proximity to media.
helper plan rebalance against the example lab returns at least three candidate plans with tradeoffs narrated by the planner agent, e.g. "rebalance Proxmox storage with current hardware," "rebalance with one DIMM move," "rebalance with one part purchase."
The Ceph bottleneck recommendation produces the four candidate mitigations from the day-one report (CRUSH reweight, USB 2.5GbE add, OSD relocate, accept) — generated by the framework, not hardcoded.
Reconfiguration reasoner flags the node2 surplus: 24 GB of RAM, two stopped VMs, an Example CPU — would propose either spinning the VMs back up, moving the DIMMs to a more-loaded host, or accepting capacity reserve as the intent.
NetworkPath correctly characterizes the Wyola↔Covington case: planner refuses to place a Ceph-replicated workload across the VPN, explains why (latency + reliability inheritance from worst link in path).

Effort¶

8–12 weeks. The workload library is the biggest sink — accurate profiles take measurement effort, and 50 of them isn't trivial. The solver is the next biggest — constraint modeling is finicky, cost function tuning is real work. Planner agent itself is small.

Stop-here value¶

With Phase 5 you have a homelab framework that recommends, not just reports. This is the read-only vision realized — recommendations a human applies by hand. Phase 6 is where the framework can also apply them, under operator control.

Phase 6 — L2 Execution & Trust Gradient¶

The framework gains the ability to apply the changes it has been proposing — but only through a deterministic authorization model the operator controls. L1 (propose, never apply) is the floor; L2 raises it deliberately, per class of action, opt-in. Nothing about the read-only product changes for users who never enable execution. This is the phase the rest of the roadmap was built to make safe: the proposal stream, blast-radius tagging, and L2-ready manifest schema have existed since Phase 1 precisely so this phase is a wire-up, not a re-architecture.

Goals¶

Build the Executor: consumes approved action manifests, dispatches to adapter write paths, captures rollback state, emits receipts.
Build the trust gradient: the deterministic decide() policy function and its backing tables.
Build snapshot/rollback orchestration: per-blast-radius state capture before any execution.
Build the override and elevation-window mechanics, with a live kill switch.
Wire the gradient as the single gate in front of every adapter write path.

Deliverables¶

Deliverable	Notes
Executor	Reads pending `ProposalLog` artifacts; dispatches to adapter `write()` paths; transactional where the target supports it; emits a receipt per action.
`decide()` policy engine	Pure function `(action, context) → BLOCK/PROPOSE/CONFIRM/AUTONOMOUS`. Deterministic; no LLM in the authorization path.
Trust gradient tables	`Domain`, `CellTrust`, `TrustBoundary`, `ElevationWindow`, `TrustHistory` — see `harness-schema-slice1.md`.
Auto-escalation engine	Per-cell clean-streak tracking; promotes reversible + low-blast cells after N clean approvals; instant demotion + probation on one bad outcome.
Snapshot/rollback orchestrator	Per-blast-radius capture (ZFS/LVM snapshot, VM snapshot, config backup) before execution; the verified-rollback gate that caps autonomy.
Override + elevation window	Owner-only, interactive, logged. Window scoping, hard expiry, no auto-renew, kill switch, best-effort snapshot under elevation.
Receipts + audit spine	Every executed action → receipt; `TrustHistory` append-only record of every grant, auto-promotion, demotion, override, and window open/revoke.
CLI: `helper trust grant/show`, `helper window open/revoke`, `helper exec`	The authorization surface.

Acceptance criteria¶

With every cell at its default (PROPOSE), nothing executes — Phases 1–5 behaviour is bit-for-bit unchanged. L1 is the gradient's floor, not a separate code path.
Grant CONFIRM on containers/restart/single-host: approving a proposed container restart actually executes it and records a receipt carrying the captured rollback state.
A reversible + low-blast cell auto-promotes to AUTONOMOUS after N clean approvals; a subsequent matching action runs unattended with a receipt; one bad outcome demotes the cell to PROPOSE and flags probation.
An action with no verified rollback never auto-executes — it degrades to CONFIRM — unless an elevation window is open over its cell.
An ElevationWindow over storage on node2 lets autonomous runs cross the reversibility floor for 60 minutes, captures best-effort snapshots, tags every receipt with the window id, then auto-expires; the kill switch halts it on demand.
The secrets domain and a TrustBoundary-marked host reject every override and window — only a policy-config edit changes them.

Effort¶

12–20 weeks. The Executor and decide() engine are modest; the bulk is the safety machinery — snapshot/rollback orchestration across heterogeneous targets, verified-rollback semantics, the kill switch, and audit-trail integrity. This is the "3+ months focused" effort the earlier roadmap flagged, now committed and specified.

Stop-here value¶

Phase 6 is the full original vision: a framework that plans, recommends, and — where you have explicitly extended trust — executes and maintains. Beyond it lies operate-and-maintain (incident triage, predictive failure) and the hosted tier; neither is required for L2 to be a complete product.

Phase 7 — Agentic Operations¶

Phase 6 built the execution machinery and left it parked: every cell at PROPOSE, one write surface (Proxmox guest power), and an MCP surface that can draft but never trigger. Phase 7 puts the machinery to work without moving the one line that makes it safe. An LLM still never authorizes. What changes is who may trigger: an agent may ask for a proposal to execute, and the outcome is decided by decide() plus, where policy says CONFIRM, a human tap — never by the agent. Everything an agent cannot do today (grant, elevate, override, roll back) it still cannot do.

Goals¶

Let an agent close the loop on actions the operator has already chosen to trust, with the operator one tap away for the rest.
Grow the write surface beyond guest power, in blast-radius order, every surface behind the same executor and gate.
Make the framework proactive: findings become remediation proposals on their own, and later on a schedule.

Deliverables¶

Deliverable	Notes
Approval channel	A pluggable `ApprovalChannel` consulted by the executor when `decide()` returns `CONFIRM`. First implementation: a Home Assistant actionable notification (Approve / Deny) whose tap is read back over HA's websocket event bus. The approval is recorded as a `TrustHistory` event with the channel and the responder; a timeout leaves the proposal pending for the CLI path.
`execute_proposal` MCP tool	Routes a pending proposal through the executor. `AUTONOMOUS` runs; `CONFIRM` waits on the approval channel; `PROPOSE` and `BLOCK` are refused with the policy reason and the `helper exec run` command. Two new mechanical tests pin that the MCP surface still has no grant / window / override / rollback tool.
Proxmox guest lifecycle	`migrate` (live or offline, to a named node) alongside the existing power + snapshot actions; a guest CPU-type change follows. Reversibility: migrate back, verified by both nodes being online.
Kubernetes workload actions	`workload-restart` (rollout restart of a Deployment / StatefulSet / DaemonSet, undone with `rollout undo`), `workload-scale` (replicas, with the previous count captured). A new `single-service` blast radius names one workload. The K8s adapter gains its first write methods, callable only from the executor.
Remediation playbooks	Deterministic mapping from finding kinds to manifest templates (e.g. `SERVICE_DOWN` on a K8s workload → `workload_restart`); the Triage agent drafts the manifest, policy decides.
Scheduler (the deferred Phase 2 loop)	Discovery + reconcile + assertions on a cadence, so findings and their proposals appear without a human asking.
CLI: `helper approvals`, `helper exec listen`	See pending approvals; run the approval listener as a long-lived process for operators who do not use the MCP path.

Acceptance criteria¶

With every cell still at PROPOSE, execute_proposal executes nothing — it returns the policy reason and the CLI command. Phase 6 AC1 holds through the new tool.
Grant CONFIRM on hypervisor/migrate/single-host: an agent-drafted migrate proposal produces a phone notification; Approve executes it, records a receipt and a TrustHistory approval event naming the channel; Deny or a timeout executes nothing and leaves the proposal pending.
Grant CONFIRM on containers/workload-restart/single-service: the same flow restarts a Deployment, with the current rollout revision captured so rollout undo can return to it.
A cell at AUTONOMOUS runs from execute_proposal unattended, with a receipt and a notification, and one bad outcome demotes it exactly as Phase 6 AC3 specifies.
The MCP surface still has no tool that grants, elevates, overrides, rolls back, or opens a window — enforced mechanically.
A SERVICE_DOWN finding on a trusted cell produces a remediation proposal without a human prompt, and the scheduler surfaces it within one cadence.

Effort¶

6–10 weeks. The approval channel and MCP tool are small; the surfaces are each a contained adapter + manifest + executor + rollback slice; the scheduler is the deferred Phase 2 work and the largest single item.

Stop-here value¶

A lab that fixes what it already knows how to fix, with the operator deciding how much of that happens unattended, per action class, and able to stop all of it with one switch.

Phase 8 — Operate & Optimize¶

Phases 1–7 answer "what is there, what is wrong with it, and may I fix it". Phase 8 asks the operator's next question: what should change, and why — currency, resilience, efficiency, and gaps. Everything here is L1 first: findings and proposals with deterministic fingerprints and the reconciler's lifecycle. Anything that later executes goes through decide() like every other write, and the highest-blast items (node updates) start at PROPOSE.

The one structural gap is history. The harness holds the latest observation per fact, so it can reason about what is allocated but not what is used. Slice 8.3 adds bounded usage history; slices that need it come after it.

Slices, in delivery order¶

#	Slice	Finds	Sources	Needs history
8.1	Version currency	Hypervisor nodes behind on packages or on mixed versions within one cluster; operating systems past end of life; Kubernetes node version skew; Home Assistant core / add-on / device firmware updates pending	Proxmox `version` + `apt/update`, host `os_` capabilities, K8s node info, HA `update.` entities	no
8.2	Backup posture	Guests with no enabled backup job; last successful backup older than its schedule allows; verify / prune gaps; offsite copy freshness; important state that lives outside Proxmox backups	Proxmox `cluster/backup` + task history, PBS, operator-declared extras	no
8.3	Usage history	(enabler) Bounded per-host and per-guest CPU / memory / disk / network rollups; backfilled from Proxmox RRD on first run	Proxmox `rrddata`, K8s metrics, OMV	—
8.4	Rightsizing & real-usage placement	Guests allocated well above p95 use; K8s requests vs usage; rebalance driven by p95 instead of commitment; idle guests as retire candidates	8.3 + existing planners	yes
8.5	Storage efficiency	Pool headroom trends and time-to-full; stale snapshots; detached disks; released PVCs; template / ISO clutter; backup retention cost	Proxmox, PBS, K8s, OMV	partly
8.6	Weekly digest	One summary per week (phone notification + a readable page) instead of a stream: what changed, what is recommended, what was done	all of the above	no
8.7	Service suggestions	Unused capability (GPU / accelerator present, no workload using it), missing building blocks (metrics, alerting, a local LLM the router expects), matched deterministically against the workload library	inventory + library	no

Update orchestration (rolling node update: drain → update → reboot → verify → next) is the first Phase 8 write path. It lands only after 8.1 has run long enough to trust its findings, as a new action kind behind the trust gate, at PROPOSE.

Acceptance criteria¶

Mixed versions are named. A cluster whose nodes report different hypervisor versions, or a node with pending package updates, produces a version-drift finding naming the nodes, the versions, and the pending count; updating the node resolves it on the next pass.
End of life is a date, not an opinion. A host whose OS release is past its published end-of-life date produces a finding citing that date; the table of dates is data in the repo, not model output.
Every guest's backup is accounted for. A guest with no enabled backup job, or whose last successful backup is older than twice its job interval, produces a backup-gap finding.
History survives restarts and stays bounded. Usage rollups are backfilled on first run, retained to a configured horizon, and never grow without bound.
Rightsizing cites numbers. A rightsizing proposal names the allocation, the observed p95, the window, and the proposed value; a guest with too little history gets no proposal.
One digest a week. The digest is generated deterministically from findings and receipts; an LLM may narrate it but does not choose its contents.
Absence never resolves. Every new finding kind follows invariant 1: a source that was not observed this run cannot resolve its findings.

Effort¶

6–10 weeks. 8.1, 8.2 and 8.6 are small adapter-read + reconcile slices; 8.3 is the largest (storage, retention, backfill); 8.4 and 8.5 are planner work on top of it; 8.7 is mostly library curation.

Stop-here value¶

A lab that tells its operator, once a week and with evidence, what is out of date, what is unprotected, what is wasting capacity, and what it could be doing — and fixes the parts the operator has already chosen to trust.

Phase 9 — Road to 1.0¶

Phases 1 to 8 were built against mocks and verified against one lab. That is why the framework works and also why nobody else can run it yet: the install story is a README, half the acceptance criteria have never met real hardware, the planners quietly assume every host is a Proxmox node, and nothing says which surfaces a plugin author may depend on. Phase 9 adds no capability. It makes the existing capability real for someone other than its author — and tells that author which of its promises are actually tested.

The on-ramp already exists and is undocumented: helper discover replay seeds a synthetic lab that ships in the wheel and produces 13 findings with no hardware, no credentials and no SSH. A stranger should meet that in the first five minutes, not discover it in the backlog.

Goals¶

A stranger installs it, points it at their own lab, and gets a finding table — from written instructions, in an hour.
Every Phase 4 to 8 acceptance criterion is either validated against real hardware or honestly marked unvalidatable-here, with the reason.
The assumptions that leaked in from one lab are named, then fixed or written down.
"1.0" means something: a stated promise about the surfaces other people build on.

Slices, in delivery order¶

#	Slice	Delivers
9.1	Close the validation gate	The `⏳` rows in `live-validation.md`: onboarding (interactive), Phase 6 steps 5 and 6 live, and Part 4's rolling update on one node. The two `⚪ n/a` rows (Ceph narration and mitigations) get a synthetic asymmetric replay fixture so the derivation runs end-to-end through the real CLI, or a stated reason they stay unit-tested only.
9.2	De-assume the lab	Hosts gain a role (or it derives from cluster membership plus capability) so `plan surplus` stops calling a Docker-running NAS surplus — the mirror of the `plan rebalance` defect fixed in #51. The runbook's 1 GbE / 2.5 GbE text matches a symmetric fleet. Any other place the fleet's shape became logic.
9.3	Docs site	mkdocs-material: getting-started led by `discover replay`, CLI reference, probe SDK guide, adapter authoring guide, NetBox custom-field reference, and an operator's explainer for the trust gradient — what `PROPOSE` / `CONFIRM` / `AUTONOMOUS` mean and how to raise a floor without regretting it.
9.4	Declare the 1.0 surface	`docs/stability.md`: what is stable and what is not — the probe entry-point contract, the `ProposalLog.artifact` manifest schema, the DB through migrations, the CLI verbs, the MCP tool names — plus a deprecation policy. Python matrix pinned in CI (3.12 now, 3.13 when upstream deps allow).
9.5	The last Phase-1 probes	`host.raid` / `host.shares` (mdraid composition, exported shares), dmidecode DIMM depth so a serialled module becomes a `PhysicalPart` + `Placement`, and `talos.host` CPU/DIMM depth. These are what a new user with an mdraid box or a Talos cluster hits on day one.
9.6	Finish the Phase-2 tail	Probe-level schedules and assertion cadences inside the daemon, so "continuous" means per-probe cadence rather than one discovery interval for everything.
9.7	Status endpoint, then (maybe) a Web UI	9.7a: a read-only `GET /status` rollup (findings, pending approvals, discovery age, trust cells) and `helper status`, so the harness lands on the operator's existing dashboard (Homepage, Home Assistant) instead of growing its own. 9.7b, optional and deliberately last: chat, a findings browser, and the trust surface read-only. The CLI and MCP already cover this, and a UI is the easiest thing to half-build.

Non-goals. No new adapters, no new action kinds, no predictive work, no hosted service. Phase 9 earns its keep by finishing things, not starting them.

Acceptance criteria¶

The sign-off table has no ⏳ rows. Every Phase 4 to 8 criterion is validated live, or marked ⚪ with a reason naming what about this lab makes it unobservable.
A stranger gets to a finding table from the docs site alone — install, helper db init, helper discover replay, helper findings list — without reading the repo or asking its author.
plan surplus against a lab whose NAS runs Docker does not call the NAS surplus, and says what makes a host a placement target.
The asymmetric replay fixture reproduces the Ceph bottleneck finding and its four mitigations through the CLI, so P4-AC2 and P5-AC4 are demonstrable without owning a mismatched fleet.
A host with an mdraid array reports its composition, and a DIMM reporting a serial becomes a PhysicalPart with a Placement.
Two probes on different cadences both run from one helper daemon, and an assertion runs on its own schedule.
docs/stability.md names every public surface and its promise, and CI runs the declared Python matrix.

Effort¶

8–14 weeks, plus 3–4 if the Web UI is built. 9.1 is mostly operator time on real hardware rather than coding; 9.3 is the largest coding-adjacent item because documentation is slow; 9.5 is three probes, each with its own parse tests.

Stop-here value¶

A framework a stranger can install, understand and trust — and whose author knows exactly which of its promises have met real hardware and which have only ever met a mock.

Phase 10 — Predictive operations¶

Phase 8 answered "what is wrong now"; Phase 10 answers "what is about to be". The substrate is already in place — a year of daily usage rollups from 8.3, and per-drive SMART counters the host.smart probe has recorded since Phase 1 that nothing consumed. With roughly 26 subjects this is thresholds, trends and per-subject baselines, named as such; it is not fleet ML, and nothing here needs a model.

Goals¶

A drive that is failing, wearing out or quietly reallocating sectors is a finding before it is an outage, judged on its own history rather than on a single reading.
Capacity and load trends say when something runs out, not only that it is high today.
Every rule is a stated threshold or trend in code, with its evidence on the finding, so an operator can disagree with a number rather than with a model.

Slices, in delivery order¶

#	Slice	Delivers
10.1	Disk health	`engine/disk_health.py`: `host.smart` → `storage-health` findings per drive — SMART FAILED, reallocated/pending sectors (HIGH when growing), CRC errors that grow inside the window, temperature over the line for that kind of drive (judged on the window's peak), NVMe wear past a threshold or on a rate that reaches 100% within six months, and spinning drives past 50 000 hours. History is the host's own earlier observations; resolution is per host, like RAID.
10.2	Capacity trends	Time-to-full for every pool and filesystem the usage history covers (8.5 does Proxmox storage; extend to hosts' root filesystems and the NAS), memory and CPU trend per node with a "crosses the constrained line in N days" finding, and the digest naming what runs out first.
10.3	Baselines and anomalies	Per-subject baselines from the usage history (weekday/hour shape) so a guest that departs from its own normal — a backup that silently stopped, a service that started spinning — raises a finding with the baseline as evidence. Deterministic (percentile bands), no model.
10.4	Maintenance runs	A `maintenance_run` that sequences drain → `node-update` → `node-reboot` → return guests as one operator-approved procedure with one receipt per step; `node-reboot` as the second no-inverse action kind. The rolling procedure the operator scripts by hand today, inside the gradient.

Non-goals. No machine learning, no hosted service, no new adapters. A threshold the operator can read beats a score they cannot.

Acceptance criteria¶

A drive with growing reallocated sectors is a HIGH finding, a drive with a static count from years ago is at most MEDIUM, and a CRC count that has not moved raises nothing — demonstrated on the bundled fixtures and on the live fleet's NAS.
A replaced drive closes the old finding and starts clean: identity is the serial, resolution is per host.
Every pool and filesystem with history carries a time-to-full, and the digest leads with the earliest.
A guest departing from its own baseline is a finding whose evidence shows the band it left.
One maintenance run on one node produces one receipt per step, stops on the first failure, and never reaches AUTONOMOUS.

Effort¶

6–10 weeks. 10.1 is a week — the probe and the finding machinery exist. 10.2 and 10.3 are mostly deciding what a trend means for this fleet; 10.4 is the largest because sequencing a no-inverse action is new ground for the executor.

Stop-here value¶

After 10.1 alone: no disk in the lab dies unannounced.

Phase 11 — Talos as a managed plane¶

Through Phase 10 the Kubernetes nodes were inventory: talos.host probed them, talos-eol / k8s-eol findings said when they were behind, and the operator upgraded them with a shell script. Phase 11 brings them inside the gradient — the same cells, receipts, rollback findings and maintenance-run sequencing the Proxmox nodes already have — without a new adapter: talosctl is already the adapter.

Goals¶

A Talos upgrade is an action kind with a verified inverse: Talos keeps the previous install in its other boot partition and talosctl rollback boots it, so this is the first node-level action that can earn autonomy.
A Kubernetes upgrade is an action kind with no inverse, pinned at CONFIRM like node-update.
A whole-cluster upgrade is one maintenance run — control planes first, then workers, each step health-gated — drafted by a playbook from the EOL findings.
Workload usage (metrics-server) joins the usage history so 10.3's baselines and 8.4's rightsizing see pods, not only guests.

Slices, in delivery order¶

#	Slice	Delivers
11.1	`talos-upgrade`	`TalosTarget {node, version}` manifest variant — a release tag, never an image; the executor derives the image from the node's own `machine.install.image`. `TalosAdapter.upgrade_node` / `rollback_node` writes (executor-only), `prior-image` rollback strategy whose verifier reads the node's current release and readiness, a dispatch that waits for the node to leave and come back ready on the new tag (fails the receipt after 15 minutes), `propose_talos_upgrade` over MCP. In `REVERSIBLE_ACTION_KINDS`.
11.2	`k8s-upgrade`	`talosctl upgrade-k8s` against the cluster endpoint as a `cluster`-radius action with the `no-inverse` strategy: CONFIRM for good. Preflight: every node ready, control planes on one Talos release, the target a supported skew.
11.3	Talos maintenance run	`MaintenanceRun` grows a Talos plane: `talos-upgrade` per control plane, then per worker, each gated on the previous node's return and on cluster health (`talosctl health`), then `k8s-upgrade`. Drafted with `helper maintenance plan-talos` / `plan_talos_maintenance_run` from the version a `talos-eol` / `k8s-eol` finding names.
11.4	Workload usage	metrics-server pod CPU/memory into `usage_sample` as `workload` subjects, so baselines, rightsizing and the digest cover the cluster's workloads.

Non-goals. No Talos machine-config edits through the gradient (a config patch is remote code execution with a YAML accent), no node adds/removes, no Kubernetes API writes beyond the Phase-7 workload kinds.

Acceptance criteria¶

talos-upgrade of one node produces one receipt whose rollback state names the prior release, and helper exec rollback on it boots that release again — demonstrated against the lab.
A manifest cannot name an installer image; a version that is not a release tag is refused before the gate.
A node that never comes back is a failed receipt, never a success.
A cluster upgrade is one maintenance run that stops at the first node that does not return healthy and leaves the rest un-upgraded.
A pod with a week of usage history has a baseline like a guest does.

Effort¶

4–6 weeks. 11.1 reuses the node-reboot dispatch shape; 11.3 reuses the maintenance-run sequencer; 11.2 is small; 11.4 is a probe and a subject type.

Stop-here value¶

After 11.1 alone: the lab's Talos upgrades go through the same gate, receipts and rollback as everything else, with a one-command undo.

Post-roadmap (Phase 12+)¶

These are real future phases, deliberately not committed in this roadmap:

Multi-tenant + hosted service. The Nabu-Casa-style subscription tier — managed LLM access, off-site backup, remote access, mobile push, community template marketplace.
Heterogeneous architecture support beyond Linux. FreeBSD probes (TrueNAS Core, pfSense/OPNsense), macOS probes (Mac mini servers), Windows probes (Windows hosts in mixed labs).

Each of those is its own roadmap-scale effort. Not promising any of them; just naming them so they don't accidentally creep into the committed phases (1–11). Predictive operations became Phase 10; Talos as a managed plane became Phase 11.

Cross-cutting tracks¶

Things that happen alongside every phase, never owned by a single one.

Probe catalog growth¶

Probes are plugins from day one (Phase 1 decision). The catalog grows continuously:

Phase 1: ~10 core probes covering Linux hosts.
Phase 2: agent-side probes for continuous mode.
Phase 3: API-based probes per adapter (Proxmox, K8s, UniFi, etc.).
Phase 4+: workload-specific probes (Ceph internals, K8s detailed health, etc.).

A community contribution path matters by Phase 3 — by then you have non-toy users who want to add their own.

Documentation¶

Phase 1: getting-started, CLI reference, probe SDK guide, NetBox custom-field reference.
Phase 2: agent deployment, assertion authoring.
Phase 3: adapter authoring guide (so users add adapters for tools you don't support).
Phase 4: chat interaction patterns, MCP integration recipes.
Phase 5: workload profile contribution, solver tuning.

A docs site (mkdocs-material or similar) starts in Phase 1 and grows with each phase. Don't defer it.

Test fixtures¶

Phase 1: example lab as integration test fixture. Replayable, CI-friendly.
Phase 2: synthetic continuous-mode scenarios (drift injection, intent transitions).
Phase 3: mock external APIs (Proxmox, K8s) for adapter testing without a real homelab.
Phase 5: workload-placement scenarios with known-correct answers.

Replay-based testing for the reconciler is non-negotiable — without it, regressions in reconciliation logic are devastating.

Risks worth tracking¶

Reconciler bugs are catastrophic. Bad reconciliation logic produces wrong findings or, worse, corrupts NetBox. Mitigation: heavy unit testing, replay tests, dry-run mode for all NetBox writes during development, "trust but verify" comparison runs early on.

Probe brittleness. Linux distros vary; kernel versions vary; tools have different output formats across releases. Mitigation: defensive parsing, version-aware probes, graceful degradation, observation confidence field used aggressively.

LLM hallucination at the wrong level. Even with the reconciler/narrator split, the LLM could narrate a finding incorrectly or invent a recommendation. Mitigation: every LLM output is citation-bound to specific findings/observations; UI surfaces "this came from finding X" explicitly; no LLM output is presented as fact without a source.

Scope creep into L2. The pull to "while we're at it, let's just fix this" is constant — and now that L2 is a committed phase, the temptation to wire execution early is stronger. Mitigation: L2 is the last phase and every write path must route through the trust gradient. Phases 1–5 stay propose-only; every PR reviewer checks "does this enable a write path that isn't gated by decide(), or that ships before Phase 6?" The gradient existing on paper from Phase 1 (cells pinned to PROPOSE) is what keeps "L2-ready" from quietly becoming "L2-enabled."

Workload library accuracy decay. Service profiles drift as software changes. Mitigation: schema includes version/last-verified per profile; community contribution lowers the maintenance burden on the project owner.

NetBox upstream changes. NetBox occasionally makes breaking schema changes. Mitigation: pin a tested NetBox version range; track NetBox releases; integration tests against new NetBox releases before bumping the supported range.

Burnout. Hobby pace + ambitious roadmap = real risk. Mitigation: stop-here markers at every phase mean every phase ships something complete. Easier to step away from a complete Phase 2 than a half-built Phase 3.

Total roadmap effort¶

Adding the phase ranges: 28–42 weeks for Phases 1–5, plus 12–20 weeks for Phase 6, 6–10 for Phase 7, 6–10 for Phase 8 and 8–14 for Phase 9 — 60–96 weeks end to end, at the optimistic "focused hobby pace" estimate. Realistic (2× factor) is 120–192 weeks — three to four years of evening/weekend work. That's a lot. The stop-here markers exist precisely because every point along that road produces a working system; you decide phase-by-phase whether to continue. Phases 1–5 are a complete read-only product on their own — Phase 6 is the deliberate, gated decision to let the framework act.