Framework and Northstar audit - 2026-09-23

Conclusion: conceptually well aligned, operationally blocked. The framework follows the vocabulary and responsibility boundaries of Developing-in-Agentic-AI-Systems-Learning-Paths.md. The reference has substantial implementation and application evidence, but green existing tests do not establish evidence integrity, cross-host operation, or hosted acceptance. The guide was not changed to excuse defects.

Immutable scope and evidence

Item	Result
Framework starting revision	9c8be3c (documentation-site baseline); audit changes are additional
Guide canonical-text SHA-256	c247b45ed53bb7b901954611c3bc03a37294d9adfb02a6542d71522f694f62be
Northstar revision/branch	b65c2de5c8224342c72c37eeed7ef9f965ad8a2c, main; known defective, not an accepted release
Scope inspected	Framework documents and executable scripts; all 35 Northstar runtime .mjs scripts, relevant policies/workflows/hooks/agents/application/tests; declaration files covered by typecheck, not all manually reviewed
npm run validate	Passed: instruction sync, offline governance 77 checks, lint, typecheck, build, 220 unit tests
npm run validate:all	Exit 0, not clean validation: 220 unit + 8 PostgreSQL acceptance tests passed; Poutine printed 5 errors despite overall zero exit
Dependency audit	2 moderate Vitest/@vitest/mocker findings, GHSA-82fw-gwwq-j7x9; configured high threshold did not fail
Other scanners	Supplemental secret scan passed, 146 files; gh-aw v0.81.6 compiled with 0 errors/warnings; zizmor 1.30.0 reported 0 findings and 81 suppressed, offline
npm run demo:system	Emitted ready_for_review, 6/6 criteria; fixture-based local output, not approval or an independent proof of the flawed evidence validator
PostgreSQL boundary	Existing 8 acceptance tests use two services/Fastify inject in one Node process against real PostgreSQL
Additional process proof	Separate local server PIDs 52104 and 51688, HTTP requests sharing PostgreSQL: 12 concurrent same-key calls, one 201 and eleven 200 replays, one order ID; changed payload 409; no-key calls distinct; replay survived replacement PID 57612
Snapshot	117 canonical-text files match exact b65c2de revision; inspection only

The additional process proof was an audit harness outside Northstar, not a new committed acceptance test, hosted evidence, or implementation change. An initial six-second cold-start deadline failed; the thirty-second deadline completed with readiness checks and explicit child-process cleanup.

Alignment and defects

Guide locations refer to the unchanged snapshot. Defects are not legalized by listing them as extensions. Detailed conformance is in GUIDE-CONFORMANCE.md.

Finding	Guide basis / extension	Evidence and consequence
Live task bootstrap and freshness	Task contracts and durable source of truth, 553-624, 3088-3439; EXT-001, EXT-018	Real cloud plan lacked cache; second local plan saw another task’s cache. Both agents correctly refused. Resolver failure paths can retain stale authority.
Host isolation and edit paths	Least privilege/hooks, 1145-1183, 1748-1836; EXT-003, EXT-004	Observed cloud branch is rejected by local-only naming policy; authorized absolute in-repo path is rejected. Complete host payload support is unproved.
Risk and workflow-choice mismatch	Risk-based autonomy / plan-first versus plan + execution, 635-867, 3835-3967; EXT-002, EXT-003	docs/example.md classified medium because catch-all ** rule wins by maximum. Authorizer still requires approved plan/branch for all edits. Lower-risk claims overstate implementation.
Missing artifact can pass readiness	System signals, outputs/artifacts, 1016-1094, 2708-2816; EXT-006	Synthetic passing envelopes referencing nonexistent files produced ready_for_review through null === null digests. Probes were in-memory test inputs, not authority.
Stop and recovery	Bounded retries/escalation, 2834-2881, 4432-4447; EXT-006, EXT-010	Stop ignores evidence-command failure and can read old report; JSON load is treated as plan validity. Repair-budget helper exists but is not integrated into the Stop loop.
Audit accuracy and diagnostics	Post-action/error observability, 1145-1183; EXT-005	Failure event without explicit error fields logged success; malformed JSON diagnostic echoed a synthetic private marker.
Scanner result integrity	Security/system evidence, 1127-1145, 4038-4048; EXT-011, EXT-015	SARIF {} passed as zero findings; unreadable secret-scan inputs skipped; Poutine errors failed to propagate; zizmor uses latest and covers only generated workflow.
Hosted governance unavailable	Rulesets or branch protection, continuous governance, 3998-4307, 4412-4457; EXT-009	Real governance artifact says unavailable because GH_TOKEN missing; online evaluator requires both alternative mechanisms; optional feature files are mandatory in offline audit.
Incomplete identity/pagination handling	Durable attributable evidence, 2708-2816; EXT-006, EXT-007	Evidence-comment discovery is first-page/marker-based without author validation; review/PR lookups also need pagination review. No observed impersonation is claimed.

Other evidence-quality concerns: base SHA/run attempt are recorded but not fully compared during fan-in; HEAD alone does not attest a dirty working tree; JUnit regex parsing and test-name-to-criterion mapping require stricter negative cases. Acceptance teardown obscures initial database setup failure when the harness was never created. These are repair requirements, not reported fixes.

Actual hosted observations

The earlier terminology, workflow-syntax, and artifact-handoff repairs merged: webmaxru/northstar-orders-api-demo#7, webmaxru/northstar-orders-api-demo#10, webmaxru/northstar-orders-api-demo#12. The old assertion that GitHub cannot parse the workflow is no longer current.

Hosted evidence	Observation
Governed Change run 33997954356, head a9f6e86ff64a77389d562eaed6a706dd1c638009	Jobs started; quality, acceptance, CodeQL, dependency, scope and other jobs passed; repository-controls, human-review, plan-approval and evidence failed
Governance Review run 35610027135, b65c2de	Downloaded artifact governance-review ID 10643307594: online.available=false, ready=false, zero online checks; missing GH_TOKEN
Daily Repository Status run 35799033646	Agent job failed; configured model invalid/unavailable for engine/account
Cloud WI-1842 rehearsal	Completed session, blocked task: no live contract cache, no plan/implementation/acceptance

Protected environments, App identity/permissions, rules, secret scanning, push protection, and trusted acceptance remain unverified. A successful workflow job or merged PR is not proof of those settings.

Script-quality inventory

All names below are Northstar scripts/*.mjs. Each runtime file was read; grouping does not mean each behavior was exercised in every host.

Scripts	Assessment / needed work
task-contract, fetch-task-contract, resolve-task, session-start, resolve-pr-task, resolve-workflow-pr	Explicit contracts/digests and fixture separation are sound; clear stale caches, test actual host bootstrap and paginated PR/task resolution
plan-contract, plan-approval, fetch-approved-plan, check-plan, plan-stop, publish-plan	Useful schema/digest/approval separation; preserve read-only planner; prove transcript/Stop compatibility and paginated latest-head approval handling
risk-policy, authorize-tool, check-scope	Good layered intent; correct catch-all precedence, path/payload normalization, cloud isolation and risk-specific execution semantics
audit-hook, agent-stop, repair-budget	Minimized records and bounded helper are useful; fix failure classification, payload echo, stale report reuse and actual retry integration
evidence-record, build-execution-report, import-evidence-artifacts, import-workflow-results	Strong provenance schema; reject missing/null artifacts, mismatched base/attempt and dirty-source claims; strengthen parser cases
check-human-review, check-merge, check-validation-authority, maintenance-manifest, verify-maintenance-runs	Preserve independent latest-head review and trusted maintenance boundary; full hosted proof remains missing
publish-evidence, publish-acceptance-status	Keep trusted publication; paginate and verify comment author as well as marker
governance-audit, check-sarif, secret-scan, run-zizmor	Fix live auth/alternative controls, SARIF validation, unreadable inputs, scanner pinning/coverage and failure propagation
sync-agent-instructions, system-demo	Single-source instructions and explicit fixture mode are useful; demo is not live authority and does not itself run the separately documented hostile-hook call

Framework fixes in this change: tools/verify-architecture.ps1 recognizes the existing ordinary docs.yml publisher without allowing other workflows, ignores generated/dependency Markdown using Git enumeration, and retains blocked-release enforcement. Seven regression tests cover those boundaries. docs-site/scripts/sync-docs.mjs now preserves nested document routes, resolves links by path, rejects collisions/overlapping roots before deletion, and reports watcher errors without poisoning later sync attempts. Four regression tests cover generation; an Astro production build also ran. tools/verify-reference.ps1 retains exact source verification.

Customization seams

Only target-specific settings are customizable. Schema semantics, trust checks, evidence requirements and approval boundaries are not convenience switches.

Surface	Mark/customize	Do not silently change
Framework verifiers	Adjacent CUSTOMIZE comments identify checkout location	Locked identity/hash or blocked conformance
Documentation generator/config	Adjacent CUSTOMIZE comments identify authored/generated roots, Pages origin/base, branding/repository link	Source-authority boundary or collision checks
Northstar governance policy	Paths, commands, owners, checks, retention, risk patterns, App/environment IDs	Fail-closed behavior, approval identity, guide alternatives
Hooks/agents/prompts	Runtime invocation, host payload adaptation, context paths and selected tool names	Role capability boundaries and task/plan binding
Workflows/scanners	Supported pinned versions, model/budget, runner, triggers, database port, artifact retention	Least privilege, safe-output limits, scanner failure or independent acceptance
Application demo	Fictional SKU/key, local port/database endpoint, selected acceptance criteria	Durable idempotency invariant or privacy claims

Northstar adjacent comments remain pending the approved repair. JSON cannot contain JavaScript comments: use schema-supported $comment only where accepted, otherwise an adjacent documented field map. Do not hand-edit the snapshot to add comments absent from its exact reference source.

Repair and acceptance boundary

Created webmaxru/northstar-orders-api-demo#14, AES-SURFACE-EVIDENCE, with scope, constraints, eleven proving criteria and explicit approval stop. Runtime/control-plane repairs are high-risk validation-authority changes. The audit request and unattended continuation are not an independent approval of a digest-bound plan-only commit.

No Northstar implementation was changed. Required next evidence is a reviewed machine plan, focused negative tests for these defects, clean full validation, committed process-boundary coverage, host canaries, protected maintenance, and real trusted hosted acceptance. Until then both locks remain blocked. This audit is not an exhaustive security certification.