Benchmark your organisation
How current is your organization?
Five levels, Initial to Optimizing, with one twist: AI capability is woven into every one. A Level 4 from 2020 may be a Level 2 today.
Score your own organisation.
Eight disciplines, five levels. Score where all the statements are true.
RequirementsTurning human intent into buildable, testable, compliant specification.
DevelopmentHow code gets written, reviewed, and kept safe by construction.
Testing & QAHow quality is proven rather than hoped for.
DevOps & CI/CDHow change reaches production, and what can stop it.
Security & ComplianceHow obligations become enforced controls.
ObservabilityHow the running system tells you what is true.
Data & AnalyticsHow data is governed, minimised, and made useful.
AI IntegrationHow AI is governed as a team member, not a toy.
–Level
—
—
Nothing is stored or sent. Close the page and the score is gone.
Most organisations land a level lower than expected once the AI dimension is included. That gap is the roadmap.
1
Initial
Ad-hoc, heroic, person-dependent. Success is accidental; AI is a curiosity used informally by individuals.
2
Managed
Basic practices exist per team but are inconsistent. AI tools adopted without policy or measurement.
3
Defined
Organization-wide standards, documented and enforced. AI usage governed by policy with approved tooling.
4
Quantitatively Managed
Practices measured with honest metrics; AI assistance embedded in workflows with measured uplift and guardrails.
5
Optimizing
Continuous improvement institutionalised; AI a governed team member, with feedback loops that improve the system.
Requirements Engineering & Product Management
How an organization turns human intent into buildable, testable, compliant specification. Full deep-dive →
| Level | What it looks like | The AI dimension |
|---|---|---|
| 1 · Initial | Requirements live in hallway conversations and slide decks; "the code is the spec." Acceptance criteria are discovered by users in production. | Individuals occasionally paste ideas into a chatbot; outputs are unreviewed and untraceable. |
| 2 · Managed | Teams write user stories in a backlog tool, but formats vary; acceptance criteria are inconsistent; compliance is checked late, by someone else. | Some PMs use AI to draft stories; quality depends entirely on individual prompting skill; no shared standards. |
| 3 · Defined | Org-wide story and BDD acceptance-criteria standards; Definition of Ready/Done enforced; non-functional requirements are first-class backlog items; traceability epic → story → test exists. | AI-assisted refinement is an approved, governed practice with prompt libraries, review checklists, and human sign-off; compliance checklists are AI-suggested and human-verified. |
| 4 · Quant. Managed | Requirements quality is measured: rework rate from misunderstood requirements, acceptance-criteria coverage, compliance-gap escape rate; refinement cycle time is tracked and improved. | AI flags ambiguous language, missing acceptance criteria, and compliance gaps against a live regulatory knowledge base; uplift and false-positive rates are measured. |
| 5 · Optimizing | The requirements system learns: post-release outcomes feed back into refinement standards; compliance rules update from regulatory change feeds; the pipeline from natural-language use case to code-level constraints is AI-assisted end-to-end with human accountability at every gate. | AI is an embedded team member — drafting, challenging, trace-checking, and keeping the compliance matrix current — while humans own every decision. This is the North Pole's operating target, demonstrated in the deep-dive. |
Software Development Practices
How code is written, reviewed, and kept healthy.
| Level | What it looks like | The AI dimension |
|---|---|---|
| 1 · Initial | No shared standards; long-lived branches; merges are events; review is optional and adversarial. | Developers paste proprietary code into unapproved external tools; no policy, no guardrails. |
| 2 · Managed | Style guides exist but are enforced by humans in review; PRs are large and slow; branching strategy varies by team. | AI coding assistants are tolerated but ungoverned; generated code is not reviewed differently from human code — or is trusted blindly. |
| 3 · Defined | Trunk-based development; small PRs; automated lint/format/type checks; secure coding standards mapped to OWASP; CODEOWNERS review; docs-as-code. | An AI Use Policy governs assistants: approved tools, prohibited contexts (secrets, child-data schemas), and the rule that generated code is reviewed to the same standard as human code. |
| 4 · Quant. Managed | PR latency, review coverage, escaped-defect rates, and build times are measured and improved; golden-path templates make new services compliant by construction. | AI pair-programming uplift is measured (cycle time, defect density of AI-assisted vs. unassisted changes); AI review assistants flag security and privacy implications in PRs. |
| 5 · Optimizing | The development system improves itself: postmortem patterns become linter rules; golden paths absorb every lesson; AI agents handle routine maintenance (dependency upgrades, flaky-test quarantine) under human review. | AI is a governed contributor across the codebase — drafting, reviewing, refactoring, and documenting — with provenance tracking and human accountability for every merge. |
Quality Assurance & Testing
How confidence is built before — and after — release.
| Level | What it looks like | The AI dimension |
|---|---|---|
| 1 · Initial | Testing is manual, end-of-cycle, and performed by whoever is available; "QA" means a person clicking before release. | None — or ad-hoc copy-paste of test ideas from a chatbot with no traceability to requirements. |
| 2 · Managed | Some automated tests exist but coverage is patchy; a separate QA team gates releases; environments are shared and stale; production data is copied to test systems. | AI is occasionally used to write unit tests; generated tests are not reviewed for behavior value; coverage numbers rise without confidence rising. |
| 3 · Defined | A defined test pyramid; BDD acceptance criteria drive automation; contract tests prevent integration surprises; synthetic test data is mandatory; critical journeys have 100% E2E coverage. | AI generates candidate test scenarios from acceptance criteria — including edge and adversarial cases — reviewed and owned by engineers; compliance validation tests are derived from the regulatory matrix. |
| 4 · Quant. Managed | Escaped defects, flaky-test counts, mutation scores on high-risk modules, and change-failure rate are measured; load tests certify 2× projected peak annually. | AI test generation is measured on fault-detection effectiveness, not volume; AI triages failures, clusters flaky tests, and proposes fixes; visual and accessibility regressions are AI-detected. |
| 5 · Optimizing | The quality system learns from every escaped defect: production incidents automatically become regression tests; chaos experiments feed the reliability backlog; the December Drill certifies the whole business, not just the software. | AI co-designs test strategy from risk profiles, maintains the test suite as requirements evolve, and predicts high-risk changes deserving deeper verification — with humans owning release decisions. |
DevOps & CI/CD
How change flows safely into production.
| Level | What it looks like | The AI dimension |
|---|---|---|
| 1 · Initial | Deployments are manual, infrequent, and feared; "it works on my machine"; rollbacks are restores from hope. | None. |
| 2 · Managed | CI exists but is slow and flaky; deployments are scripted but gated by a separate ops team; environments drift. | AI occasionally writes pipeline YAML; nobody validates it; secrets have, on occasion, been sent to external models. |
| 3 · Defined | Every PR runs a sub-10-minute pipeline with security scans and contract tests; GitOps reconciliation; infrastructure as code with policy-as-code; signed artifacts and SBOMs; feature flags decouple deploy from release. | AI assists pipeline authoring and IaC review within approved tooling; policy-as-code blocks AI-generated misconfigurations from reaching production unreviewed. |
| 4 · Quant. Managed | DORA metrics are tracked and elite-trending; canary releases with automated SLO analysis and rollback; change freeze governance is calendar-driven and enforced by tooling. | AI analyzes deployment risk from change content and history, recommends canary parameters, and drafts release notes and freeze-exception justifications for human approval. |
| 5 · Optimizing | Any engineer can ship a safe change in under an hour; the pipeline improves itself — flaky stages are auto-quarantined, slow stages auto-optimized; the December freeze is a non-event because confidence is continuous. | AI is a standing member of the delivery system: predicting failure risk, auto-remediating routine pipeline failures, and keeping the release calendar, capacity plan, and freeze gates synchronized — with humans holding the keys. |
Security & Compliance
How trust is engineered, verified, and evidenced.
| Level | What it looks like | The AI dimension |
|---|---|---|
| 1 · Initial | Security is a pre-launch pen test and a prayer; secrets live in config files; compliance is a binder assembled for auditors. | Developers use unapproved AI tools with sensitive code and data; shadow AI is the biggest unmeasured risk. |
| 2 · Managed | Basic scanning in CI; MFA for admins; a security policy exists but enforcement is manual; compliance is per-country patchwork. | An AI use policy exists on paper; enforcement is honor-system; no visibility into AI tool usage. |
| 3 · Defined | Secure SDLC with threat modeling; SAST/SCA/secret/image scanning in every pipeline; zero-trust networking; a single control set mapped to NIST CSF, SOC 2, and ISO 27001; strictest-common-denominator privacy design across GDPR, COPPA, CCPA, LGPD, and peers. | AI assists threat modeling and control mapping; the compliance traceability matrix is AI-maintained and human-ratified; AI tool usage is brokered, logged, and policy-enforced. |
| 4 · Quant. Managed | Vulnerability MTTR and SLA adherence measured; evidence collection automated from CI logs and cloud config; detection coverage mapped to ATT&CK; DPIAs are living documents gated in the release process. | AI triages findings, deduplicates scanner noise, drafts incident timelines, and continuously cross-checks code and infrastructure against the compliance matrix — with measured precision. |
| 5 · Optimizing | Security is a product feature with a public trust center; controls are code; audits are demonstrations, not archaeology; the compliance fabric absorbs new regulations by updating rules, not projects. | AI is a governed security teammate: hunting anomalies, simulating attacker paths, keeping the regulatory knowledge base current, and proving — continuously — that every requirement, test, and deployment gate still maps to the law of every land served. |
Observability & Monitoring
How the organization sees, understands, and responds to its systems.
| Level | What it looks like | The AI dimension |
|---|---|---|
| 1 · Initial | Monitoring means "the server is up" checks; users are the alerting system; logs are grep'd by hand on individual machines. | None. |
| 2 · Managed | Dashboards exist per team; alerts fire on CPU and disk; on-call is informal and exhausting; every incident is a novel. | AI chatbots are occasionally pasted log excerpts; PII handling in those prompts is nobody's job. |
| 3 · Defined | OpenTelemetry-standard metrics, logs, and traces; symptom-based alerting on SLO burn; every alert links a runbook; no service ships without instrumentation; business metrics are first-class series. | AI assists incident summarization and runbook drafting; log pipelines enforce no-PII policies before any AI processing; anomaly detection augments static thresholds. |
| 4 · Quant. Managed | SLOs ratified by the business with error budgets governing release pace; MTTR and alert-precision measured; follow-the-sun on-call with humane rotations; the Christmas Dashboard unites executives and engineers on one live picture. | AI correlates signals across services into probable-cause narratives, drafts status communications, and predicts capacity pinch-points against the seasonal demand model — all reviewed by humans. |
| 5 · Optimizing | The system sees itself: postmortems feed detection rules; game days verify what dashboards claim; the Christmas Eve war room runs rehearsed playbooks with AI-assisted situational awareness. | AI is a standing watch officer — detecting, correlating, proposing mitigations, and executing pre-approved remediations under human command, with every action logged and reviewable. |
Data Management & Analytics
How data is governed, protected, and turned into decisions.
| Level | What it looks like | The AI dimension |
|---|---|---|
| 1 · Initial | Data lives wherever it landed; nobody knows what's sensitive; reports are spreadsheets emailed around; backups are assumed, never restored. | Analysts paste extracts into external AI tools; nobody has thought about what that means for personal data. |
| 2 · Managed | A warehouse exists; some pipelines are scheduled; ownership is informal; production data flows into analytics with identifiers intact. | AI writes SQL on request; correctness and privacy depend on the individual asking. |
| 3 · Defined | Three-plane architecture (operational / analytical / ML); a data catalog with named owners and classifications; pseudonymization of child data before analytics; retention enforced by an automated lifecycle engine; metrics defined once in a semantic layer. | AI analytics assistants query only governed, pseudonymized marts; training on identifiable child data is architecturally impossible, not merely prohibited. |
| 4 · Quant. Managed | Data quality checks with quarantine-on-failure; lineage end-to-end; restore drills with measured RTOs; unit economics (cost per wish, per delivered gift) reported to the board. | AI monitors data quality drift, proposes dbt tests, and generates first-draft analyses that analysts verify; demand forecasts feed capacity planning with measured accuracy (MAPE). |
| 5 · Optimizing | Data governance is invisible because it is engineered in; the retention engine, rights-request automation, and backup aging are provably synchronized; analytics compounds — every season's data makes the next Christmas smarter. | AI is a governed analyst and steward: keeping the catalog current, detecting misuse, forecasting demand, and proving continuously that the child-data boundary has never been crossed. |
AI & Automation Integration
How the organization itself adopts, governs, and operates AI — the meta-discipline.
| Level | What it looks like | The AI dimension |
|---|---|---|
| 1 · Initial | AI is hype-driven experimentation; a demo exists somewhere; nobody can say what data it touched or what decisions it influences. | This level is the risk: ungoverned AI with unmeasured blast radius. |
| 2 · Managed | A few production models exist; deployment is manual; no registry, no monitoring; nobody owns model behavior after launch. | Models are artifacts without lifecycle; drift is discovered by users. |
| 3 · Defined | An AI Use Policy is ratified; an MLOps lifecycle exists (versioned data, reproducible training, registry, staged promotion); hard rules are absolute — no automated decisions with significant effect on children, no training on identifiable child data; an AI review board approves use cases. | Every model has a model card, an owner, evaluation gates including fairness across regions, languages, and age brackets, and human-in-the-loop design where children are concerned. |
| 4 · Quant. Managed | Model performance, drift, and human-override rates are measured in production; AI incidents are production incidents with postmortems; value is tracked per use case against baseline. | Canary deployment and automatic rollback for models; evaluation suites run in CI; regulatory tracking (EU AI Act, children's-AI guidance) is a standing agenda item. |
| 5 · Optimizing | AI is a governed team member across every discipline in this blueprint — requirements, code, tests, pipelines, security, observability, data — with provenance, accountability, and continuous evaluation; the organization improves its AI usage as deliberately as its software. | The maturity models on this page are the evidence: at Level 5, AI is not a section of the blueprint. It is the way the blueprint is executed — under human governance, always. |
Assessment guidance. Score each discipline honestly — the level is where
all statements are true, not the best one you can claim. Most organizations find they
are one level lower than expected once the AI dimension is included; that gap is the roadmap.
The North Pole reassesses every January, at the post-Christmas retrospective, and targets
trajectory over score.
The Full Blueprint → Global Compliance → Back to the Blueprint