How Executable Is Legislation as Code? An Experiment

Rules-as-Code (RaC) is mostly written by hand today. ClauseKit is an experiment to automate the generation of RaC, run across five bodies of law from the EU AI Act to Australian social security.

The ClauseKit pipeline turns legislation into JSON Logic conditions over a typed fact schema, keeps a reference from each rule back to its source provision and evaluates the rules against facts you enter. Of the 262 rules it extracted across the five domains, 171 never reach the engine, because the provision is a judgment call, not a mechanical test. That number, and what changes when the model reads structured input instead of plain text, are what the experiment is about.

The pipeline

Five stages, two input formats.

1. ingest.py: chunks the source by article (EUR-Lex HTML, legislation.gov.au EPUB) or by section and schedule clause (Akoma Ntoso XML from the lex-au corpus)
2. extract.py: sends each chunk to Claude Opus 4.8 with a per-domain fact schema and asks for a JSON array of rules
3. provenance.py: checks each rule's citation against the source chunks
4. rules/: one committed JSON rule set per domain, plus a .REVIEW.md file listing every extracted rule grouped by codifiability, with a warning line on any rule whose citation did not verify
5. api.py and web/: a FastAPI service and a Vue 3 sandbox that evaluates rules against facts you enter

There is no tool use and no function calling. The model returns a JSON array, the code strips a leading json code fence if it added one, and Pydantic v2 validates each object into a Rule. A malformed rule fails on parse rather than reaching the engine.

Extraction: the fact schema is the contract

Every domain declares its fact variables before any extraction runs.

DOMAIN_FACT_SCHEMAS = {
    "ndb": """
Fact variables for NDB scheme:
- incident_type: "unauthorised_access"|"loss"|"unauthorised_disclosure"
- data_categories: array of "health"|"financial"|"identity"|"biometric"|"sensitive"
- encryption_status: "encrypted"|"partial"|"unencrypted"
- individuals_affected: "1-10"|"10-100"|"100-1000"|"1000+"
- likely_recipient: "unknown"|"specific_individual"|"criminal"|"broad_public"
- individual_vulnerability: "general"|"elderly"|"children"|"health_patients"
""",
    # one block per domain
}

The system prompt is the same for every domain:

You extract legislative rules into structured JSON.
Return a JSON array of rule objects. Each rule must use only the provided fact schema variables in conditions.
If a rule cannot be expressed as a deterministic condition (vague standard, judgment call), set condition to null and codifiability to "low".
Use JSON Logic format for conditions: {"and":[...]}, {"or":[...]}, {"==":[{"var":"field"},value]}, {"in":[{"var":"field"},["a","b"]]}.
Return ONLY the JSON array, no markdown fences.

Each rule that comes back is a docref plus a condition plus the model's own read of how codifiable the provision is (obligation text trimmed here):

{
  "rule_id": "annex3_1a_remote_biometric_id",
  "label": "Remote biometric identification systems are high-risk",
  "condition": {"and": [
    {"==": [{"var": "deployment_sector"}, "biometric_id"]},
    {"or": [
      {"==": [{"var": "real_time_biometric"}, true]},
      {"==": [{"var": "post_hoc_biometric"}, true]}
    ]}
  ]},
  "obligation": "AI systems used as remote biometric identification systems are classified as high-risk under Annex III ...",
  "scope": "Biometric identification systems",
  "exceptions": ["Biometric verification systems whose sole purpose is to confirm that a specific natural person is who they claim to be"],
  "codifiability": "high",
  "docref": {
    "source_doc": "EU AI Act (Regulation 2024/1689)",
    "article": "Annex III", "section": "1(a)",
    "provision_uri": "http://data.europa.eu/eli/reg/2024/1689/anx_3"
  }
}

The schema is a hard boundary; the model can only write conditions over the variables it was given. A provision whose trigger depends on something not in the schema comes back as codifiability: "low" with a null condition. That constraint is deliberate: it separates "the legislation is genuinely vague here" from "the schema did not model enough facts", which are different problems with different fixes.

Finding 1: most provisions are not yes/no questions

codifiability is a field on every rule, with three values: high (fully machine-evaluable as JSON Logic), medium (partially evaluable, some conditions need context) and low (a vague standard like "reasonable steps" or "frivolous or vexatious" that cannot be reduced to a mechanical test).

A rule with codifiability: "low" or a null condition never reaches the engine. It evaluates to matched: null, not true or false.

def evaluate_rule(rule: Rule, facts: dict) -> RuleResult:
    if rule.condition is None or rule.codifiability == "low":
        matched = None
    else:
        try:
            matched = bool(jsonLogic(rule.condition, facts))
        except Exception:
            matched = None
    ...

171 of 262 rules never produce a true or false. That is not a pipeline failure. It is the measurement, with two caveats worth stating up front.

First, codifiability is the model's own label, not an independent check. 24 rules are rated medium or high and still have no condition. The taxonomy is a self-report, and the engine drops those rules anyway because there is nothing to evaluate.

Second, the engine's except Exception: matched = None means a genuinely vague standard and a broken JSON Logic condition both surface as matched: null. The pipeline cannot fully tell "this provision resists codification" from "extraction produced a condition the engine could not run".

With those caveats, the shape still holds. The clearest case is the Notifiable Data Breaches scheme: all 54 rules from Privacy Act sections 26WA to 26WR return matched: null. The scheme turns on "likely to result in serious harm" and "a reasonable person would conclude", assessed against the circumstances of a breach. No fact schema turns that into a boolean without smuggling the judgment into a variable.

The EU AI Act has the highest share of high rules, 7 of 32, and all seven are Annex III entries: sector classifications (deployment_sector is an enumeration) and the biometric-ID conditions. The Article 5 prohibitions are not all enumerable, and several are rated low in the review file. Even for the AI Act, 13 of 32 rules have no condition.

Finding 2: structured input changes what gets extracted

This one is an observation, not a measured result. It comes from a single run each way on one short section, and Opus is not deterministic, so the sandbox will show you different output. Read it as an illustration of a real effect, not a benchmark.

Compare mode runs one AKN section, Privacy Act s.26WA (a 3KB fixture, the definition of an eligible data breach), through extract_rules_from_chunk twice with the same empty definitions. The plain-text call gets the section with its tags stripped by a regex. The AKN call gets the raw XML with a one-line note about what eId and <ref> attributes mean.

plain_rules = extract_rules_from_chunk(plain_chunk, definitions="", domain=domain, client=client)
akn_rules   = extract_rules_from_chunk(akn_chunk,   definitions="", domain=domain, client=client)

On one run the plain-text path returned three rules, none above medium:

[
  {"label": "... unauthorised access or disclosure likely to cause serious harm",
   "codifiability": "medium",
   "condition": {"and": [{"in": [{"var": "incident_type"},
                                 ["unauthorised_access", "unauthorised_disclosure"]]}]}},
  {"label": "... loss where unauthorised access/disclosure likely",
   "codifiability": "medium",
   "condition": {"and": [{"==": [{"var": "incident_type"}, "loss"]},
                         {"in": [{"var": "likely_recipient"},
                                 ["unknown", "criminal", "broad_public", "specific_individual"]]}]}},
  {"label": "Reasonable person assessment of likely serious harm",
   "codifiability": "low", "condition": null}
]

The AKN path returned four, one rated high:

[
  {"label": "Eligible data breach - triggering event and serious harm",
   "codifiability": "low",
   "condition": {"and": [{"in": [{"var": "incident_type"},
                                 ["unauthorised_access", "unauthorised_disclosure", "loss"]]}]}},
  {"label": "Triggering event for eligible data breach",
   "codifiability": "high",
   "condition": {"in": [{"var": "incident_type"},
                        ["unauthorised_access", "unauthorised_disclosure", "loss"]]}},
  {"label": "Serious harm reasonable person test", "codifiability": "low", "condition": null},
  {"label": "Factors relevant to assessing likelihood of serious harm",
   "codifiability": "low", "condition": null}
]

The plain-text second rule invents a likely_recipient constraint that s.26WA does not require. The AKN path isolates a clean trigger over all three incident types and keeps the "reasonable person" test as its own null-condition rule.

Two things to be honest about. extract_rules_from_chunk truncates input at 8000 characters, and XML tag overhead means that on a section longer than this fixture the two arms are not seeing the same amount of legal text. And a single pair of runs is an anecdote. What it suggests is that section boundaries and cross-references in the XML give the model something to follow instead of prose to guess at. The sandbox lets you run the comparison yourself and toggle the raw corpus XML behind any AKN-sourced rule.

The gap provenance does not close

Every rule carries a docref. The pipeline verifies it in two steps.

verify_docref normalises the cited article label and checks it appears in a source chunk. This confirms the citation points at something real. It is a loose substring match, not a check that the chunk supports the rule.

verify_numeric_grounding was added after a review found a rule could state "13 days" where the source says "30 days" and still pass, because the citation itself was correct. It pulls every digit-form day, month and year figure out of a rule's obligation text and checks each appears in the matched chunk, after a light whitespace and hyphen normalise. Spelled-out numbers and units it does not know ("two business days") slip through.

_NUMERIC_SPAN = re.compile(r"\d+\s*(?:day|month|year)s?", re.IGNORECASE)
 
def verify_numeric_grounding(obligation: str, chunk: dict) -> GroundingResult:
    spans = _NUMERIC_SPAN.findall(obligation)
    chunk_text = _normalise(chunk["text"])
    unmatched = [s for s in spans if _normalise(s) not in chunk_text]
    return GroundingResult(verified=not unmatched, matched_article_id=chunk["article_id"])

What neither check does is verify that the JSON Logic condition captures the clause. The invented likely_recipient constraint from the plain-text 26WA run is a correct citation, correct numbers and a wrong condition. Nothing in the pipeline catches it. Citation identity and numeric fidelity are mechanical. Condition fidelity is a reading of the law, and this experiment surfaces that gap rather than closing it.

What was hard

The engine is the smallest file in the repo. The friction was everywhere else.

The rule engine's dependency ships Python 2 code

json-logic 0.6.3 is pinned exactly, because 0.6.2 is worse, and both call tests.keys()[0] and use reduce without importing it. The venv is hand-patched with two edits, reapplied every time it is rebuilt.

A stale bytecode cache silently nulled every rule

On the Modal deployment, the image build precompiled a .pyc for json_logic before the patch step rewrote the source. Python kept loading the old bytecode. Every rule evaluated to matched: null while every endpoint returned 200, so the app looked healthy. The fix that landed in code was to delete __pycache__ after patching. The lesson noted next to it: smoke-test /evaluate with a real input that expects true, not a 200 on /domains.

The schema has a field the engine ignores

Rule.exceptions is populated by extraction and shown in the UI, but the engine never resolves an exception as an override of its parent rule. Statutory exceptions are stored as metadata, not evaluated. JSON Logic has no defeasibility, so "rule A applies unless rule B" cannot be expressed at all.

Codifiability is the model's call

No domain expert has checked the high, medium and low assignments against the source. A wrong high is a rule that looks trustworthy and is not.

Where it goes next

Two things this needs before it is more than an experiment. The JSON Logic ceiling (no temporal constraints, no modal operators, no defeasibility, no role-relative obligations) is where a purpose-built formalism like Catala or defeasible logic earns its complexity, for the subset of provisions that need it. And condition fidelity wants a second independent formalisation of the same provision, with any disagreement between the two used as the trigger to send that rule for human review.

The shape generalises to any regulation with a bounded set of facts: a compliance standard, an eligibility rule, an internal policy. The transferable part is not the extraction. It is keeping an explicit bucket for the provisions that should not be forced into a boolean and routing those to a person instead of a default answer. If you try this on your own regulation, read that bucket first. It is the result.

Code and further reading

The source is on GitHub and the sandbox is live at clause-kit.netlify.app. The stack is Python 3.12, FastAPI, Pydantic v2, json-logic and Vue 3.

Legislation used:

• EU AI Act: Regulation (EU) 2024/1689 on EUR-Lex (Articles 5, 6, Annex I, Annex III)
• NDB scheme and Privacy APPs: Privacy Act 1988 (Cth) on legislation.gov.au (NDB: ss.26WA to 26WR; APPs: Schedule 1)
• SSA bereavement: Social Security Act 1991 (Cth) on legislation.gov.au (bereavement provisions)
• SIS death benefits: Superannuation Industry (Supervision) Act 1993 (Cth) on legislation.gov.au (ss.55A to 55C, 68AA to 68AAF)

Standards and libraries:

• JSON Logic for the condition format
• Akoma Ntoso and the European Legislation Identifier for the provision URIs
• The lex-au AKN 3.0 corpus on Hugging Face

AI Tools

Claude Code was used to build the pipeline and sandbox. Claude Opus 4.8 is the extraction model inside the pipeline itself. Claude was used to draft this post.