Red-evidence: N-AALP OPA/Rego policy-enforcement integration¶
Mutation-survival evidence for the load-bearing properties of naalp_opa_policy. Each row
was produced by actually editing naalp_opa_policy/opa_policy.py (never any impl/python/naalp/*
file, which is Part-1 and out of scope for this task), confirming the named test(s) flip RED
for the stated reason, then restoring the file from a .bak copy (never git checkout),
clearing __pycache__ (PYTHONDONTWRITEBYTECODE=1 set for every run so no stale bytecode
can mask a mutation or fake a revert), and re-running the full suite GREEN with the file's
hash re-confirmed identical to the pre-mutation baseline.
Baseline / reverted file hash for naalp_opa_policy/opa_policy.py (SHA-256), confirmed
identical before mutation 1 and after every one of the four reverts below:
Run command (from ecosystem/naalp-opa-policy/, invoking the real installed Python
interpreter for this platform -- on Windows the Microsoft-Store python/python3
execution-alias stubs resolve ahead of a real install on PATH and hang, so on that platform
invoke the actual interpreter binary directly rather than the bare python command):
All commands below were run from ecosystem/naalp-opa-policy/ with
PYTHONDONTWRITEBYTECODE=1 set and __pycache__ removed immediately before every test
invocation (mutate, revert, and re-confirm), so no cached .pyc could report a result that
did not come from the file on disk at that moment. Full suite size: 10 tests. Every mutation
and revert below was performed with a real, running opa binary (github.com/open-policy-agent/opa
v1.19.1, installed this session via winget install --id open-policy-agent.opa -e
--accept-package-agreements --accept-source-agreements) -- nothing in this evidence is
produced against a stub or fake engine.
OPA engine install + invocation notes (established before any test was written)¶
- Installed via
winget install --id open-policy-agent.opa -e --accept-package-agreements --accept-source-agreements(the naivewinget install openpolicyagent.OPApackage id does not exist in the winget index;winget search opasurfaced the real idopen-policy-agent.opa, version 1.19.1). Confirmed viaopa versionin a freshly spawned process (a new process picks up the winget-updated PATH without any manual refresh):Version: 1.19.1,Rego Version: v1. - Reproduced Windows defect, hand-confirmed this session: the real
opabinary's file loader mis-parses an absolute path whose first path component is a drive letter followed by a colon (the shapeopa eval -d <driveletter>:<backslash>...\policies ...) as a URL whose scheme is that drive letter, and fails with a bogusGetFileAttributesEx \Users\...: The system cannot find the path specifiederror -- confirmed by running the identicalopa evalinvocation twice, once with an absolute drive-rooted path (fails) and once with acwd-relative path (succeeds).OPAEngine._runworks around this on every platform by ALWAYS invokingopawithcwdset to the parent of the policy directory and passing only the directory's basename on the command line -- an absolute path is never handed toopa. opa evalresult shapes, hand-confirmed this session (these are exactly whatOPAEngine._run/decidebranch on):- A clean Allow:
{"result": [{"expressions": [{"value": true, ...}]}]}, exit 0. - A clean Deny:
{"result": [{"expressions": [{"value": false, ...}]}]}, exit 0. - A missing policy file/dir or a Rego parse error:
{"errors": [{"message": "..."}]}(or, for a parse error, an additional"code": "rego_parse_error"and"location"), exit 2. - An undefined query (a package with no matching rule): bare
{}, exit 0 -- no"result"key at all. -I/--stdin-inputreads the JSON input document from stdin;-d <dir>recursively loads every*.rego/*.json/*.yamlunder that one directory (confirmed against a real run loadingnaalp_authz.rego+data.jsontogether via a single-d).
Non-circular vector provenance (established before any mutation test)¶
OracleGroundingTests.test_conformance_vectors_match_real_engine confirms, before any
fail-closed or mapping test below relies on it, that all 9 hand-derived {input, expected}
pairs (5 allow, 4 deny) in vectors/conformance_vectors.json -- each expected value derived by reading the
three rule bodies (effect_within_ceiling / signer_matches_grant / audience_ok) in the
checked-in naalp_opa_policy/policies/naalp_authz.rego against that vector's own input
fields, exactly as an administrator reading the policy would reason about it -- match what the
REAL opa engine actually returns for that policy. The oracle is the real engine evaluating
the real checked-in .rego/data.json; nothing in this package's own Python code produced
any expected value. MappingConformanceTests then reproduces four of those same logical
scenarios (matching-everything allow, effect-over-ceiling deny, wrong-audience deny, and the
exact input-document shape) through REAL, cryptographically signed-and-verified N-AALP
objects (naalp.envelope.sign / naalp.envelope.verify, real ML-DSA-65 keys via
naalp.cose.mldsa_keygen) and confirms naalp_opa_policy.build_input +
naalp_opa_policy.authorize_with_policy reach the identical Allow/Deny outcome the grounded
vector predicts -- so the code genuinely under test by this package (the mapping and the
fail-closed integration, not the policy engine) is graded against a REAL cryptographic
pipeline, never a synthetic dict standing in for a real object.
M1 -- fail-open on an unreachable/erroring engine (test_engine_unreachable_denies_even_a_would_be_allow)¶
Mutated: OPAEngine.decide(), the exception branch around the real opa subprocess call:
try:
result = self._run(input_doc)
except OPAEngineError:
- return False # fail-closed: an unreachable/erroring engine denies, never allows
+ return True # MUTATION M1 (red-evidence): fail-open on an unreachable/erroring engine
RED, confirmed: 1 of 10 tests failed --
The test builds a real, policy-conformant object, first confirms it is genuinely Allowed by
a REAL reachable engine (so the scenario is provably a "would-be-allow"), then repeats the
identical call against an OPAEngine configured with a nonexistent opa_binary. With the
mutation, an unreachable engine is silently treated as an Allow and no PolicyError is
raised -- exactly the fail-open regression this property exists to prevent. The direct
lower-level check (test_opa_engine_error_raises_opaengineerror_internally, which asserts on
OPAEngineError at the _run() layer rather than decide()'s boolean) is UNAFFECTED, since
_run() itself is untouched by this mutation -- confirming the mutation targets exactly the
decide() fail-closed wrapping and nothing upstream of it.
Reverted: the file was restored from its .bak copy (never git checkout), __pycache__
cleared, hash re-confirmed 584a6e5057bef754a6599d8e6a346ec7e98e7428a3c5bd012fb305440953f922.
GREEN, confirmed: 10/10 tests pass.
M2 -- skip the OPA consult entirely (test_deny_is_actually_enforced_before_a_would_be_allow +4 others)¶
Mutated: authorize_with_policy(), the enforcement choke point:
input_doc = build_input(obj, alg, verified_pubkey, grant)
- if not engine.decide(input_doc):
+ # MUTATION M2 (red-evidence): skip the OPA consult entirely -- always proceed as allowed.
+ if False and not engine.decide(input_doc):
raise policy.PolicyError(
RED, confirmed: 5 of 10 tests failed --
FAIL: test_deny_is_actually_enforced_before_a_would_be_allow
AssertionError: PolicyError not raised
FAIL: test_engine_unreachable_denies_even_a_would_be_allow
AssertionError: PolicyError not raised
FAIL: test_undefined_query_result_denies_even_a_would_be_allow
AssertionError: PolicyError not raised
FAIL: test_destructive_effect_over_ceiling_is_denied
AssertionError: PolicyError not raised
FAIL: test_wrong_audience_object_is_denied
AssertionError: PolicyError not raised
Every test that expects authorize_with_policy to raise PolicyError flipped, because the
mutated if condition can never evaluate its right-hand side and the function always falls
through as if the (never-run) engine had allowed the object -- exactly the "the policy
consult was silently bypassed" failure this property exists to prevent. The 5 tests that
expect NO exception (a genuine Allow) were unaffected, since a function that always allows is
indistinguishable from a correct one on those inputs alone -- exactly the boundary this
mutation targets.
Reverted: restored from .bak, __pycache__ cleared, hash re-confirmed
584a6e5057bef754a6599d8e6a346ec7e98e7428a3c5bd012fb305440953f922.
GREEN, confirmed: 10/10 tests pass.
M3 -- trust the object's self-asserted signer field (test_signer_resolved_from_verified_pubkey_not_self_asserted_field +4 others)¶
Mutated: build_input(), the principal-resolution line (the property this
package's own module docstring names explicitly):
- principal = policy.resolve_auth_principal(
- policy.SOURCE_SIGNATURE, identity.signer_id(alg, verified_pubkey)
- )
+ # MUTATION M3 (red-evidence): trust the object's own self-asserted `signer` body field
+ # instead of the cryptographically-resolved principal.
+ principal = obj.signer.decode("utf-8", errors="replace")
return {
RED, confirmed: 5 of 10 tests failed/errored --
FAIL: test_signer_resolved_from_verified_pubkey_not_self_asserted_field
AssertionError: 'attacker-claimed-identity-not-the-real-signer' != 'bciqn4bwilfr74zzc6ou6xlkgqzi7selw2tc2lxepf34pbcpilun6f4q'
FAIL: test_build_input_matches_grounded_vector_shape
AssertionError: {...'signer': '<mojibake decode of the raw pubkey bytes>'...} != {...'signer': 'bciqo2gxjfh673...'...}
ERROR: test_undefined_query_result_denies_even_a_would_be_allow
naalp.policy.PolicyError: EffectNotAuthorized: policy engine denied the object (...signer='<mojibake>')
ERROR: test_matching_read_only_object_is_allowed
naalp.policy.PolicyError: EffectNotAuthorized: policy engine denied the object (...signer='<mojibake>')
ERROR: test_engine_unreachable_denies_even_a_would_be_allow (via its "confirm a real reachable
engine allows this" sanity assertion)
The dedicated test flipped directly (the deliberately spoofed signer body field
b"attacker-claimed-identity-not-the-real-signer" leaks straight into doc["signer"] instead
of being replaced by the resolved principal). More significantly, every OTHER test that builds
an ordinary real object (signer=pk, the real raw ML-DSA-65 public-key bytes, which is NOT a
valid identity string and does not survive UTF-8 decoding cleanly) also broke, several of them
turning a previously-clean Allow into an unexpected PolicyError, because obj.signer's raw
bytes are never the correct principal string -- confirming this mutation corrupts the identity
on EVERY real object, not only an adversarial one, which is exactly why obj.signer must
never be trusted as an identity (policy.py's own module docstring: only a signature-derived
identity is an authorization principal).
Reverted: restored from .bak, __pycache__ cleared, hash re-confirmed
584a6e5057bef754a6599d8e6a346ec7e98e7428a3c5bd012fb305440953f922.
GREEN, confirmed: 10/10 tests pass.
M4 -- fail-open on an undefined query result (test_undefined_query_result_denies_even_a_would_be_allow)¶
Mutated: OPAEngine.decide(), the undefined-result branch:
results = result.get("result")
if not results:
- return False # fail-closed: an undefined query result denies
+ return True # MUTATION M4 (red-evidence): fail-open on an undefined query result
RED, confirmed: 1 of 10 tests failed --
FAIL: test_undefined_query_result_denies_even_a_would_be_allow
AssertionError: PolicyError not raised
The test first confirms a real object is genuinely Allowed by an OPAEngine pointed at the
package's real query (data.naalp.authz.allow), then repeats the identical call with an
OPAEngine configured to query a nonexistent rule (data.naalp.authz.no_such_rule_exists),
which opa eval resolves cleanly (exit 0) but with NO "result" key in its JSON output --
the "engine ran fine, but there is nothing to allow" case, distinct from M1's "the engine
itself could not run at all". With the mutation, this undefined result is treated as an
Allow and no PolicyError is raised. Exactly the one test targeting this branch flipped (9 of
10 still passed), confirming this mutation is isolated from M1's exception-handling branch and
from M2's enforcement choke point.
Reverted: restored from .bak, __pycache__ cleared, hash re-confirmed
584a6e5057bef754a6599d8e6a346ec7e98e7428a3c5bd012fb305440953f922.
GREEN, confirmed: 10/10 tests pass.
Why these four, and why every mutation targets opa_policy.py rather than naalp.* or the real opa engine¶
naalp_opa_policy is glue over TWO real, independently-graded systems it never re-implements:
the Part-1 SDK (naalp.envelope/naalp.identity/naalp.policy/naalp.cose, whose own
cryptography and effect-lattice logic carry their own graded red-evidence and are out of
scope for this task) and the real opa binary (whose Rego evaluation semantics are OPA's own
concern, not this package's -- re-implementing them would be exactly the fake-verifier
failure this programme's rules forbid). What IS this task's own code, and therefore what these
four mutations target, is this package's own choices: that decide() actually treats an
unreachable/erroring engine as a Deny (M1) rather than silently allowing, that
authorize_with_policy() actually consults the engine before ever returning cleanly (M2)
rather than becoming a no-op gate, that the identity fed into the policy is the
cryptographically-resolved principal and never the object's own self-asserted signer field
(M3) -- the property this integration exists to preserve across the OPA boundary -- and
that decide() treats an undefined query result as a Deny (M4) exactly as it treats an
unreachable engine, but via a genuinely distinct code path (a clean opa run with no matching
rule, rather than a failed opa invocation). All four are exactly the properties this
integration must get right -- not the SDK, and not the OPA engine itself.