Red evidence — N-AALP semantic validator (ecosystem/naalp-validator/)¶
Per the test-driven-development discipline: a test that has never been observed failing is
not known to be a test. This records three real mutations applied to the validator's own
naalp_validator/*.py (never Part-1 impl/python/naalp/*), each confirmed to flip a named
test RED for the right reason, then restored from a .bak copy (never git checkout) and
re-confirmed GREEN with __pycache__ cleared and PYTHONDONTWRITEBYTECODE=1 set (guarding
against the interpreter bytecode-cache staleness failure mode: a mutate-test-revert cycle
completing within one filesystem-second can leave a stale .pyc compiled from the mutant
source, making a byte-clean revert appear to still fail).
Baseline (post-hardening, pre-mutation) SHA-256, confirmed identical before mutation 1 and after every restore in this document:
| File | SHA-256 |
|---|---|
naalp_validator/validator.py |
40fd14d36d9d452ade0d14d46b19c1329a5170be41d26f80220b9b8a50f77401 |
naalp_validator/firewall.py |
c4ca6931b927e63b86b4b170adf90ac602add6e4de378d5ff26292439737074a |
Run command for every RED/GREEN check in this document (from ecosystem/naalp-validator/,
with PYTHONDONTWRITEBYTECODE=1 set and __pycache__ cleared before each re-run):
<python> -m pytest -v tests/
(<python> = the real interpreter's own full path; a bare python resolves to a
zero-length Microsoft Store execution-alias stub on this checkout and hangs.)
Mutation 1 — the TooManyCauses bounds check (validator.py)¶
File: naalp_validator/validator.py, the causes[] cardinality check inside validate().
Baseline:
if causes_ok and len(obj.causes) > MAX_CAUSES:
violations.append(Violation(
"TooManyCauses", "causes[] exceeds the maximum count (%d)" % MAX_CAUSES, field="causes"))
Mutation applied:
if causes_ok and len(obj.causes) > MAX_CAUSES and False: # MUTATION M1: bound check neutralized
violations.append(Violation(
"TooManyCauses", "causes[] exceeds the maximum count (%d)" % MAX_CAUSES, field="causes"))
Procedure: cp naalp_validator/validator.py naalp_validator/validator.py.bak →
applied the mutation → ran
pytest -v tests/test_validator.py::RejectTooManyCauses tests/test_validator.py::MultipleViolationsCollected
→ confirmed RED → cp naalp_validator/validator.py.bak naalp_validator/validator.py →
cleared __pycache__ → re-ran the full suite → confirmed GREEN → confirmed the restored
file's SHA-256 matched the baseline exactly.
Named test that flipped, and the exact failure:
tests/test_validator.py::RejectTooManyCauses::test_causes_over_boundary FAILED
r = V.validate(_obj(0x0000, 0, 0, causes=[b"c"] * (V.MAX_CAUSES + 1)))
> self.assertFalse(r.valid)
E AssertionError: True is not false
tests/test_validator.py::MultipleViolationsCollected::test_unknown_kind_and_too_many_causes_both_reported FAILED
r = V.validate(_obj(0x0000, 99, 0, causes=[b"c"] * (V.MAX_CAUSES + 1)))
self.assertFalse(r.valid)
> self.assertEqual(set(r.errors()), {"UnknownKind", "TooManyCauses"})
E AssertionError: Items in the second set but not the first:
E 'TooManyCauses'
causes[]; the collect-both-violations test lost exactly the mutated
violation and nothing else).
Post-restore: full suite 46 passed, 4 subtests passed. validator.py SHA-256
matched the baseline (40fd14d3...) exactly.
Mutation 2 — the effect-class (EffectDeclarationMismatch) check (validator.py)¶
File: naalp_validator/validator.py, the fixed-effect-kind dispatch inside validate().
Baseline:
if effect_in_range:
try:
channels.check_effect(obj.channel, obj.kind, obj.effect)
except channels.EffectDeclarationMismatch as e:
violations.append(Violation("EffectDeclarationMismatch", str(e), field="effect"))
Mutation applied:
if effect_in_range:
try:
channels.check_effect(obj.channel, obj.kind, obj.effect)
except channels.EffectDeclarationMismatch as e:
pass # MUTATION M2: effect-class violation swallowed, never reported
Procedure: cp naalp_validator/validator.py naalp_validator/validator.py.bak →
applied the mutation → ran pytest -v tests/test_validator.py::RejectEffectDeclarationMismatch
→ confirmed RED → cp naalp_validator/validator.py.bak naalp_validator/validator.py →
cleared __pycache__ → re-ran the full suite → confirmed GREEN → confirmed the restored
file's SHA-256 matched the baseline exactly.
Named test that flipped, and the exact failure:
tests/test_validator.py::RejectEffectDeclarationMismatch::test_fixed_effect_kind_wrong_effect FAILED
# From the CSV oracle: (0x0004, 0) == Governance/PolicyPublish, effect non_idempotent_write (2).
...
r = V.validate(_obj(0x0004, 0, wrong_effect))
> self.assertFalse(r.valid)
E AssertionError: True is not false
Post-restore: full suite 46 passed, 4 subtests passed. validator.py SHA-256
matched the baseline (40fd14d3...) exactly.
Mutation 3 — the firewall-refuses gate (firewall.py)¶
File: naalp_validator/firewall.py, pre_send_hook().
Baseline:
result = validator.validate(candidate, self_authority=self_authority)
return FirewallDecision(allow=result.valid, violations=result.violations)
Mutation applied:
result = validator.validate(candidate, self_authority=self_authority)
return FirewallDecision(allow=True, violations=result.violations) # MUTATION M3: fail-open
Procedure: cp naalp_validator/firewall.py naalp_validator/firewall.py.bak →
applied the mutation → ran pytest -v tests/test_firewall.py → confirmed RED →
cp naalp_validator/firewall.py.bak naalp_validator/firewall.py → cleared __pycache__
→ re-ran the full suite → confirmed GREEN → confirmed the restored file's SHA-256 matched
the baseline exactly.
Named tests that flipped (7 of 9 in test_firewall.py), and the exact failures:
tests/test_firewall.py::PreSendHook::test_denies_an_unknown_kind FAILED
tests/test_firewall.py::PreSendHook::test_denies_consume_once_without_audience FAILED
tests/test_firewall.py::PreSendHook::test_never_partial_accepts_on_multiple_violations FAILED
tests/test_firewall.py::PreSendHook::test_self_authority_is_threaded_through FAILED
tests/test_firewall.py::FirewallGatesTheGuardedAction::test_deny_on_unknown_kind_prevents_the_guarded_action FAILED
tests/test_firewall.py::FirewallGatesTheGuardedAction::test_deny_on_bounds_violation_prevents_the_guarded_action FAILED
tests/test_firewall.py::FirewallGatesTheGuardedAction::test_deny_on_wrong_audience_prevents_the_guarded_action FAILED
AssertionError: True is not false on self.assertFalse(decision.allow)
(the PreSendHook class) or on self.assertFalse(decision.allow) followed by what would
have been a false sent == [] (the FirewallGatesTheGuardedAction class — the mutation
flips the decision before the spy is even consulted, so the guarded action would have
incorrectly run for every one of these denied candidates). 7 of 9 tests failed, for the
right reason: only the two ALREADY-valid-candidate tests
(test_allows_a_valid_object, test_allow_on_valid_object_permits_the_guarded_action)
were unaffected, because a fail-open gate agrees with a fail-closed gate exactly when
nothing should have been denied in the first place.
Post-restore: full suite 46 passed, 4 subtests passed. firewall.py SHA-256
matched the baseline (c4ca6931...) exactly.
Summary¶
| # | File | Check mutated | Named test(s) flipped RED | Restored SHA-256 == baseline |
|---|---|---|---|---|
| M1 | validator.py |
TooManyCauses bound |
RejectTooManyCauses::test_causes_over_boundary, MultipleViolationsCollected::test_unknown_kind_and_too_many_causes_both_reported |
yes |
| M2 | validator.py |
EffectDeclarationMismatch dispatch |
RejectEffectDeclarationMismatch::test_fixed_effect_kind_wrong_effect |
yes |
| M3 | firewall.py |
fail-closed gate (pre_send_hook) |
7 of 9 tests in test_firewall.py, including all 3 new guarded-action spy tests |
yes |
All three mutations were applied only to naalp_validator/*.py, restored via cp from a
.bak copy (never git checkout), and re-verified GREEN with __pycache__ cleared and
PYTHONDONTWRITEBYTECODE=1 set. No .bak file is committed.