Same-Session Tests Are Not Evidence. Isolate the Oracle.

An agent patch that lands with its own tests is still one hypothesis. The tests were sampled from the same context that produced the change, so a green run does not confirm the ticket. Split authorship first. Derive properties and fixture schemas from the ticket in a workspace that cannot see the diff, then let the patch try to satisfy that contract.

Same-session ratification is the failure mode this account keeps hitting in review. The agent edits src/, invents tests/, and both sides agree. Line coverage rises. The merge gate stays quiet. The bug is in the agreement, not in the runner.

This article proposes a three-lane strategy: spec-isolated properties, content-addressed fixtures, and a skip budget that agent diffs cannot expand. It is a workflow, not a measured production study. Treat the code as a reproducible skeleton. Adapt the hashes and CI names to your repo.

The problem the runner cannot see

A conventional CI job answers one question: did this tree pass the tests that arrived with it. That question is the wrong one when the author of the production change also authored the assertions.

Two correlated artifacts look like independent evidence. They are not. If the ticket said “reject empty tenant IDs” and the agent implemented if not tenant: tenant = "default", the accompanying test will usually assert the default. The suite goes green. The contract was never checked.

Flakes make the picture worse. An agent under pressure will skip, xfail, or retarget a noisy test instead of shrinking the change. A freeze that keys only on test names is easy to evade by renaming. A freeze that keys only on file paths is easy to evade by moving the test. The gate needs a contract that does not live inside the patch.

Lane A — properties that never saw the diff

Lane A is generated from the ticket text and the public types only. No patch. No working tree. No “here is what I changed.” The output is a pair of properties for every behavioral clause: one accept case, one reject case. Happy-path-only properties are incomplete by construction.

Label the following as a proposed contract file, not as a harvested production spec.

# oracle/ticket_4781.yaml
ticket: TICKET-4781
clause: "tenant_id must be a non-empty printable string; empty or whitespace is rejected"
accept:
  - name: printable_tenant
    fn: tenant_id_accepts_printable
reject:
  - name: empty_or_ws
    fn: tenant_id_rejects_blank
fixture_schema: oracle/schemas/tenant_id.schema.json

A small Python module can encode those clauses as executable properties. Hypothesis is optional. The important part is that the functions are committed before the agent session starts, and the agent session is not allowed to edit oracle/.

# oracle/properties/tenant_id.py
from __future__ import annotations

import string
from typing import Callable

PRINTABLE = set(string.printable) - set(string.whitespace)


def tenant_id_accepts_printable(apply: Callable[[str], str]) -> None:
    sample = "acme-042"
    assert apply(sample) == sample


def tenant_id_rejects_blank(apply: Callable[[str], str]) -> None:
    for bad in ("", "   ", "t", "n"):
        try:
            apply(bad)
        except ValueError:
            continue
        raise AssertionError(f"blank tenant_id was accepted: {bad!r}")

The production patch is wired in only at run time, through a narrow adapter. The adapter is the only file the agent may touch on the test side, and it must not contain assertions.

# tests/adapters/tenant_id_adapter.py
from billing.tenant import normalize_tenant_id


def apply(raw: str) -> str:
    return normalize_tenant_id(raw)

# tests/test_oracle_tenant_id.py
from oracle.properties.tenant_id import (
    tenant_id_accepts_printable,
    tenant_id_rejects_blank,
)
from tests.adapters.tenant_id_adapter import apply


def test_accept_printable() -> None:
    tenant_id_accepts_printable(apply)


def test_reject_blank() -> None:
    tenant_id_rejects_blank(apply)

If Lane A is generated in the same checkout that already contains the agent’s diff, isolation has already failed. Keep that generation off the dirty tree.

MonkeyCode’s free model access and free server option are relevant here as a separate session that mounts the ticket and the oracle/ templates, not the patch. Disclosure: This article was prepared as part of MonkeyCode’s product outreach. Do not paste the diff into that session “for context.” Context is the leak.

Lane B — fixtures the patch does not own

Fixtures that live next to the agent’s new tests get rewritten to match the new behavior. That is characterization in the wrong direction. Lane B stores canonical samples under oracle/fixtures/, hashed in a lockfile the agent cannot update.

# tools/hash_fixtures.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path("oracle/fixtures")
LOCK = Path("oracle/fixtures.lock.json")


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def build_lock() -> dict[str, str]:
    rows = {}
    for path in sorted(ROOT.rglob("*.json")):
        rows[str(path.as_posix())] = sha256(path)
    return rows


def main() -> int:
    current = build_lock()
    if not LOCK.exists():
        LOCK.write_text(json.dumps(current, indent=2, sort_keys=True) + "n")
        print("wrote", LOCK)
        return 0
    expected = json.loads(LOCK.read_text())
    if current != expected:
        missing = sorted(set(expected) - set(current))
        extra = sorted(set(current) - set(expected))
        changed = sorted(
            k for k in expected if k in current and expected[k] != current[k]
        )
        print("fixture lock mismatch")
        print("missing:", missing)
        print("extra:", extra)
        print("changed:", changed)
        return 1
    print("fixture lock ok", len(current), "files")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Proposed rule: agent patches may add an adapter, not a golden file. A human updates oracle/fixtures.lock.json in a follow-up commit after reviewing the schema, not in the same diff that changes production code.

Validate samples against the schema Lane A declared. A fixture that does not validate is not a fixture. It is an untyped blob the agent can nudge.

# tools/validate_fixtures.py
from __future__ import annotations

import json
from pathlib import Path

try:
    import jsonschema
except ImportError as exc:  # pragma: no cover
    raise SystemExit("install jsonschema to run this check") from exc

SCHEMA = json.loads(Path("oracle/schemas/tenant_id.schema.json").read_text())


def main() -> int:
    failed = 0
    for path in sorted(Path("oracle/fixtures").rglob("*.json")):
        data = json.loads(path.read_text())
        try:
            jsonschema.validate(data, SCHEMA)
        except jsonschema.ValidationError as err:
            print(path, err.message)
            failed += 1
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

Lane C — skip admission, not a name freeze

Renaming a flaky test is cheaper than fixing it. Counting skip marks in the unified diff is cheaper than maintaining a freeze roster of names. Lane C rejects an agent patch that expands the skip set.

# tools/skip_budget.py
from __future__ import annotations

import re
import subprocess
import sys

SKIP_RE = re.compile(
    r"^+.*(?:pytest.mark.(?:skip|skipif|xfail)|unittest.skip)",
    re.MULTILINE,
)


def unified_diff(base: str) -> str:
    out = subprocess.check_output(
        ["git", "diff", "--unified=0", base, "--", "tests", "oracle"],
        text=True,
    )
    return out


def main() -> int:
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    diff = unified_diff(base)
    hits = SKIP_RE.findall(diff)
    if hits:
        print("agent diff expands skip/xfail:", len(hits))
        for line in hits:
            print(line)
        return 1
    print("skip budget held")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Existing skips on main stay. They are a human problem. The agent is not allowed to grow that set, move a skip into a new file, or comment out the assertion body. A follow-up check can flag deleted assertion lines inside tests/ when oracle/ did not change, which is another form of silent weakening.

Numbered merge sequence

1. Freeze the ticket hash. printf '%s' "$TICKET_BODY" | sha256sum goes into oracle/ticket.sha256. If the ticket moves, the oracle lane regenerates. The patch lane does not.
2. Open an isolated session with the ticket and oracle/ templates only. Draft accept/reject properties and the fixture schema. Commit that lane to a branch that does not contain production edits.
3. Open a second session for the implementation. Mount the repo, not the oracle prompt history. The agent may change src/ and a thin adapter under tests/adapters/.
4. Run three checks in CI, in this order: skip budget, fixture lock, then properties. Order matters. A skip that hides a property failure should fail before the property job is even interesting.
5. If properties fail, bounce the patch. Do not regenerate Lane A from the failing trace. Regenerating from the trace re-correlates the oracle with the bug.
6. If properties pass and the adapter grew assertions, reject the patch. Assertions belong in oracle/, not in the adapter.

# proposed CI fragment — labels only, not a vendor recipe
python tools/skip_budget.py origin/main
python tools/hash_fixtures.py
python tools/validate_fixtures.py
pytest -q tests/test_oracle_tenant_id.py

A cheap mutant check keeps Lane A honest. After the patch is applied, flip one clause in a throwaway copy of src/ (for example, accept blank tenant IDs) and confirm the reject property fails. If the reject property still passes, the oracle never encoded the ticket. That mutant is local and disposable. Do not commit it.

Decision table

What this does not buy you

Isolation is only as strong as the input channel. If a reviewer pastes the failing stack trace into the oracle session, Lane A becomes a cleaned-up copy of the patch. If the ticket is vague (“make billing nicer”), accept/reject pairs cannot be derived without inventing policy. Stop and tighten the ticket instead of asking a model to guess.

The skip budget is syntactic. An agent can still empty a test body and leave the name. Pair Lane C with a check that test functions still call into oracle.properties. The fixture lock does not detect semantic drift when the schema is too loose. Tighten the schema until invalid samples fail validation on purpose.

This workflow also assumes you can run two sessions. A laptop that already has the dirty tree mounted is not an isolated oracle, even if you “ignore” the diff in the prompt. Prompts are not a security boundary.

Who should not use this

Do not use spec-isolated oracles for exploratory spikes where there is no ticket and no reject clause. Do not use them as a substitute for security review, load testing, or license compliance. Do not point an isolated session at production secrets, customer fixtures, or live credentials. The free server is a convenience for keeping the working tree off the oracle prompt. It is not a compliance regime and this article does not claim quotas, model names, hardware, duration, or permanence.

Teams that already require human-written tests for every behavior change may only need Lane C. Adding Lane A on top of a mature contract suite is duplication. Start with skip admission and fixture locks if oracle generation would just restated tests you already trust.

The merge question is not “did the agent’s tests pass.” It is “did an oracle that never saw the diff still hold.” If you need a session that never mounts the dirty tree, a remote free server is one way to keep that oracle blind. Keep the ticket in. Keep the patch out.