Build a Factory Loop

A coding agent fixes a bug, the tests pass, and the branch merges. A week later, another agent makes the same mistake in a different feature. The first fix may have been perfectly good. What was missing was a way for the next worker to use what the previous one had discovered.

I call the process that carries those lessons forward a factory. It includes the tests, project instructions, tools, and release rules that an agent encounters while building a feature. After a failure, some part of that process changes. A test catches the error next time; an instruction explains a constraint that was easy to miss. The model's weights can stay the same while the conditions of its work improve.

OpenAI describes this approach in its account of harness engineering: the team put project knowledge in the repository and used tooling to enforce architectural constraints. An agent starting work could find the decisions and encounter the checks without someone repeating the history in a prompt.

The following example shows how I would build such a loop. The application, people, and agent attempts are invented. The Firestore behavior is documented in the linked sources; this is a worked illustration, not an executed demo.

A private drafts page

Imagine a small issue tracker where users can save a private draft before filing a bug. The drafts live in a Firestore collection. Each document has an ownerUid field, and a security rule permits reads only when that field matches the signed-in user's ID. Alice owns drafts a1 and a2; Bob owns b1.

We want to add a "My drafts" page. Before the agent starts, an engineer writes down what the page must do and how to check it:

Done: a signed-in user sees their own drafts and no one else's.
Invariant: the security rule stays as it is. Any change to the rules
file needs a human's sign-off before merge.
Deciding check: client tests against the emulator return Alice's two
drafts and Bob's one draft to their owners, and deny both cross-user
reads and signed-out access.

An agent can help write this specification. The engineer takes responsibility for accepting it, including the conditions that would prevent release.

There will still be things to discover. A prototype might show that a failed request looks identical to an empty list. The agent points this out, and the engineer realizes the page needs to distinguish the two. They update the specification before judging the next attempt against it. Throughout that revision, the drafts remain private.

Set up the work and its checks

Give the agent its own branch and isolated checkout so concurrent edits stay apart. A checkout offers no security isolation, so restrict credentials separately. For this loop, the worker cannot merge to the protected main branch, change required checks, or deploy. Release credentials stay outside the environment that executes generated code. Anthropic uses this separation in its Managed Agents architecture. It is useful even for a factory built from a script and a CI job: a failed attempt can be discarded without affecting the shared release.

The access tests need to exercise Firestore's enforcement. Its documentation on securely querying data explains a behavior that can surprise developers: rules are not filters. Firestore rejects a client query if it could return a document the caller cannot read. It checks the possible results, even when the collection happens to contain only permitted documents. Server client libraries bypass these security rules, so tests using a server SDK would not establish that client access is restricted.

Use the emulator with the checked-in rules loaded. Firebase's testing guide provides authenticated and signed-out contexts, success and failure assertions, and a way to seed fixtures with rules temporarily disabled. Seed the three drafts, then run the assertions through client contexts. Call the application's own query function; a correct query written inside the test would tell us nothing about a different query used by the page. Alice should receive exactly a1 and a2, and Bob should receive b1. Cross-user document reads and signed-out reads must fail. The Firebase testing quickstarts provide examples to adapt to the repository's JavaScript test runner.

Make the test repeatable. Pin the emulator version, keep the fixtures in the repository, clear the data between tests, and send all test requests to the local emulator. These practices follow the principle described in Bazel's hermeticity documentation: declare the inputs and remove dependence on the host's incidental state. They can be adopted without switching build tools.

Protect the tests and CI policy as well as the application. Otherwise, a patch can pass by deleting the assertion that would reject it. Require separate review of changes to the approved access tests, workflow, and security rules. Configure approvals so an approval of an earlier push cannot authorize a later unreviewed change. GitHub's code-owner rules support required ownership review; the ownership file itself needs protection too. The agent can propose changes to these files, but cannot approve its own acceptance criteria.

Follow the failed query

Suppose the first implementation reads the whole drafts collection and filters by user ID in the browser. A mocked database returns all three documents. Alice sees the expected two, so the mocked test passes.

With the real rules loaded in the emulator, that query is denied. It could return Bob's draft. Filtering afterward cannot make the request permissible, even if every document currently stored happens to belong to Alice.

There are two places the agent could change the code. Allowing any signed-in user to read the collection would get the page working, but it would also let Alice read b1. The cross-user assertion catches that failure, and the rules-file change requires a person's approval. The repair that satisfies the specification is to constrain the query with where('ownerUid', '==', uid), taking the ID from the authenticated session. The security rule stays unchanged and still rejects a client that supplies someone else's ID.

Before accepting a repair, ask why the mock passed and the emulator failed. That explanation tells us which test to keep and which instruction the next agent will need.

Then try the page. Sign in as Alice and confirm that the interface uses the tested query. Repeat with Bob, a signed-out session, and an account with no drafts. Check the empty state and the error state that the prototype exposed. A working query function is insufficient if the page never calls it. Anthropic's application-development experiments found similar gaps by having an evaluator operate the running application.

Bound retries and preserve progress

The program running the agent should enforce the loop. Google's open-source Agent Development Kit is one possible implementation; its ADK 2.0 design separates deterministic routing from tasks that require model reasoning. A simpler project may only need a script. Either way, a completion message from the model cannot bypass a required check.

For this example, I would allow three attempts. Each attempt implements a change, runs the check, and uses the failure output to decide what to try next. After a third failure, the runner stops and reports the attempts, results, and files changed. Three is a proposed limit, not an experimentally established optimum. Choose a per-attempt timeout and an overall time and spending budget as well; one hung command can consume an entire run before a retry is counted. Preserve the last useful state when a limit is reached. Changes to access rules or expected results still need review regardless of how few attempts were used.

Ordinary code controls when work advances from branch to attempt, check, review, queue, and release. The model reasons about the implementation within those stages.

For work that spans sessions, leave a short handoff with the current commit, accepted requirement, failing command and output, and next unresolved question. Record the model and harness versions. A returning agent should inspect the checkout and rerun the relevant check because the summary may be stale or incomplete. Anthropic's long-running-agent study found that incremental work and explicit progress artifacts helped fresh sessions continue.

Review and release

Have someone who did not write the change review the specification, diff, and test output first. Keep the agent's run log available for investigating disputed actions, but avoid supplying its explanation before the reviewer has formed their own view. They should check whether the rules changed, whether the tests cover denied access, and whether the patch satisfies the requirement.

A person is the default reviewer here. Another model can help find defects, though a fresh context alone does not make its judgment independent. Models can share blind spots, and the same narrative can lead both reviewers toward the same assumptions. A separate assignment helps, while executable checks and human responsibility for the requirements remain necessary.

Before merging, test the change together with the target branch and any queued predecessors. GitHub's merge queue supports this. Required GitHub Actions workflows must also run on the merge_group event. If another branch changes the access rules while this feature is under review, the combined candidate needs to pass.

In staging, check the deployed client path with disposable accounts and the intended rules. Production can expose conditions the emulator missed, so the release owner should choose a limited rollout, its duration, and rollback thresholds in advance. Google's SRE chapter on canarying releases describes a partial, time-limited deployment evaluated against a control. Keep measurements separate by version so healthy traffic cannot conceal a failing candidate.

For the drafts page, a small cohort gets the new feature while the control uses the established application. Watch the page's permission-denied rate and latency against the chosen thresholds. Compare shared application health with the control; the control has no drafts page to compare directly. Hold the rollout if telemetry is missing or traffic is too sparse to judge it.

Authorization failures need separate treatment. Passing tests cannot prove privacy in every circumstance, and error rates cannot establish it either. Any evidence of unauthorized access stops the rollout. A known privacy defect must be fixed before exposing the feature to users. The person responsible for release policy owns these decisions.

Make the next feature use what you learned

After the repair, keep the application query test and its exact expected results. Temporarily remove the owner constraint from the application's query and confirm that the test fails, then restore it. If it still passes, investigate whether it reaches the changed query and checks the returned documents. Also retain a separate test that confirms a broad collection query is denied.

Add a short explanation to the agent's instructions: Firestore rules are not filters, so the owner-only drafts page constrains ownerUid; do not weaken the rules to make a test pass. Make the required human approval for rules-file changes part of the merge policy. Assign an owner to each follow-up, as Google's postmortem guidance recommends.

Suppose the next request is to share a draft with one collaborator. Now the access policy does need to change. Revise the expected behavior for that shared draft: the owner and invited collaborator may read it; an uninvited third person may not. Keep unshared drafts private. Update the query contract and agent instructions with the approved policy, and pass the rules change through the required review. The earlier work gives this feature a useful starting point and makes the new privacy obligation explicit.

Check whether the process improves

This factory takes time to build and maintain. To find out whether it helps, evaluate it on representative tasks, including failures like the drafts query. Compare the current and proposed harness from the same starting snapshots under comparable budgets. Record successful outcomes, policy violations, cost, elapsed time, and work that people had to redo.

Anthropic's agent-evaluation guide recommends repeated trials and inspection of both the final state and the execution trace. One passing patch verifies that patch under its tests; it does not establish how reliably the process will produce another.

For the drafts task, check whether the page works with its access policy intact and whether the agent ran the approved tests before claiming completion. Google's guide to evaluating coding harnesses uses observable behaviors like these to help explain broader outcome scores. It recommends batches of runs to see variation and cautions against requiring a fixed tool sequence when several approaches are valid. Use those observations alongside the application tests.

Investigate failures in the evaluation too. An emulator that failed to start says nothing about the correctness of a query. A grader that rejects a valid solution needs revision. I want the evidence to change my judgment of the process, even when I chose it and have invested in it. The engineer's specification and the factory's checks need that scrutiny as much as the agent's patch does.

Some steps will become unnecessary. In the Managed Agents account, context resets that helped one model became redundant with a later model. Remove such scaffolding one component at a time and compare the results. Continue testing the privacy requirement as the way of meeting it changes.

Finally, measure delivery beyond the agent run. DORA's metrics cover throughput through lead time, deployment frequency, and failed-deployment recovery time, alongside instability through failed changes and unplanned rework. More generated pull requests alone would not establish improvement. The extra review and repair they require also count.

For this project, I would keep the drafts task in the evaluation set. When changing a model, prompt, or workflow, run it again and inspect what happened.