TL;DR: The second post in the loop engineering series. The first post covered the official delegation ladder structure. This one uses TQEF, a translation quality system I built myself, to illustrate two things with very different lifespans: the harness, the mechanisms that keep things running, gets replaced constantly; the criteria, the standards that determine whether output meets the bar, accumulates through every rebuild. The post includes a few mistakes I actually made, and an honest account of what this system still has not finished.

The last post ended with a debt: “How I spent these past few months colliding with the same structure, and which mistakes I made along the way: that’s for the next post.” This is that post.

Conclusion first, then how I arrived at it.

In a continuously operating AI system, some things expire quickly: prompts, tools, databases, scheduling. I call all of these mechanisms collectively the harness. One accident, one upgrade, one architectural change, and the harness has to be rebuilt.

The standards that determine whether output is good are different. I call this layer the criteria. Defined with enough specificity, they survive every rebuild and grow more complete with each failure.

What the System Actually Looks Like

TQEF (Translation Quality Evaluation Framework) is a scoring system I built to serve Agora Square, a real-time meeting translation tool I developed. It runs across four nested loops.

The innermost loop handles translation: the model receives input from four speech-recognition streams and produces a target-language segment. The second loop handles verification: TQEF scores output across five dimensions, and anything that fails is sent back for re-translation. The third loop operates at the session level: after a meeting ends, transcripts enter the corpus and become material for future improvement. The outermost loop runs periodically, reviews patterns across multiple sessions, and feeds findings back into prompts and glossaries.

Agora's four nested loops (agent loop, verification loop, event-driven loop, hill-climbing loop) operating in sequence

TQEF scores on five dimensions.

First: terminology accuracy, the highest-weighted dimension. If “semiconductor” gets rendered as something else, no amount of fluency in the surrounding sentences matters. Second: fidelity, checking whether the translation drifts from the original meaning. Third: key information retention, confirming that numbers and proper nouns are neither omitted nor mistranslated. Fourth: naturalness, whether the translation reads like something a person would actually say. Fifth: severe errors, any single occurrence of which results in an automatic fail.

These five dimensions are the criteria layer. The translation, verification, and event-triggered operations described above are the harness layer. Every incident below happened in the harness.

How the Harness Expired: Three Incidents

The first was the most dramatic. During a collaboration session, I used osascript to write to a file. Nested quote handling failed, and the 226-line core evaluation script eval_runner.py was truncated to a single line: #!/usr/bin/env python3. The entire script was gone. This was not just a “rewrite it” situation. It was a systemic warning: when a single session handles strategy, code-writing, and deployment simultaneously, something going wrong is a matter of when, not whether. The nested quotes happened to be the trigger.

The response that followed was worth more than the incident cost. I split the entire collaboration workflow into three tracks: one for reconnaissance and strategy only, with no code access; one for bulk file operations; one for precise single-file edits and deployment. No two tracks may touch the same file simultaneously. Every phase ends with a confirmed save, a commit, and a recoverable version. This division was not something I designed in advance: the incident forced it. Looking back, it was the highest-return decision in the entire development history, because nothing like that version confusion has happened since.

The second incident was quieter. While extending the system, I guessed at database column names based on documentation. When I actually queried the database, the structure did not match what the documentation described. This happened twice. Both times, the discrepancy was caught during early reconnaissance and caused no real damage. The rule it produced was simple and now non-negotiable: column names are verified by actual query results, never by documentation or memory.

The third was a duplicate-entry bug. Scoring results for the same sentence in the same evaluation round were written to the database twice. The cause was a missing uniqueness constraint. The fix was direct: add the constraint, then clean up the duplicate records.

There was also a less dramatic but more common pattern that kept recurring: written but not wired. A feature would look complete (the code was there), but an import was missing, a route was unconnected, a deployment never ran, or the front end was not rendering anything. Individual pieces existed; the full path had never actually been walked. This built one firm habit: after any change, run a full end-to-end test from the beginning, confirm the thing is actually alive, and do not trust “it should be working.”

How the Criteria Compounded: One Improvement with a Clear Cause

The sharpest contrast with those incidents came the day three glossaries went live. The baseline score at that point was 4.69 out of 5, with one severe error still in the results. In that round, I deployed three industry glossaries alongside a business-vocabulary ruleset, covering semiconductor, circular economy, regenerative medicine, and common commercial terminology in one push.

The next round: overall score jumped to 4.97. Severe errors dropped to zero.

That improvement can be traced to the glossary deployment. The score increase corresponds to a specific engineering change. This is the value of the criteria layer: even as the harness is rebuilt repeatedly, the criteria make it possible to compare results across rounds using the same standard. Whether the underlying script was once destroyed by osascript, whether column names were verified against actual queries, or whether the duplicate-entry bug was ever fixed, the five scoring dimensions, while their details were continuously refined, have not changed their core logic from the first round to the present. Every engineering change can be measured against the same scale. That scale has never been rebuilt alongside the harness. That is what compounding means here.

Scores in subsequent rounds have moved between 4.78 and 4.90. That variation is normal: 29 sentences is a small sample, and the scoring model itself has some randomness. But the overall trend is upward, and each round can be traced back to specific engineering changes and their likely effects. This connects directly to a point from the previous post: loops can automate how a process runs, but what counts as a sentence passing review is always something you have to define yourself, specifically enough to score against.

The Parts That Are Not Finished

What TQEF has completed is Phase 1: a single scoring engine, 29 test sentences, seven evaluation rounds. Phase 2 plans five automated corpus-collection channels; two are complete, the rest are still in design. I also planned a Stage B specifically for validating whether the scorer itself is reliable. Beyond adjusting the scoring prompt, Stage B calls for cross-validation with a separate metrics set, calibration with human expert scores, and repeated testing of the same data to confirm stability. This stage has not really moved forward. It is sitting behind other priorities. That is an honest outstanding item, not a throwaway remark.

It cannot be left out. If I am arguing that criteria outlast harness and are worth investing in first, I also have to be clear: criteria are not finished once you write them down. They still require cross-validation, human calibration, and stability testing. The difference is that this work is usually about refining the same scale, not replacing the entire judgment framework each time. I have not finished that validation myself.

What Both Posts Together Are Trying to Say

The previous post ended with one sentence: loop engineering can automate how a process runs, but it cannot decide for you what running correctly means. This post puts that sentence on solid ground. The harness layer: I destroyed it, repaired it, and rebuilt it. The three-track division was forced by an accident, not planned in advance. The criteria layer: defined clearly from day one, it survived every rebuild, and grew more complete with every incident.

If you do not have a system like TQEF, the contrast still applies. In any continuously operating AI system, the mechanisms keeping it running will eventually be torn down and rebuilt, and that is close to certain. The thing worth thinking through first is what “done” means. Write the standard specifically enough that different people or models can use it to evaluate whether a result passes. That is very hard to fully delegate to a model. But once the standard exists, it will usually outlive any version of the prompt, the tooling, or the workflow.

References

Previous post in this series

TQEF source material

  • All scoring data, incident records, and development timelines cited in this post are first-hand records from Paul’s own system, not summaries of third-party reports