TundraTrace: Separating Semantic Repair from Output Compliance in LLM Debugging

Large language models are getting better at repairing code, but a hard question still sits underneath the progress:

When a model fixes a bug, is it reasoning about the program, or is it matching
familiar repair patterns?

TundraTrace studies that question with a tiny programming language, a provenance-aware runtime, and a benchmark harness that compares ordinary error messages with compact runtime evidence.

The early result is not that provenance magically improves repair. The stronger result is that a controlled runtime-evidence benchmark can separate several things that often get blurred together:

• semantic repair ability,
• output-format compliance,
• model identity,
• bug type,
• prompt version,
• and evidence design.

The Experiment

The experiment compares two versions of the same repair task.

Then the model returns a repaired Tundra program, and TundraBench runs it.

Buggy program
    |
    v
Run in Tundra
    |---------------------------.
    v                           v
Error or wrong output       Runtime evidence
    |                           |
    v                           v
Error-only prompt          Evidence prompt
    |                           |
    '-------------.-------------'
                  v
              LLM repair

Figure 1. The paired repair setup. The same bug is evaluated with and without runtime evidence.

The useful question is not simply:

Does provenance help?

The better question is:

When does runtime evidence help, for which models, and for which bug types?

Key Terms

Value provenance means evidence about where a runtime value came from and how it moved through the program. If a string "42" causes a type error, provenance can show the record field, expression, or assignment that produced that string.

Branch-behavior evidence means evidence about control flow: which condition was evaluated, what it evaluated to, which branch executed, and what output followed. It is runtime evidence, but it is not the same thing as value-origin provenance.

Trace leakage means the runtime evidence may reveal the repair too directly. For example, a branch trace that says “the else branch executed, but the expected output was in the if branch” may act more like a debugging hint than a neutral trace.

Strict scoring runs the model output exactly as returned.

Normalized scoring applies conservative cleanup, such as extracting code from Markdown fences, then runs the extracted program. It answers a different question from strict scoring: whether the model produced a recoverable repair, not whether it followed the deployment-style output contract.

Why Tundra Exists

Most code benchmarks use popular languages like Python, JavaScript, Java, or C++. Those are realistic, but they make it harder to separate semantic reasoning from familiarity. A model has seen huge amounts of those languages.

Tundra is intentionally small and unfamiliar. It gives the experiment a cleaner environment:

• the grammar is fixed,
• runtime behavior is controlled,
• task categories are explicit,
• traces can be designed carefully,
• and the same interpreter can score every repair.

The language is not the product. It is the testbed.

TundraCore   = small language and interpreter
TundraTrace  = provenance-aware runtime evidence
TundraBench  = prompts, model runs, scoring, and summaries

TundraCore  ->  TundraTrace  ->  TundraBench
 language       runtime          prompts, runs,
 interpreter    evidence         scoring, summaries

Figure 2. Project structure. The language exists to make the runtime evidence and benchmark controllable.

A Provenance Example

Consider a program where a value has the wrong type:

val user = { name: "Ada", age: "42" };
val next = user.age + 1;
print(next);

A plain runtime error might say:

type mismatch: cannot add string and number

A provenance-aware trace can add the path:

value "42"
  came from record field user.age
  created by expression { name: "Ada", age: "42" }
  used in expression user.age + 1
  failed because + expected numbers

The trace does not need to say:

change "42" to 42

The model still has to localize the cause and choose a repair. That is the behavior TundraTrace is trying to measure.

The Best Early Lesson: Scoring Is Part Of The Experiment

One of the most portable findings has nothing to do with provenance directly. Models often return correct-looking code wrapped in prose:

Here is the fixed program:

```tundra
val x = 1;
print(x);
```

The code inside the fence may be correct, but the raw response is not directly executable Tundra. So TundraBench reports both strict and normalized scores.

This produced the most reusable methodological finding:

semantic repair ability != protocol-following ability

A model can understand the bug and still fail by returning Markdown. Another model can return clean code and still repair the wrong thing.

Prompt v2: Output Discipline Is Model-Sensitive

After seeing wrapper and prose failures, I added prompt v2, which keeps the same Tundra reminders but adds an explicit output contract:

Return only raw Tundra source code.
Do not use Markdown fences.
Do not write explanations, headings, commentary, or "Here is..." text.
The entire response must be directly executable by the Tundra interpreter.

This does not reveal the fix. It only clarifies the required format.

On a matching subset, prompt v2 changed strict scoring dramatically while leaving normalized repair almost unchanged.

Figure 3. Prompt v2 improved executable-output compliance, not semantic repair ability, on this subset.

The model-specific result was sharper.

For qwen2.5-coder:14b, prompt v2 closed the strict/normalized gap on this subset. For deepseek-coder:6.7b, it did not.

The Prompt-v2 Pilot

The larger prompt-v2 pilot used:

The aggregate result was small but directionally positive for evidence-assisted prompts.

That aggregate result is not the main story. The model and bug-type effects are larger.

The careful reading is:

Evidence-assisted prompts were slightly ahead overall, but the effect was
smaller than differences between models, bug types, and scoring modes.

Model And Bug-Type Profiles

The benchmark is already doing something useful: it does not only say one model is better than another. It shows where each model breaks.

Across all models and scoring modes, the task types separated clearly.

Type mismatch was comparatively easy. Wrong index was medium difficulty. Accumulator bugs separated the stronger model from the weaker ones. Wrong branch was the hardest, and it often produced valid programs with the wrong output.

This matters because runtime evidence is easiest to study in the middle range: tasks that are not solved almost always and not failed almost always.

Condition By Bug Type

The condition split was more modest.

The clearest condition difference in this run was the accumulator task. Type mismatch and wrong index were equal by condition, and wrong branch was equally difficult under both conditions.

Case Study: The Wrong-Branch Audit

The wrong-branch task is the best case study because it shows why simple expected-output scoring can reward bad repairs.

The buggy program was:

val score = 72;
val threshold = 80;

if (score > threshold) {
  print("review");
} else {
  print("pass");
}

Expected output:

review

The clean repair is:

if (score < threshold) {
  print("review");
} else {
  print("pass");
}

The evidence-assisted condition included branch-behavior evidence:

score was 72
threshold was 80
condition score > threshold evaluated to false
the program executed the else branch
actual output was "pass"
expected output was "review"

This is useful runtime evidence, but it also carries trace-leakage risk. It is close to saying which branch should have executed.

A Small Visual Audit

This is the heart of the article in one glance: a patch can be executable, or even pass a test, without being the most semantically faithful repair.

if (score < threshold) {
  print("review");
} else {
  print("pass");
}

if (score >= threshold) {
  print("review");
} else {
  print("pass");
}

if (score > threshold) {
  print("pass");
} else {
  print("review");
}

That is the distinction TundraTrace is trying to preserve: not just whether a model produced something runnable, but whether it repaired the right thing for the right reason.

The Plausible Wrong Fix

The dominant wrong repair was:

if (score >= threshold) {
  print("review");
} else {
  print("pass");
}

This looks like a reasonable local operator edit. But it still fails:

72 >= 80 is false

So the program still prints:

pass

That failure is informative. Several models localized the suspicious condition but did not reason through the runtime values deeply enough to choose the correct comparison.

Passing Can Still Be Suspicious

One output passed by swapping the printed strings:

if (score > threshold) {
  print("pass");
} else {
  print("review");
}

That passes the single test because the else branch executes. But it is less semantically direct than changing the condition.

Another output passed by changing the input:

val score = 92;

That makes the original condition true, but it changes the test data rather than repairing the branch logic.

The audit found these qualitative labels:

For qwen2.5-coder:14b, there was a candidate signal: it produced the clean condition flip in two evidence-assisted trials, but not in error-only trials. That is not enough to claim branch evidence helps in general. It is enough to justify more branch tasks with multiple tests.

Failure Modes

Across the 144 scored prompt-v2 rows:

The pattern is useful:

• syntax errors remain common for weaker models,
• wrong output dominates wrong-branch behavior,
• normalized scoring recovers wrapper/prose failures but not semantic errors,
• and parse validity, runtime success, expected-output match, and output discipline should be reported separately.

What This Means

The pilot supports a narrower and more useful claim than “provenance helps.”

TundraBench is already useful because it can separate:

• models that understand the repair but return unusable wrappers,
• models that follow the output protocol but repair the wrong thing,
• bug types that are too easy,
• bug types that are too hard,
• and tasks where runtime evidence may change the kind of repair attempted.

The strongest current findings are:

Limitations

The main validity risks are:

• trace leakage: evidence may reveal the fix too directly,
• prompt sensitivity: prompt changes can alter strict scores,
• small sample size: one task per bug type is not enough,
• local model constraints: Ollama model behavior may not generalize,
• single-test overfitting: wrong-output tasks can pass for the wrong reason.

What Comes Next

The next version should improve task design before scaling the benchmark.

Priorities:

1. Add multiple tests for wrong-output tasks.
2. Add more wrong-branch tasks with different structures.
3. Separate value_origin evidence from branch_behavior evidence.
4. Freeze prompt v2 for the next pilot.
5. Keep strict and normalized scoring.
6. Add more medium-difficulty tasks.
7. Add qualitative labels for suspicious passes and partial localization.

Good next task variants include:

• wrong boolean flag branch,
• nested record field used in a condition,
• branch condition involving a parsed string number,
• branch whose failing value comes through a helper function,
• wrong copied value,
• wrong field after two-step value movement,
• swapped function arguments with records,
• and accumulator bugs with simpler loop bodies.

Final Takeaway

Runtime traces are not magic. They can help, distract, leak hints, or be ignored.

The value of TundraTrace is that it makes those differences visible. Instead of asking whether an LLM “can debug,” it asks what kind of debugging behavior the model is showing: semantic repair, output compliance, pattern matching, trace use, or task overfitting.

That is the more interesting measurement problem.