DeepSeek Harness vs Pi: fix the same bug in both

What was actually checked

Keep the task and success criteria identical

Inputs are orders.csv, report.mjs and report.test.mjs. The deliverable is a focused report.mjs change, three passing assertions, the total 2000 and a short explanation. Do not let either agent change the input data or weaken the tests. Record the actual model/provider, tool configuration, additional prompts and elapsed time; if they differ, do not attribute the difference solely to the harness.

Why these checks belong in the exercise

Pi creator Mario Zechner’s 2025 design essay emphasizes visible context and a small tool set. Apply that idea here: record which of the three files each harness actually read and any extra instructions it loaded. Anthropic’s agent-evaluation article separates a conversation transcript from the final outcome. Accordingly, inspect both the tool log and the saved report.mjs with its unchanged tests; do not grade only the closing message. The 2025 essay provides design background; current Pi commands come from its current documentation.

Create a small project with a visible failure

Use Node.js 24 and a Bash-compatible terminal in a new disposable directory. These commands work as a Bash exercise on macOS/Linux; on Windows use Git Bash or WSL. The new harness-lab directory must not already exist. There are no npm dependencies, accounts, private data or production files in the fixture. The deliberately simple CSV contains no quoted commas; this exercise is not a general CSV parser.

create_harness_lab() {
mkdir harness-lab || return
cd harness-lab || return
cat > orders.csv <<'CSV'
id,status,cents
A,paid,1200
B,refunded,700
C,paid,800
D,pending,400
CSV
cat > report.mjs <<'JS'
export function totalPaid(csv) {
  const rows = csv.trim().split(/\r?\n/).slice(1);
  return rows.map(row => row.split(','))
    .filter(([, status]) => status !== 'pending')
    .reduce((sum, [, , cents]) => sum + Number(cents), 0);
}
JS
cat > report.test.mjs <<'JS'
const { default: assert } = await import('node:assert/strict');
const { readFileSync } = await import('node:fs');
const { default: test } = await import('node:test');
const { totalPaid } = await import('./report.mjs');
const csv = readFileSync(new URL('./orders.csv', import.meta.url), 'utf8');
test('only paid orders count', () => assert.equal(totalPaid(csv), 2000));
test('refunds alone count as zero', () => assert.equal(totalPaid('id,status,cents\nB,refunded,700\n'), 0));
test('an empty ledger counts as zero', () => assert.equal(totalPaid('id,status,cents\n'), 0));
JS
cp report.mjs report.original.mjs
node --test report.test.mjs
}
create_harness_lab

The first run should fail two assertions and pass one. The buggy code counts the 700-cent refunded order, producing 2700 instead of 2000. The rule is exact: count only rows whose status is paid; refunded and pending rows contribute zero. All values use integer cents.

Make two independent starting copies

Run the copy command before either agent edits a file. Use harness-lab for DSH and harness-lab-pi for Pi. Both contain the same failing implementation and original backup. A conversation branch is not a separate working tree: never run both agents against one directory during this comparison.

copy_harness_lab() {
  (cd .. && test ! -e harness-lab-pi && cp -R harness-lab harness-lab-pi)
}
copy_harness_lab

Walk through the task in DSH

1. 01

   Confirm the runtime and model

   Complete the linked Alpha quick start, using its exact source-build installation. Open Settings → Models and configure your own provider credentials there. Keep keys out of the task prompt. Record the actual model selected.

2. 02

   Select the project explicitly

   In the Web UI, choose workspace and add the absolute path to harness-lab. A fresh Web UI has no selected workspace, even when the process was launched from a project directory.

3. 03

   Inspect the effective permissions

   For this disposable local task, use the workspace-write preset where supported and inspect approval requests. That preset combines workspace-write sandbox mode with ask; the configured executor enforces the boundary. It does not imply that every file write creates a popup.

4. 04

   Run and inspect

   Send the shared prompt below. Observe the file reads, commands, edits and test results. If a request reaches outside the fixture or requests unrelated installation, stop and narrow the task before continuing.

Walk through the same task in Pi

Install Pi using its official quick start, then enter harness-lab-pi. Use /login and /model to configure and select a supported model; use the same model as DSH when possible. Start with the following reduced tool set to inspect the files. This command disables optional discovery so the comparison does not silently load unrelated extensions or project instructions.

npm install -g --ignore-scripts @earendil-works/pi-coding-agent
cd ../harness-lab-pi

pi --no-extensions --no-skills --no-prompt-templates --no-themes --no-context-files --tools read,grep,find,ls

Pi does not include a built-in permission system for files, processes, the network, or credentials. The allowlist limits tools exposed to the model, not the process’s OS permissions. For untrusted projects, use a container, virtual machine, or another verified operating-system boundary. The official isolation guide explains why tool-routing extensions alone do not isolate other host extensions.

pi -c --no-extensions --no-skills --no-prompt-templates --no-themes --no-context-files --tools read,edit,write,bash,grep,find,ls

Give both agents the same bounded task

In this disposable project, fix report.mjs so totalPaid counts only paid rows. Read orders.csv and report.test.mjs first, run node --test report.test.mjs and report the original failure. Do not edit orders.csv, report.test.mjs or report.original.mjs; do not install packages or access other directories. Make the smallest implementation change, rerun all three assertions, and calculate the total from orders.csv. Return the changed file, the observed test result, the total and any limitation. If a command fails for an environmental reason, distinguish that from a failing assertion.

Verify outside the agent’s final message

node --test report.test.mjs
node --input-type=module -e "import {readFileSync} from 'node:fs'; import {totalPaid} from './report.mjs'; console.log(totalPaid(readFileSync('orders.csv','utf8')))"
diff -u report.original.mjs report.mjs

Expect three passing tests and the separate total 2000. The diff should normally change the filter to status === 'paid'. A different implementation is acceptable if it meets the same rule and preserves the tests. diff returns exit code 1 when files differ; that is expected here, not a failed test. A missing Node executable or permission denial is an environment failure, not evidence that the calculation is wrong.

Compare what you actually had to do

Pause, resume and recover deliberately

Ask the agent to summarize the goal, files changed, exact command results and next step before stopping. In DSH, reopen the same stored session and confirm its workspace. In Pi, use pi -c in that project to continue the most recent session, or pi -r to choose one. /tree and /fork branch conversation history; they do not roll back files. Before undoing a failed attempt, preserve the attempted file, then restore only this fixture’s original implementation.

cp report.mjs report.attempt.mjs
cp report.original.mjs report.mjs
node --test report.test.mjs

After restoration, the original two test failures should return. This is an intentional recovery check. Keep report.attempt.mjs for comparison. In your real repository, use a clean branch/worktree and a reviewed file-specific restore; never apply this fixture’s overwrite command to unrelated work.

Choose from a completed review, not a slogan

DSH is worth trying when a shared Web workbench, visible session/tool state and composed workflows fit your day. Pi is worth trying when a terminal, explicit CLI startup and a small extensible core fit your existing tooling. Choose after completing the same task and reading the diff. Neither interface guarantees a correct patch or makes untrusted code safe on its own.

Harness/version:
Model/provider:
Workspace and starting files:
Effective tool/permission settings:
Additional prompts and approvals:
Tests: original __ failed; final __ passed
Observed total:
Files changed:
Elapsed time / reported usage (or not measured):
Recovery result:
Remaining limitations: