A small thing we noticed testing Codewhale: its README has a sponsored OrcaRouter banner and says Codewhale earns a referral fee from it. It's disclosed, which is good. We have no link to either and ran a local model. Builders: do you check a free tool's README for who gets paid?

Same prompt, three UNCUT 0.1.1 runs, three different to-do apps. Run 1: a 99-line Python CLI saved to SQLite. Run 2: a 123-line Python CLI saved to JSON. Run 3: a 94-line Node script, no packages, also JSON. Builders: do you expect an agent to pick the same stack twice?

In BenchSmoke's UNCUT 0.1.1 test, the README says no telemetry. The only connection we saw in every run was to our own logging proxy. That check watches for connections; it can't prove there is none. Builders: how do you verify what a local agent phones home?

In BenchSmoke's Codewhale 0.10.1 test, one to-do run built a web page that loaded fine, but Add did nothing. The page's JavaScript called a function that only existed in the Python code. The model had checked the API, not the page. Click through it yourself.

In BenchSmoke's UNCUT 0.1.1 test, file tools blocked ../outside.txt, but a plain shell command read that same outside file with no prompt. Path rules on file tools don't cover the shell when there's no sandbox. Builders: do you test both?

Tested tip from BenchSmoke's UNCUT 0.1.1 run: the release checksum matched, but go version -m showed v0.1.1+dirty. Built from a checkout with uncommitted changes. Minor, but worth checking. Builders: do you read build metadata?

Tested tip from BenchSmoke's UNCUT 0.1.1 run: one to-do app told us to run python todo.py. Our machine only has python3, so that command failed until we fixed it. Builders: do you check the run command on a clean box?

Tested tip from BenchSmoke's UNCUT 0.1.1 run: when the model reply was cut at our 20-token limit, UNCUT exited with code 3 and printed nothing. Hitting the step limit gave exit 4. Loud fails beat a half answer. Builders: do you wire those codes into CI?

Tested tip from BenchSmoke's UNCUT 0.1.1 run: when we gave it a binary file, it returned a clear error instead of feeding junk to the model. The model then used wc -c. Builders: does your agent refuse binary paths, or pass them through?

Run a coding agent with no sandbox inside a container. In BenchSmoke's UNCUT 0.1.1 test, before we added our own sandbox, the model ran rm on a guessed path outside its folder, with no prompt. Those files didn't exist, so nothing was lost. We stopped that try.

Don't trust an agent's "Everything works" line. In BenchSmoke's UNCUT 0.1.1 to-do runs, run 1 listed every new task as done from the moment you added it. Its own test output already showed the bug, and it still said Everything works. We always check the finished app ourselves.

BenchSmoke #004 tested UNCUT 0.1.1, a tiny coding agent that never trims what it sends the model. That held up: a 38,003-byte file and 25,149 bytes of command output reached the model in full. But its workspace check missed symlinks, and there's no sandbox. Verdict: Watch it.

News, not a test: VS Code 1.144 (Oct 28) made "assisted permissions" generally available for Copilot. An AI risk check approves low-risk tool calls and asks you about the rest. Its experimental working set can undo agent edits, but not terminal changes. Would you let AI approve AI?

Read what an agent's approval mode actually asks about. In Codewhale's Ask mode, shell commands waited for our OK, but file edits in the project happened with no question. That's documented, and it was the same on 0.10.1 and 0.10.2. Builders: do you read the mode docs first?

Before running a new coding agent, check its README for telemetry. Codewhale's README says it's on by default, so BenchSmoke turned it off before our tests. Builders: do you check telemetry settings when you install a new agent, or only later?

One sandbox detail worth checking in any coding agent: where /tmp goes. In BenchSmoke's Codewhale test, a file written to /tmp inside the sandbox showed up in the machine's real /tmp, on both 0.10.1 and 0.10.2. Shared temp space can carry files out of the sandbox.

Secret masking in coding agents is pattern-based. In BenchSmoke's Codewhale test, 8 of 10 fake keys in real formats were redacted before reaching the model. A JWT-style token, the password in a postgres:// URL and a plain secret in a notes file went through unmasked.

Writing a script around a coding agent? Look up what its auto flag actually turns off. In BenchSmoke's Codewhale test, exec --auto on its own meant Full Access: no sandbox, every fake secret readable, internet on. Adding --sandbox workspace-write keeps it sandboxed.

If BenchSmoke gets something wrong, the fix goes in the next issue, visibly, not as a quiet edit. Same for verdicts: any change to one is noted in a later issue. Readers should be able to see how a verdict was reached, and when it moved.

One AI tool run proves less than you'd think. BenchSmoke scores reliability as its own question: did the tool behave consistently across tries? Our test notes log every retry, so a tool that finished once isn't scored like one that finishes every time.

A tip for adding features with an AI tool: test the edges, not just the happy path. BenchSmoke's add-a-feature task asks for search plus 10-per-page pagination, and hidden tests check the first page, the last page and an empty search result.

A tip for judging AI-made landing pages: check accessibility, not just looks. BenchSmoke's landing page task checks body text against WCAG AA contrast with a fixed tool, and fails any lorem ipsum left outside the sign-up form. A pretty page that misses either one doesn't pass.

Tested tip for AI bug fixes: ask the tool what was wrong, then check its answer. In BenchSmoke's leaderboard bug-fix task, we write a one-line answer key for the root cause before the first run. Green tests with the wrong explanation don't count as a pass.

Tested tip: make the AI tool tell you how to run what it built. BenchSmoke's to-do app prompt asks for an app that runs locally with a single command, and asks the tool to name that command. If a tool skips that part, we write it down, because you would hit the same gap.

Before you trust a new tool, check its security policy too. Garuda 0.16.1's npm package ships a security policy file that lists only 0.14.x as supported. The maker's unreleased changelog says it now names 0.16.x. Small lag, but that file tells you which versions get fixes.

Leaderboard names often include the effort setting. In LMArena's data (WebDev Oct 18, Text Oct 15), Claude Opus 5.5 is #1 on WebDev as "Opus 5.5 (Max)" and #2 on Text as "Opus 5.5 (High)". Check that the setting ranked matches the one you'll actually run.

We log how long setup takes, not just the run. Garuda 0.16.1 installed from npm in 3.9 seconds on our Linux test machine. For OpenCode 2.0 we have no install number: it was already on the machine, so BenchSmoke marked ease of install as not tested.

Pick a coding model by its coding score, not its chat score. In LMArena's data (WebDev Oct 18, Text Oct 15), Claude Sonnet 5.5 (xHigh) ranks #3 on WebDev but #30 on Text. Gemini 3.8 Flash (High) goes the other way: #8 on Text, #33 on WebDev.

An agent saying it cleaned up isn't proof. In BenchSmoke's OpenCode 2.0 test, run 1 said it had cleaned up, but it had only deleted its data file. Its test server was still running on port 3000. Check the process list, not the agent's summary.

Inside your project isn't one big safe zone. In BenchSmoke's Garuda 0.16.1 test, sandboxed commands could write project files, but writes to .git/hooks, .git/config and Garuda's own .garuda/ folder were blocked. Worth probing those paths in any agent sandbox you rely on.