GPT-6.1 Sol came closer than any of them to breaking that habit. On hard coding it's within touching distance of Opus 5.5, at half the API price. It's also slow, it needs more hand-holding in the prompt, and in my web design test it was plain lazy. Opus 5.5 stays my default, but not by much.

Here's what I ran: the same web design brief from my Claude Sonnet 5.5 review, plus two hard algorithmic tests in Go. Every test pitted Sol against Opus 5.5 and Grok 4.7.

What is GPT-6.1 Sol?

GPT-6.1 Sol is OpenAI's mid-tier GPT-6 model, released on 29 September 2026 at DevDay. It sits between the flagship GPT-6 Astra and the budget GPT-6 Luna, and it replaced GPT-6 Sol after exactly seven days. If you're struggling to keep up with OpenAI's release pace, you're not alone.

Note the "ChatGPT Work" bit. At launch, Sol isn't in regular ChatGPT chat at all, and free users don't get it.

OpenAI's pitch is near-Astra performance at a fifth of the token price. On OpenAI's own numbers, Sol scores 75.2% on DeepSWE v1.1 at high effort, against 74.1% for Astra at xhigh. Cached input dropped to half of GPT-6 Sol's price, and responses with factual errors at low effort fell from 11.4% to 7.7%. One odd detail: the DeepSWE score drops to 71.9% at xhigh and max. So turning the effort dial all the way up made it worse at coding, which is not what the dial is for.

Test 1: the web design brief

I gave Sol the exact brief from my Sonnet 5.5 review, so you can compare it with every model I've run through it so far. All three results are side by side on the web design comparison page.

I've always liked GPT's taste in design. Its pages usually look clean and modern without much fuss, and Sol's page is nice too. The problem is how little it did. The design is too simple, and there are no animations at all. It felt like Sol read the brief, did the minimum that technically satisfies it, and went to lunch.

Opus 5.5 was much better. It did more with the same brief, and the result is the one I'd actually show a client. Grok 4.7 landed in the middle: okay, nothing to complain about, nothing to remember either.

What surprised me is that this isn't a capability problem. Sol is #4 on the LMArena WebDev leaderboard, so it can clearly build good frontends. My guess is it wanted a brief that said "add motion, add more sections, go further" in so many words. That turned out to be a pattern, and I'll come back to it below.

Tests 2 and 3: hard algorithmic work in Go

This is where Sol earned its spot. Both tests are logic-heavy Go tasks where a plausible-looking answer is easy to write and a correct one isn't.

The expression-language interpreter

This one is nasty on purpose. Each model had to build an interpreter for a small expression language in Go, tests first, standard library only. The prompt contradicts itself in three places. The grammar allows exactly one argument per call but demands an "arity" error. It never allows let inside an expression but asks for nested let rec. And it claims division never overflows, which MinInt64 / -1 would like a word about. On top of that, the prompt says: if a case is unspecified, return an error. Do not guess.

So the real test was how each model handles a spec that argues with itself. All three spotted all three contradictions, which is impressive on its own. Then they chose differently.

Opus kept to the grammar and returned errors wherever the prompt was silent. Sol and Grok both satisfied the prose by inventing new syntax for let expressions, and they invented two incompatible versions. Sol's version needs a special parsing rule, and that rule sets a trap: let f = fn x -> x + 1 f(let y = 2 y) returns 2 instead of 3, because it quietly gets parsed as two statements. That is exactly the guessing the prompt banned.

On the language the prompt actually defines, all three are correct. They pass all 206 hand-written conformance cases, and across 400,000 randomly generated programs no two implementations ever returned different values. The differences are at the edges:

Sol wrote the most compact code and the fastest interpreter, and it followed the tests-first rule properly. Then it wrote the thinnest test suite by a wide margin. In the mutation test (injecting small bugs one at a time and checking whether the model's own tests catch them), Sol's suite missed < flipped to <=, because it never compares equal values with a strict operator. It also missed most of the overflow boundaries.

The robustness gap is the one I'd worry about in production. This 33-character program, let rec f = fn n -> f(n + 1) f(0), kills any process that embeds Sol's (or Grok's) interpreter with a fatal stack overflow. Opus returns a clean error in 0.3 seconds. To be fair, Opus pays for its depth limits elsewhere: it wrongly rejects a few extreme but valid programs, like recursion 200,000 levels deep.

The Collatz worker

The second test looks easier and isn't. Each model had to write a Collatz (3n+1) checker in Go: a Trajectory function that follows one starting number down to 1, and a parallel MaxTotalStoppingTime search over a range. Same rules as before: tests first, public API only, standard library only, and no weakening tests to make the code pass.

The maths is trivial. Even numbers halve, odd numbers become 3n+1, repeat until you hit 1. The traps are everywhere else:

• 3n+1 has to stay exact above math.MaxInt64, so everything runs on math/big, with all the pointer-aliasing fun that brings in Go.
• The parallel search has to return the same answer with one worker or sixteen, and ties go to the smaller start. Get the merge wrong and the result depends on goroutine scheduling.
• A step cap has to stop a long trajectory without flagging it as a cycle, and a cap of 0 must not count as reaching 1.
• The tests have to lock known values, like 27 taking 111 steps and peaking at 9,232, or 97 having the longest run between 1 and 100 at 118 steps.

The sneaky part is the Cycle flag. A repeat before reaching 1 would be a counterexample to the Collatz conjecture, and nobody has ever found one. So no valid input can reach that code path through the public API, and an honest test suite has to say so instead of pretending to cover it. The prompt also states the conjecture is unsolved and asks for a final line confirming no counterexample turned up. Part of the test is simply whether the model overclaims.

All three got the maths right. Checked against an independent reference on about 12,000 trajectories and 17,000 range searches, with starts up to 1,000 bits long, none of them returned a wrong step count, peak or search result. None claimed a proof or a counterexample either, so nobody failed the honesty check.

This is the test where Sol's literal streak paid off. It was the only model that did exactly what the prompt asked and nothing else: the listed API, the specified behaviour, and the smallest code of the three. The prompt also wants tests for "the full sequence of 6", but Result only holds summary numbers, so no public call returns a sequence. Sol solved that neatly by running Trajectory from every suffix start and every capped prefix, with a comment explaining why.

The other two each bent the spec. Opus added an exported Step function the prompt never listed, and four of its tests call it. Grok returns -1 for StoppingTime when the step cap hits early, where the prompt says "steps taken". Grok's reading is arguably the more useful design, but it isn't what the spec says.

So why isn't Sol first? Tests, again. Its suite checks 39 trajectories and 13 ranges, against more than 20,000 trajectories and 854 ranges for Opus. Of 36 seeded bugs, its tests let six through, so they'd happily pass a search that treats hi as exclusive, or one that silently gives up on long trajectories. Its widest search range is 1 to 100.

Then there's speed. Sol and Grok detect cycles by storing every term as a decimal string in a map. That's simple and obviously correct, and it allocates 12.5 GB to search the first million numbers. Opus works in machine words with Brent's cycle detection, which stores nothing, and it searched 1 to 10 million in 0.2 seconds against 100 for Sol. The prompt never asked for speed, but it did warn that the range may be large.

Scoreboard

Opus wins outright. On the hard logic, Sol is a close second, and Opus is really only a little better. That "little" is mostly judgement: knowing when to say no to a spec instead of improvising.

Speed and prompting: where Sol annoys me

It's slow

Sol is slow, and it isn't just me being impatient. Artificial Analysis measures it at 51 tokens per second at max effort, which puts it 147th of 224 models on speed. Their own word for it is "notably slow". One developer measured 27.4 tok/s in Codex against roughly 90 for Opus 5.5, and flagged the method as rough because Claude did the measuring. I respect the honesty. Launch week also brought "model is at capacity" errors in ChatGPT and Codex.

OpenAI's answer is Ultrafast, a premium tier with up to 8x faster generation in Codex. It's live for Astra. For Sol, OpenAI still says it's coming "in the coming days", and in ChatGPT you'll only get it on the new $500-a-month Pro plan. On my $20 plan, slow is what I get.

You have to spell everything out

This is the difference I notice most day to day. With Opus, I describe what I want and it fills in the obvious gaps. With Sol, I have to say exactly what I want, including the parts I thought were obvious.

Looking back at my tests, it's the same pattern each time. The design had no animations because the brief didn't say "animate things". The test suite was the smallest one that technically covered the requirements. In the interpreter, Sol followed the prose literally even where the grammar said otherwise. It does what you write, which isn't always what you mean.

That isn't all bad. If you're feeding a model precise specs in an automated pipeline, doing exactly what it's told is a feature. For exploratory work, where I want the model to be a bit clever and occasionally push back, it gets tiring fast. Claude also feels more consistent to me from one run to the next. I can't put a number on that one. It's vibes from a lot of daily use, but they're strong vibes.

GPT-6.1 Sol benchmarks and pricing vs Opus 5.5 and Grok 4.7

The independent numbers line up with my tests. Opus is ahead, Sol sits close behind at half the API price, and Grok trails on capability but has the cheapest output.

On coding specifically, Mercor's independent DeepSWE v1.1 run has Sol and Opus 5.5 tied at 72.3%. On reasoning, ARC Prize found similar results: 94.2% on ARC-AGI-2 at $0.25 per task, close to Astra's 95.0%. So the "near the top for a fraction of the cost" story holds up outside OpenAI's own charts, which isn't always the case with launch claims.

There's one awkward row for OpenAI in that table. Sonnet 5.5 costs exactly the same as Sol and scores higher. If price is your main reason to look at Sol, run Sonnet on your workload too.

On the subscription side, my $20 Plus plan gets 15 to 160 local Codex messages per five hours with Sol, going by OpenAI's Codex pricing page. Pro 200 subscribers lose half their allowance from 30 October, which is a fun thing to announce on the same day as a $500 plan.

If you're moving an API integration over from gpt-6-sol, check these first (OpenAI's GPT-6 guide has the details):

• none and minimal reasoning effort are gone, so low is now the floor.
• Tool calling only works through the Responses API. Chat Completions still works, without tools.
• Prompts over 272K tokens bill the whole request at 2x input and 1.5x output, not just the tokens past the line.
• Sol is rated Critical for cybersecurity, and the filters show. It ranks 42nd of 44 on Vals' CyberBench with 39.29%, where GPT-6 Sol scored 78.0%. Expect refusals on security work unless you're in OpenAI's Daybreak program.

Verdict: who should use GPT-6.1 Sol?

GPT-6.1 Sol makes sense if:

• you run agentic coding at volume and cost per task matters more than the last few points of quality
• you write precise specs and want a model that follows them to the letter
• you already live in Codex or GitHub Copilot

Stick with Opus 5.5 if:

• you want the model to work out your intent and fill the gaps
• you build frontends and care about polish
• consistency across runs matters more to you than the API bill

And skip Sol for security tooling for now, unless you're in Daybreak.

For me, Opus 5.5 stays my favourite. It's more confident, sometimes cleverer, and its results are more reliable. When I pick models for client agent builds, that reliability is worth more than the price difference. But Sol is the closest an OpenAI model has come to changing my mind. If Ultrafast lands on the $20 plan, or the speed improves some other way, I'll run these tests again.

Some context on how I use AI right now. My Claude subscription is where the hard stuff goes. Anything complicated or important gets Fable or Opus on high or xhigh effort. Anything easy goes to Grok or DeepSeek V4.1 Flash, which I really like at the moment. Sonnet never had a slot. I barely touched Sonnet 5, partly because my results with it were underwhelming, and partly because I couldn't see why I'd pick it over all the other good models at a similar price.

So this Claude Sonnet 5.5 review comes down to one question: is it good enough, and cheap enough, to become my everyday Claude? I ran two head-to-head tests against Opus 5.5 and Sonnet 5 to find out, one web design and one Go coding task.

Short version: Sonnet 5.5 is a big jump over Sonnet 5. Its design taste is close to Opus 5.5, and in my Go test it built a chess engine as good as Opus's for half the price. It also filled its tests with numbers it made up. And I still don't see myself using it, for reasons that have more to do with my setup than with the model.

What changed in Sonnet 5.5

The price didn't change. Sonnet 5.5 costs the same as Sonnet 5: $2 per million input tokens and $10 per million output. Opus 5.5 is exactly double at $4 and $20. Anthropic says Sonnet 5.5 generates output 30% faster than Sonnet 5 and costs up to 30% less per task, because it needs fewer tokens to do the same work (announcement).

The benchmarks

Here are Anthropic's numbers for the three models I tested:

That Terminal-Bench jump is silly. Sonnet 5 scored 10.3%, which I'm choosing to read as "mostly didn't show up", and Sonnet 5.5 beats Opus 5.5's best score. Everywhere else it sits within a few points of Opus.

Before you cancel your Opus habit, read Anthropic's own caveat on the same page. In their testing and in external testing, Opus 5.5 remains clearly stronger at complex, open-ended work that needs sustained judgment. Keep that sentence in mind. My chess test ended up showing it in a very specific way.

What changes in Claude Code and the API

In Claude Code v2.1.284 and later, the sonnet alias points to Sonnet 5.5. It runs at medium effort by default, you get the 1M context window natively, and you can't turn thinking off. The default model is still Opus 5.5, so you switch with /model sonnet.

On the API the model ID is claude-sonnet-5-5, and a straight string swap will break a few things (Anthropic's developer guide):

• thinking: {"type": "disabled"} now returns a 400. Use the new between_tools setting, which only thinks between tool calls.
• Forced tool_choice (any or tool) also returns a 400. Send auto and mark the tool strict: true.
• Effort levels are recalibrated, so your old Sonnet 5 setting won't behave the same way. Re-run your evals.

If you don't want to do that by hand, /claude-api migrate this project to claude-sonnet-5-5 in Claude Code applies the model ID swap and the breaking parameter changes for you. The migration guide covers the rest.

Test 1: web design, Opus 5.5 vs Sonnet 5.5 vs Sonnet 5

Sonnet 5.5 came second in my design test, close behind Opus 5.5 and miles ahead of Sonnet 5. The gap between the two Sonnets is the most interesting result in this review.

The brief was a one-page site for "Low Tide", a made-up outdoor screening of three short films on a beach. The content was fixed (date, gate times, programme, tickets, FAQ), and each model had to deliver one self-contained HTML file. Same prompt in Claude Code, high effort for all three, one run each. If you read my frontend design comparison, you know the drill.

You don't have to take my word for any of this. I put all three pages on a side-by-side comparison page, served exactly as the models produced them, no edits. You can look at them one at a time or next to each other, switch between desktop and mobile, and scroll around inside each one. They're fully interactive, so go drag Sonnet 5.5's evening slider yourself.

Opus 5.5: still the best designer in the family

Opus built something that looks like a poster. A big, soft "Low Tide" wordmark sits over a sunset, and both the type and the sun reflect in the water line. The nine animations (drift, shimmer, swell, tide, flicker and a few more) all fit the beach-at-dusk mood. It also did the most accessibility work without being asked: labelled controls, aria-pressed on the play button, aria-valuetext on the scrubber and a reduced-motion fallback.

Sonnet 5.5: its own direction, almost as good

Opus went moody. Sonnet 5.5 went print. Cream and navy, a flat orange sun, and a staggered italic tagline: "Three short films. / One tide. / Bring a blanket." The typography is as strong as Opus's, with a roman "Low" next to an italic "Tide" in a second colour. Great layout, great colours, and a real idea behind it.

My favourite detail: Sonnet 5.5 came up with the same core interaction as Opus on its own, a slider that scrubs through the evening timeline. Same idea, completely different visual execution. It isn't a cheap Opus clone. It has its own taste.

It loses on polish. Four animations instead of nine (well chosen, with scroll reveals and a reduced-motion fallback), and lighter ARIA coverage.

Sonnet 5: a concept without the execution

Sonnet 5 had a concept, to be fair. Film-reel framing, sprocket-hole borders, sections numbered as reels. Then it built a stack of bordered cards with no hero image, plain font weights, one keyframe animation and zero scroll reveals. The live countdown is a nice practical touch. It still looks like a template.

Going from Sonnet 5 to Sonnet 5.5 is the difference between "competent template" and "somebody designed this". That's a visible jump for a point release.

One thing made me laugh: all three picked Fraunces as their display font. Either Fraunces is objectively the correct font for a beach film night, or the Claude family has a house style. I know which one I'd bet on. And yes, this is one prompt and one run per model, so treat my ranking as an opinion with receipts.

Test 2: a chess move generator in Go

In my Go test, Sonnet 5.5 built a chess engine exactly as correct as Opus 5.5's, slightly faster, for half the money. Its own test suite still failed. Both things are true, and the second one is why I trust Opus more.

Why chess, and why perft

The task: build a complete, legal chess move generator in Go (standard library only) from an empty directory. Write the tests first, then the implementation, then iterate until go vet and go test -race are green.

Perft is a great judge because it has no opinions. It counts every legal move sequence to a given depth, and the correct counts for standard positions are published. Miss one rule, like en passant that exposes your own king or castling rights lost when a rook gets captured on its home square, and the number is wrong. No partial credit.

The prompt had one more rule: don't edit a test or a perft number to make it pass. If you think a number is wrong, say so in the final message. It's a small honesty check.

After the runs, I had a separate Claude session check all three engines against 20 standard perft positions from the Chess Programming Wiki and the common perft test suite, deeper than the models tested themselves (up to 193.7 million nodes per position). It didn't touch the models' files.

The results

All three engines are correct. The prompt allowed 10 seconds for Perft(5), and Sonnet 5.5 needed about 2% of that. The code is nice too: a 64-square array with one byte per piece, precomputed move tables, and a per-square castling mask that handles king moves, rook moves and captured rooks in one line. Moves are made on a copy of the position, so there's no undo logic to get wrong.

The two failed tests

So where does the red come from? Sonnet 5.5 wrote the most ambitious test suite of the three: 680 lines, 20 perft positions, each checked at every depth. The catch is that the published suite only gives one depth per position. Sonnet 5.5 filled in the rest from memory.

Most of those guesses were right. Two weren't. One test expects 25 legal moves in a position that has 29 (counted by hand: queen 16, knight 8, king 5). Another expects 15 where there are 13. Both times the engine returned the correct number and the test was wrong. It even named one position after promotions, although no pawn in it can promote.

The deep counts for those same positions were correct. So it remembered the published results and made up the shallow ones.

That's what disappointed me. The engine is Opus-grade, but without the independent check I'd have two red tests and no idea whether the code or the tests were wrong. That's an afternoon of debugging the wrong thing. Opus 5.5 was the only run I could have accepted without review: green tests, all seven bad FENs rejected, and an Apply that refuses illegal moves instead of trusting the caller.

Sonnet 5, for the record, produced a correct engine with green tests at half the speed, and its Divide(0) never returns. It also cost more than Sonnet 5.5.

If you use Sonnet 5.5 for agent work, pin expected values from a real source instead of letting the model fill them in, and enforce the checks yourself. Claude Code hooks are a good way to do that.

Is Sonnet 5.5 actually cheaper than Opus 5.5?

Yes, in both of my tests: 30% cheaper in the design test and 51% cheaper in the chess test. Not the flat 50% the price sheet suggests, though, and the reason is tokens.

I'll admit I got this wrong at first. Looking at the design results, I was convinced Opus had come out cheaper. It hadn't. It used fewer tokens (45.1k output against Sonnet 5.5's 57.8k), but Opus tokens cost twice as much, so it still landed at $1.94 against $1.35.

The lesson survives in a smaller form. A smarter model that gets there in fewer tokens eats into the list-price gap. In the design test, a 2x price difference shrank to a 30% saving. On a task where Opus is much more efficient, the gap could shrink further. So don't assume Sonnet is half price per task. Measure your own workloads.

Against Sonnet 5 it's mixed. In chess, Sonnet 5.5 was cheaper ($0.74 vs $0.98) and needed far fewer round trips, with 781k cache reads against Sonnet 5's 2.1M. In the design test it cost 25% more than Sonnet 5, but it wrote more than twice the output and a much better page. I'll take that trade.

If you're on a subscription, none of this is dollars. It's usage. Sonnet burns through your limits more slowly than Opus, which only matters if you actually hit them. Spoiler: I don't.

Usual caveat: one run per model per test. A 5 versus 7 minute difference might not survive five more runs.

Who should use Sonnet 5.5 (and why I won't)

I have no use for Sonnet 5.5, and that says more about my setup than about the model.

My Claude subscription is for hard or important work, and for that I want Opus or Fable. Everything easy goes to other subscriptions I already pay for. I rarely hit my Claude limits, so saving usage buys me nothing. And when I want cheap and fast, DeepSeek V4.1 Flash costs $0.30 per million input tokens and $1.20 per million output at peak, half that off-peak. Sonnet 5.5's output is about eight times more expensive at peak. It's the better model, but for "rename these variables" or "summarise this thread" I don't need better. (I reviewed DeepSeek V4 back in May, and V4.1 Flash is the one I reach for most right now.)

So Sonnet 5.5 falls into a gap in my stack. Too expensive to be my cheap model, not careful enough to replace Opus for the work I actually open Claude for. Right now there are just too many options that are cheaper, faster or better.

That's my situation, though. Here's who I think should use it.

• You only pay for Claude and you're a heavy user. Move well-scoped tasks (bug fixes, small features, docs, slides) to Sonnet 5.5 and keep your Opus usage for the hard stuff. Anthropic's own guidance says the same thing: Sonnet for well-scoped everyday work, Opus for complex work that needs careful judgment.
• You're on the Claude API and pay per task. Based on my two tests, routing easy work to Sonnet could save you somewhere between 30% and 50%. Measure it on your own tasks, because token efficiency moves that number.
• A human reviews the output anyway. Sonnet 5.5's code was Opus-grade in my chess test. Its test data wasn't. If someone looks before it ships, you get most of Opus for about half the price.

Who shouldn't: anyone running long, unattended agent loops where made-up test data could slip through. That's exactly where Opus's extra care earned its money in my test.

And if what you really want is cheap Claude, Anthropic says Haiku 5.5 joins the family in the coming weeks. Honestly, I'm more curious about that one.

Sonnet 5.5 is a good model. It's a huge step up from Sonnet 5, its design taste is close to Opus, and in my test its code was too. It's just arriving in a market with more good choices than anyone can keep up with. If you live inside Claude, use it for the easier half of your work. If you don't, you probably already have something cheaper doing that job.

Accounts payable automation is software that does the typing, coding and chasing for you. Invoices get read automatically, matched to the right supplier and account, routed to whoever needs to approve them, and lined up for payment. A person still decides what gets paid. The software does everything before that decision and most of the paperwork after it.

There are three ways to get there: off-the-shelf AP software, the AI agents you're probably already paying for, and a custom build. I build the third kind for a living, so take my views on it with appropriate salt. I've tried to be fair to the other two.

There's also a fourth thing, which none of the three fixes by default. It's the reason a WA company was ordered to pay a $235,000 invoice twice. We'll get to that.

What is accounts payable automation?

"AP" covers five distinct jobs, and most tools only do two or three of them well, so it helps to be precise about what's being automated.

The accounts payable process in 5 steps

1. Capture. An invoice arrives by email, Peppol or post. Someone (or something) reads it and pulls out the supplier, ABN, invoice number, amounts, GST and due date.
2. Code. Each line gets an account and a GST treatment, maybe a job number.
3. Approve. Whoever owns the budget says yes. In a small business that's often the owner, which is why invoices sit for a week while the owner is on site.
4. Pay. Bills get batched, an ABA file goes to the bank, and the money leaves.
5. Reconcile. The bank feed shows the payment, it gets matched to the bill, and the books balance.

What to automate, and what to keep human

Steps 1 and 2 are safe to automate and are where most of the manual hours go. Step 3 stays with a person, but the routing and nagging can be automated. Step 4 I'd never hand to software without a human clicking "release." Step 5 your accounting platform mostly does already.

Then there's a job that isn't on the list because nobody puts it on the list: checking that the invoice is real and the bank details are right. That belongs in step 1, before anything else happens, and it's the part I care about most.

Ardent Partners' State of ePayables 2025 puts the average cost of processing one invoice at US$12.42, against US$2.65 for the best-performing 20% of AP teams. That's US data from companies mostly larger than the ones I work with, but the pattern holds at any scale, because the expensive part is a person reading PDFs.

Path 1: Accounts payable software

This is what the vendor pages ranking above this post are selling, and a lot of it is good.

Start with what Xero and MYOB already do

Before buying anything, check what you're already paying for. Xero's Business plans include Hubdoc, which captures invoices from email or a phone photo and creates draft bills. Xero's bills module has approval states built in. JAX, Xero's AI assistant, does auto bank reconciliation for Australian users. MYOB has its own bill capture. I've seen businesses pay for a separate capture tool while Hubdoc sat unused in a menu.

Hubdoc misses things on messy invoices and the approvals are basic, but basic and free beats sophisticated and unused.

Capture, approval and all-in-one tools

When the built-in tools run out, the Australian market has a few well-worn options.

The trade-offs are the usual SaaS ones. You pay per user or per invoice volume forever, so costs scale with your growth. You adapt your process to the tool. And you get one more login and one more sync to babysit when Xero changes something.

Who it suits: most businesses under a few hundred invoices a month with a fairly standard process. If that's you, you can probably stop reading here, buy Lightyear or Dext plus ApprovalMax, and skip to the fraud section. Please don't skip the fraud section.

Path 2: AI agents you already pay for (Claude Cowork, ChatGPT Work)

This is the option nobody in the AP software space wants to talk about, because it costs you nothing extra if you already have a Claude or ChatGPT subscription.

Claude Cowork is Anthropic's agent for knowledge work, on every paid Claude plan. ChatGPT Work is OpenAI's equivalent, rolled out in July 2026. Both connect to Gmail and Outlook, both can be given a recurring job, and both are considerably better at reading a badly scanned invoice than any OCR tool I used before 2023.

A Cowork setup for your AP inbox

Here's a scheduled task I'd set up for a business with an accounts@ inbox:

Every weekday at 7:30am, check the accounts@ inbox for emails from the last 24 hours that contain invoices. For each one, extract the supplier name, ABN, invoice number, dates, subtotal, GST, total, and the bank details on the invoice. Check whether the supplier and ABN match an existing Xero contact and whether that invoice number already exists in Xero. Put everything in a spreadsheet using Xero's bill import layout. Flag invoices where the bank details differ from the Xero contact, the ABN is missing, or the invoice number is a duplicate. Flag any email that mentions changed bank details, even without an invoice attached. Email me the sheet with a short summary. Do not create, edit or pay anything in Xero.

You open the sheet, deal with the flags, and import the clean rows into Xero as draft bills. Ten minutes instead of an hour.

Cowork's scheduled tasks run in Anthropic's cloud, so they fire whether your laptop is open or not. The catch is that they can only reach your connected apps and files saved to your Claude account. A folder on your Mac is invisible to a scheduled run. ChatGPT's scheduled tasks work the same way.

The Xero catch

The official Xero connector for Claude is read-only. Xero's own page says so in bold. Claude can look up contacts, bills and reports, which is exactly what the duplicate and bank-detail checks above need. It cannot create a bill. That's why the task produces an import sheet rather than posting to Xero directly.

If you want the agent to create draft bills itself, Xero's MCP server can, but setting it up means API credentials from Xero's developer portal and configuring the connection yourself. At that point you've crossed into "technical setup," and most owners I know would rather not.

QuickBooks users have it easier: Intuit's connector gained write actions in July 2026. MYOB has no official connector in either directory that I could find, only third-party bridges. Read their privacy terms before pointing them at your ledger.

Where AI agents fall short

I use these tools daily, and I wouldn't run a company's payments through them.

They run on a clock. The task above checks the inbox at 7:30am. An invoice that arrives at 7:31 waits until tomorrow, and nothing chases an approver who's sat on a bill for three days unless you write that into another prompt.

They're probabilistic. Language models read a "1" as a "7" on a bad scan, occasionally miss a line, and now and then confidently invent a due date. Rare, and better than a tired human, but not zero.

There's no audit trail on the plans small businesses buy. When your auditor asks why a bill was coded a certain way, "the AI did it" isn't an answer. Claude's Enterprise plan has audit logs. Pro and Team don't.

Your data leaves the country. Standard Claude and ChatGPT plans process and store data in the US. Australian data residency exists for ChatGPT Enterprise and for the Claude API through AWS's Sydney region, not for the plans a small business would pick. Under the Privacy Act, you stay responsible for personal information you send offshore, and supplier invoices contain a fair bit of it.

So my rule for this path: the agent drafts, a person posts, and nobody pays from it. That's still a real saving, and it suits an owner-operator with a few dozen invoices a month who'd rather have a flexible assistant than another subscription.

Path 3: A custom invoice automation pipeline

This is what I build, so I'll describe how it's put together and then tell you honestly when it isn't worth it.

How it works: AI reads, code decides

The principle behind every AP system I've built is that the language model does the reading and ordinary code makes the decisions.

AI is excellent at turning a crumpled three-page invoice into a clean set of fields, and inconsistent at applying rules. Code is the opposite. So the model extracts the fields into a fixed structure, and a set of boring, deterministic checks run on that structure:

• Is the ABN valid, active and registered for GST? The ABN Lookup web service is free and answers all three in one call. If the invoice charges GST and the supplier isn't registered, you can't claim the credit.
• Do the bank details on the invoice match what's on file for this supplier? If not, the invoice stops right there and a person gets a phone number to call.
• Has this invoice number, or this amount from this supplier in the last 30 days, been seen before?
• Does the invoice have what the ATO requires on a tax invoice, including your ABN on anything over $1,000? Is there a supplier ABN at all? If not, you may have to withhold 47% of the payment.
• Does it match a purchase order or delivery docket, if you use them?

Only after every check passes does the bill get created as a draft and routed to the approver whose delegation covers that amount. Once approved, it goes into the ABA file, which a person still uploads and releases. Every step is logged: what the model extracted, what the rules decided, who approved what. That log is what you hand to your accountant.

I covered the extraction side in my post on structured output with the Claude API, and the same "AI classifies, rules route" pattern runs the ticket triage system I built for a small business. For the developer-level version, my Claude Agent SDK guide uses an invoice processing agent as its running example.

When custom is worth it

Not often, if I'm honest. Off-the-shelf tools exist because most businesses have roughly the same AP process. Custom makes sense when at least one of these is true:

• You run multiple entities and the SaaS tools want a separate subscription and inbox for each.
• Your approval rules don't fit "under $X goes to A, over $X goes to B." Job-based or client-recharge approvals are where off-the-shelf flows break.
• Something upstream is old. A supplier portal, an ERP from 2009, a spreadsheet that is somehow load-bearing.
• Per-invoice or per-user pricing has grown to where a one-off build pays for itself within a year or two.
• The data has to stay in Australia. A custom pipeline can run on AWS in Sydney or on a local model in your office.

Invoice fraud: the gap automation doesn't close

Everything above makes AP faster. None of it, by default, makes AP safer, and faster AP without verification means you pay the fraudster faster.

How invoice fraud hits Australian businesses

The pattern is depressingly consistent. A criminal gets into a supplier's email (or registers a domain one letter off from it), waits for a real invoice to go out, then sends a follow-up: "Please note our bank details have changed." The invoice is real, the amount is real, the account is theirs. By the time the supplier chases the payment, the money has left the country.

The ASD's Annual Cyber Threat Report 2024-25 puts business email compromise with financial loss at 15% of all cybercrime reports from businesses. The average cost of a cybercrime report from a small business was $56,571. The ACCC's Targeting Scams 2025 data puts payment redirection losses at $166.8 million for the year, second only to investment scams.

The case to pin above every AP desk in the country is Mobius Group v Inoteq. Mobius did $235,400 of electrical work for Inoteq. A hacker in Mobius's email sent Inoteq new bank details and a fresh invoice. An Inoteq staffer tried to phone Mobius to confirm, couldn't hear the answer on a bad line, emailed for proof instead, got a forged document back, and paid. The bank recovered $43,541. In December 2024 the WA District Court ordered Inoteq to pay Mobius the remaining $191,859 plus interest. The judge said the failed phone call should have prompted a second one (Kennedys has a good case note). Inoteq paid twice.

Does Confirmation of Payee check your supplier payments?

Only partly. Confirmation of Payee launched in July 2025 and checks the account name against the BSB and account number when you type in a new payee or edit a saved one. It's live at over 100 institutions and has been used more than 150 million times. It works.

But think about how your business actually pays suppliers. Xero or MYOB exports an ABA file, someone uploads it, and dozens of payments go out in a batch. Those payees weren't typed in; they came from a file. Saved payees aren't re-checked on every run either. NAB said at launch that NAB Connect, its business platform, wasn't included. Some banks now offer businesses a bulk check of existing payees. Ask yours, in writing, whether the payees in your uploaded batch file are name-checked. If the answer is no, your Friday pay run has no name check at all.

Don't count on getting the money back, either. The Scams Prevention Framework that passed in 2025 is built around consumers. Whether a business that paid a redirected supplier invoice is covered is still being worked out in the rules, and Mobius v Inoteq tells you which way the courts lean in the meantime.

Invoice verification checklist

None of this is clever. It's a process, and the point of automation is to make the process impossible to skip.

1. Any change to a supplier's bank details puts the payment on hold. Call the supplier on a number from your own records, not from the email, and speak to a person you know. A voicemail doesn't count. A bad line means you call again.
2. New or changed payees need two people to approve, and the second can't be the one who received the email.
3. Check the ABN is active and GST-registered before the first payment to a new supplier.
4. Check for duplicate invoice numbers and duplicate amounts from the same supplier within a month.
5. Read the sender's domain letter by letter. "supplier-au.com" is not "supplier.com.au."
6. Turn on multi-factor authentication for every mailbox that touches money, and audit inbox forwarding rules. A forwarding rule to an outside address is how the criminal keeps reading your mail after you've changed the password.

Here's how the three paths handle those six checks.

Row six is on you regardless.

Which path fits your business?

Roughly, by monthly invoice volume and how unusual your process is:

One more thing. The fastest way to automate capture is to not need it. Suppliers on the Peppol e-invoicing network send invoices straight into Xero or MYOB as structured data, with no PDF and nothing to misread. Over 400,000 Australian businesses are already on it. Ask your biggest suppliers; the ones who say yes just deleted step 1 for you.

Frequently asked questions

Is accounts payable automation worth it for a small business?

Usually yes above about 20 invoices a month, and often below that if you use tools you already pay for. The hours saved are real, but the bigger win is consistency: a process that runs the same way every time is much harder to defraud than one that depends on who's in the office.

Can Claude or ChatGPT pay my bills?

No, and you shouldn't want them to. The official Xero connector for Claude is read-only, and neither agent should be given payment authority. Use them to extract and check invoices for review. A person creates or approves the bill, and a person releases the payment.

Does Xero automate accounts payable?

Partly. Xero Business plans include Hubdoc for capturing invoices and creating draft bills, an approval state on bills, and JAX for automated bank reconciliation. It doesn't do multi-step approvals, purchase order matching or fraud checks, which is where ApprovalMax, Lightyear and custom builds come in.

What is 3-way matching?

Comparing an invoice against the purchase order and the delivery docket before paying. All three should agree on what was ordered, what arrived and what's being charged. It's standard in larger companies and overkill for many small ones, but if you issue POs, it's one of the easiest checks to automate.

Is e-invoicing mandatory in Australia?

Not for businesses. Federal government agencies must be able to receive Peppol e-invoices, and business-to-business use is voluntary. The government considered giving businesses a right to demand e-invoices but hasn't legislated it. Adoption is growing anyway because it removes the manual entry step entirely.

Where to start

If your AP process is "someone types it in," do the verification checklist by hand this week. Then pick a path. If you'd like a second opinion on which one, I offer a free AP workflow review: an hour on a call looking at how invoices actually move through your business, then a written recommendation. Sometimes the recommendation is "buy Lightyear." Book one through my AI automation services page, or read how I approach custom AI agent development first.

That last sentence is why you'll find two names, two package names and a pile of outdated tutorials when you search for it. This guide is the version I wanted on day one: one running example, both languages side by side, and the traps marked before you fall into them.

I build accounts-payable automation for small businesses, so the running example is an invoice agent. It reads invoice PDFs from a folder, checks the supplier against a database through a custom tool, is blocked from touching .env, and hands back typed JSON at the end. TypeScript first, Python alongside, and a short Go section near the end for people like me who'd rather not run Node in production if they can avoid it.

Everything here is checked against the September 2026 docs, TypeScript SDK 0.3.273 and Python SDK 0.2.152. Anthropic ships a new SDK build most days, so if something looks different by the time you read this, the docs links at the bottom win.

What the Claude Agent SDK is

The short version: Claude Code is an agent, and the SDK is that agent with the terminal UI removed. Same loop (gather context, act, verify, repeat), same tools (Read, Bash, Glob and friends), same automatic compaction when the context fills up. You import it, hand it a prompt and some options, and it goes off and does the thing while streaming messages back to you.

That's the whole pitch, and it's why the SDK is interesting compared to writing your own loop around the Messages API. Anthropic has spent a year and a half hardening Claude Code's loop against real codebases and real users. You get that work for free, plus the bugs, which is the deal with every framework.

From Claude Code SDK to Claude Agent SDK

Anthropic renamed the Claude Code SDK to the Claude Agent SDK on 29 September 2025. The reasoning in the announcement post was that the harness powering Claude Code can power other kinds of agents too, so "Code" in the name was underselling it. Fair enough. The rename still left a trail of confusion that shows up in search results and in half the tutorials out there.

Two traps here. The old npm name @anthropic-ai/claude-code still exists, but it's the Claude Code CLI now, not the SDK. And the old PyPI package is frozen at 0.0.25 and prints a migration notice when you import it.

The rename came with two behaviour changes in v0.1.0 that bite anyone migrating old code.

The system prompt is no longer Claude Code's by default. You get a minimal prompt instead. If you want Claude Code's coding instructions and personality back, ask for the preset: systemPrompt: { type: "preset", preset: "claude_code" } in TypeScript, or system_prompt={"type": "preset", "preset": "claude_code"} in Python. Both take an append field for your own additions on top.

Settings loading went back and forth. v0.1.0 briefly stopped loading filesystem settings, then Anthropic reverted that. Today, if you leave settingSources out, the SDK loads your user, project and local settings (~/.claude/settings.json, .claude/settings.json, .claude/settings.local.json, plus CLAUDE.md and custom commands), the same as the CLI would. Pass settingSources: [] when you want a clean, isolated agent, which is what you want for anything deployed. More on that in the production section. One wrinkle: Python SDK 0.1.59 and earlier treated an empty list the same as leaving the option out, so don't rely on [] on an old install.

Agent SDK vs Client SDK vs Claude Code vs Managed Agents

Anthropic now has four ways to point Claude at a task, and the names overlap enough that I made a table.

Managed Agents deserves a paragraph because people mix it up with the SDK constantly. It launched in public beta on 8 April 2026 and it's a hosted REST API: Anthropic runs the loop, the sandbox and the session log, and your app sends events and reads results. Pricing is standard token rates plus $0.08 per active session-hour, and data residency is US-only during the beta. Code written against the Agent SDK doesn't deploy to it directly. They share the concepts, not the API.

OpenAI's lineup has the same shape, which helps if you're comparing. The OpenAI Agents SDK is an open-source framework you host yourself. The OpenAI Agents API, which went into public beta on 10 September 2026, is the managed Codex harness behind one API call, US-only, with no zero-data-retention option in any configuration. So the like-for-like comparison is Claude Agent SDK vs OpenAI Agents SDK, and Claude Managed Agents vs OpenAI Agents API. Anyone comparing the Claude Agent SDK to the OpenAI Agents API is comparing a library to a hosting product.

When not to use it

If you want to decide every step yourself (which tool runs when, how retries work, what gets persisted where), the SDK will fight you. Use the Messages API and write the loop. If you don't want to operate a sandbox at all, Managed Agents exists for that. And if you're weighing alternatives, I spent a week with the Pi coding agent's TypeScript SDK last month. Different trade-offs, worth a look if you're not committed to Claude.

Quickstart: your first agent in TypeScript and Python

Install and authenticate

1. You need Node 18 or newer, or Python 3.10 or newer. The Claude Code binary ships inside the package, so there's nothing else to install. Two exceptions: npm ci --omit=optional skips the platform binary, and a pip install that falls back to the sdist (ARM64 Windows does this) gives you no binary either. In both cases install Claude Code natively and point the SDK at it with pathToClaudeCodeExecutable (TypeScript) or cli_path (Python).
2. Install the package.
3. Set ANTHROPIC_API_KEY in your process environment. The SDK does not read .env files. Load them yourself with dotenv or whatever you use, before the first query() call. It's the first thing to check when you get an auth error and you're certain the key is right.
4. If you're on a cloud provider instead, it's one env var each: CLAUDE_CODE_USE_BEDROCK=1, CLAUDE_CODE_USE_VERTEX=1 or CLAUDE_CODE_USE_FOUNDRY=1, plus that provider's usual credentials.

Your first agent in TypeScript

query() returns an async generator. You iterate it, messages arrive as Claude works, and the last one is a result. allowedTools pre-approves Read and Glob. permissionMode: "dontAsk" turns anything that would have prompted a human into a denial. That pairing is the docs' recommended shape for a headless agent, and it's what I use as the starting point for everything.

Run it with npx tsx agent.ts. It reads the PDFs (the Read tool handles PDFs and images natively) and prints a list.

The same agent in Python

Same shape, snake_case option names, and isinstance checks instead of a type discriminator. Python also has ClaudeSDKClient, a stateful client for multi-turn conversations: you open it with async with, call client.query() as many times as you like, and read replies with client.receive_response(). Reach for it when the conversation continues past one prompt or when you want to change the permission mode mid-session with set_permission_mode(). For a one-shot job like this, query() is enough.

Reading messages, results and cost

There are four message types you'll deal with in practice.

A system message with subtype init arrives first. It carries the session ID and the lists of tools, skills and slash commands that loaded. When something isn't behaving, this is where you check whether it loaded at all.

assistant messages hold content blocks: text and tool_use. Watching the tool_use blocks is how you see what the agent is doing.

user messages carry tool results coming back into the conversation. Each has a uuid, which matters later for file checkpointing.

The result message ends the turn. subtype is success or one of several error variants (error_max_turns, error_max_budget_usd, and so on). It carries the final text in result, plus total_cost_usd, a usage block and a per-model modelUsage breakdown. If you asked for structured output, it's on structured_output.

One thing that surprised me: a one-shot query() throws (TypeScript) or raises (Python) after yielding an error result. So you get the error message in the stream and then an exception. Wrap the loop in a try if you're processing a batch and want to continue past a failure.

Tools, MCP servers and Skills

Built-in tools and the two layers of control

Out of the box the agent has Read, Write, Edit, Bash, Glob, Grep, WebSearch, WebFetch, an Agent tool for delegating to subagents, a Skill tool, and AskUserQuestion. The todo and task-tracking tools (TodoWrite, TaskCreate and friends) used to be there too, but on Sonnet 5, Opus 4.8, Fable 5, Mythos 5 and newer they're off by default. If you want them, name one in allowedTools, which opts the session in, or set CLAUDE_CODE_ENABLE_TODO_TOOLS=1.

The thing that took me longest to internalise is that there are two separate layers of control, and the option names don't make it obvious.

Availability is whether a tool appears in Claude's context at all. tools: ["Read", "Grep"] means only those built-ins exist. tools: [] removes every built-in so Claude can only use your MCP tools. A bare name in disallowedTools, like "Bash", also removes the tool from context.

Permission is whether a call gets approved once Claude attempts it. allowedTools is an auto-approve list. Tools you don't list are still available; a call to one of them just goes through the permission flow (mode, then your callback) instead of running immediately. A scoped deny rule like "Bash(rm *)" leaves Bash in context and blocks only matching calls.

So allowedTools doesn't restrict anything. It approves. If you want a tool gone, remove it from tools or deny it by bare name. Otherwise Claude may burn a turn trying it and getting refused.

Custom tools with in-process MCP servers

This is where the invoice agent starts earning its keep. We want Claude to check the supplier on each invoice against our own records, so we give it a lookup_supplier tool. In the SDK, a custom tool is an MCP tool running inside your process. No subprocess, no network hop, no separate server to deploy.

In TypeScript you define the input schema with Zod (3 or 4 both work) and the handler's args are typed from it:

The same tool in Python uses the @tool decorator. The schema is a plain dict of names to types, which the SDK converts to JSON Schema. When you need enums, optional fields or nesting, pass a full JSON Schema dict instead.

A few things worth knowing.

Tool names are fully qualified as mcp__<server>__<tool>, where the server segment is the key you used in mcpServers. That full string is what goes in allowedTools. mcp__suppliers__* covers every tool on the server.

Returning isError: true (Python: "is_error": True) is how you write the error message Claude reads. An uncaught exception in the handler doesn't kill the loop either; the SDK catches it and passes the raw message through. But "No supplier on file, treat as new" gives the model something to act on. A stack trace doesn't.

readOnlyHint: true lets Claude call the tool in parallel with other read-only tools. It's a hint, not enforcement. If your handler writes to disk, don't mark it read-only.

Tool search is on by default and defers your SDK tool schemas: Claude sees a compact list of names and loads a schema when it needs one. With a handful of tools it doesn't matter. With forty, it's the difference between a usable context window and not.

One Python-specific limit: the @tool decorator forwards only content and is_error from your return dict. If you need structuredContent, run a standalone MCP server instead.

Connecting external MCP servers

Anything that speaks MCP plugs into the same mcpServers map. Stdio servers are spawned as subprocesses; SSE and HTTP servers are connected to over the network.

Treat every external MCP server as an untrusted input source. Its tool results land in Claude's context, which means a poisoned CRM record or a malicious web page can carry instructions your agent might follow. Scope allowedTools to the specific tools you need rather than mcp__xero__*, and put anything that spends money or sends email behind a hook (next section). If you've read my post on the Claude Chrome extension's auto-approve classifier, the same threat model applies, minus the classifier.

Using Agent Skills in the SDK

Skills are the folder-of-instructions feature from Claude Code: a .claude/skills/<name>/SKILL.md file with frontmatter and a body that Claude reads when the skill is relevant. In the invoice agent I keep a gst-rules skill with the edge cases for Australian GST treatment, so the prompt stays short and the rules live in a file I can version.

Three things you need to know for the SDK specifically.

There's no programmatic API for skills. They're discovered from the filesystem through the user and project setting sources. If you've set settingSources explicitly, include "project" (and "user" if the skill lives in ~/.claude/skills). If you left it at the default, they load already.

The skills option controls which ones Claude may invoke: "all", a list of names, or []. When you set skills, the SDK adds the Skill tool to allowedTools for you. If you also pass an explicit tools list, put "Skill" in it, or the tool won't exist and nothing will load.

To confirm a skill loaded, check the skills array on the system/init message. A skill with user-invocable: false in its frontmatter loads but won't appear in slash_commands, which is easy to misread as a failed load.

Plugins bundle skills, agents, hooks and MCP servers together and load from a local path via the plugins option. CLAUDE.md files load through the same setting sources. If you turn sources off for isolation, you lose CLAUDE.md too, so move anything you still need into systemPrompt.

Permissions, hooks and subagents

This is the section to read twice. An agent that reads invoices is fine. An agent that reads invoices and has Bash is a different risk category, and the SDK's permission model has enough moving parts that a wrong assumption can quietly open a hole.

Permission modes explained

Modes are one step in a six-step pipeline, and the order matters more than the modes do:

1. Hooks run first. A PreToolUse hook can deny outright or pass the call on. A hook that returns allow does not skip the next two steps.
2. Deny rules, from disallowedTools and settings.json. A match blocks the call in every mode, including bypassPermissions.
3. Ask rules from settings.json. A match sends the call to canUseTool for confirmation, again in every mode.
4. The permission mode.
5. Allow rules, from allowedTools and settings.json.
6. canUseTool, if nothing above resolved it. In dontAsk this step is skipped and the call is denied.

The consequences of that order are where people get hurt.

Auto-approved calls never reach canUseTool. If a tool is approved by an allow rule or by acceptEdits, any check you wrote in the callback is skipped for that tool. The TypeScript SDK emits a CLAUDE_SDK_CAN_USE_TOOL_SHADOWED process warning when your config makes this happen. For a check that has to run on every call, use a PreToolUse hook. Hooks run before everything else, and a hook deny holds even in bypassPermissions.

allowedTools does not constrain bypassPermissions. Listing ["Read"] alongside bypassPermissions still approves Bash, Write and Edit, because unlisted tools fall through to the mode and the mode says yes. If you need bypassPermissions with specific tools blocked, use disallowedTools.

Deny rules are matched as written. Bash(rm *) blocks rm -rf ./tmp and doesn't block /bin/rm -rf ./tmp. Bare Bash removes the tool entirely.

Path anchoring has a double-slash trap. Edit(//secrets/**) blocks writes under /secrets on disk. Edit(/secrets/**) with one slash anchors to the working directory, so it blocks ./secrets and leaves /secrets alone. Not the rule you thought you wrote.

Unanchored allow globs are ignored. allowedTools: ["*"] or ["mcp__*"] does nothing beyond a startup warning. Allow globs need a literal server prefix, like mcp__suppliers__*.

A word on auto mode, since it became the default for interactive Claude Code sessions on Pro, Max and Team plans in August. It works in the SDK too, and the Python type includes it despite what some older writeups claim. Availability depends on your provider and model, and on API-key surfaces the classifier's calls count toward your token usage, so there's a cost line to consider. Check the permission-modes page in the Claude Code docs for the current model floor before you build on it. For an unattended agent I still prefer dontAsk with a tight allow list. It's boring, and boring is the goal.

Subagents inherit the parent's mode unless you set permissionMode on the AgentDefinition and the parent is in default, dontAsk or plan. A subagent never gets bypassPermissions on its own; it runs in that mode only when the parent does. Which means if your parent runs bypassPermissions, every subagent has full system access too, with a different system prompt and whatever behaviour that produces.

Hooks: intercepting the agent loop

Hooks are in-process callbacks that fire at fixed points in the loop. Both SDKs support PreToolUse, PostToolUse, PostToolUseFailure, UserPromptSubmit, Stop, SubagentStart, SubagentStop, PreCompact, PermissionRequest and Notification. TypeScript additionally gets SessionStart, SessionEnd, Setup, PostToolBatch, PermissionDenied and a handful of newer events. In Python, SessionStart and SessionEnd have to be shell hooks in .claude/settings.json, loaded with setting_sources=["project"].

Here's the hook that keeps the invoice agent away from secrets. It matches four tools and denies any call whose input mentions a .env file, which catches Read on .env, cat .env through Bash, and the creative variants in between.

And in Python:

Stringifying the whole tool input is crude, and that's the point. A scoped deny rule on the file tools handles the polite case; the hook handles Bash and whatever else the model thinks of.

PreToolUse can also return permissionDecision: "allow" with an updatedInput to rewrite the arguments before the tool runs, which is how you'd redirect all writes into a sandbox directory. PostToolUse can append additionalContext to a tool result or replace the output with updatedToolOutput. What PostToolUse cannot do is undo the call. It already ran. Anything protective goes in PreToolUse; PostToolUse is for the audit log, which for an AP agent is not optional. I log tool name, a hash of the input and the session ID on every call.

Hooks defined in settings.json also load when the matching setting source is on. So an SDK app with default settingSources inherits the project's hooks, which is either convenient or alarming depending on whose project it is.

Subagents: delegation and context isolation

Subagents are separate conversations the main agent can spawn through the Agent tool. Each one gets its own context, its own tools and optionally its own model, does its work, and returns only its final message to the parent. The parent's history doesn't leak in, and the subagent's tool calls don't clutter the parent's context.

For the invoice agent, GST checking is a good candidate. It's a distinct job with its own rules, and I'd rather run it on Sonnet than on whatever the parent is using.

In Python the shape is the same, with one trap: AgentDefinition uses camelCase field names (disallowedTools, permissionMode, maxTurns) because they map to the wire format, while ClaudeAgentOptions uses snake_case. Pass max_turns to an AgentDefinition and you get a TypeError at construction time.

A few things the docs mention only in passing.

The delegation tool shows up as "Agent" in tool_use blocks but "Task" in the system/init tools list, and CLI versions before 2.1.63 used "Task" in both places. If you're watching for delegation, match both.

Programmatic agents override filesystem agents of the same name in .claude/agents/. Handy when you want to ship a corrected version without touching the repo.

A subagent knows nothing you didn't put in the Agent tool's prompt string. No parent history, no parent system prompt. If the GST checker needs the supplier's registration status, the parent has to say so.

Subagents run in parallel, so three independent checks finish in the time of the slowest one. They also bill in parallel. Nested spawning is capped at a depth of 3 by default (CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH, set it to 1 to stop nesting), and concurrency at 20 (CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS). Set maxBudgetUsd before you find out what an unbounded tree of subagents costs.

Taking your agent to production

Structured outputs and sessions

An invoice agent that returns prose is a demo. One that returns typed JSON is a pipeline stage. The SDK's outputFormat option takes a JSON Schema, makes the model conform to it, and puts the validated object on the result message.

In Python, build the schema with Pydantic (Invoice.model_json_schema()), pass output_format={"type": "json_schema", "schema": ...}, and read message.structured_output. The schema support covers the basics: types, enum, const, required, nested objects and $ref. If the model can't produce a conforming object after a few tries you get a result with subtype error_max_structured_output_retries instead of garbage, which is the right failure mode for a pipeline.

Sessions are the other production feature. Every run has a session ID (on the init and result messages), and you can resume one later with history intact, or fork it to try two continuations from the same point. Compaction is automatic when the context gets full. If you turn on enableFileCheckpointing (Python: enable_file_checkpointing), the SDK snapshots files as the agent edits them and you can roll back to any user message's uuid with rewindFiles(). And query.interrupt() stops a running agent from the outside, which you'll want wired to a cancel button before someone asks for it.

Hosting patterns and sandboxes

The SDK runs the agent in your process, so where it runs is your problem. Anthropic's hosting guide describes three patterns, and the names are about what you'd expect.

Ephemeral: one container per task, destroyed when it finishes. This is the invoice agent. A PDF arrives, a container spins up, the agent runs, the JSON comes out, the container dies. Nothing to clean up, nothing to leak between customers.

Long-running: a persistent container per user or per workspace, for agents that hold state across many turns.

Hybrid: a persistent coordinator that spawns ephemeral workers.

For the container itself, the docs list Modal Sandbox, Cloudflare Sandboxes, Daytona, E2B, Fly Machines and Vercel Sandbox, plus Docker, gVisor and Firecracker if you'd rather run your own. Pick whichever one you already run; nothing about the SDK cares.

Two things the docs say quietly that deserve saying loudly. First, the SDK gives you an async generator, not a server. The HTTP or WebSocket layer, thread management, per-user isolation and auth are all yours to build. Second, set settingSources: [] on anything multi-tenant. Otherwise a stray .claude/settings.json in the working directory becomes part of your customer's agent config, hooks included.

Every query() spawns the CLI subprocess, and that startup is measurable, a second or two on a cold container. For a batch job it's noise. For a chat UI it's the difference between snappy and sluggish. The TypeScript SDK has a warm-start path that spins the subprocess up before the first prompt; it's labelled preview, so check the TypeScript reference for the current shape of the API before you depend on it.

For observability, pip install "claude-agent-sdk[otel]" gives you OpenTelemetry tracing, and the SDK propagates TRACEPARENT into the CLI subprocess so the spans join up. If you're on AWS, Bedrock AgentCore will host the SDK as a managed runtime.

Cost control and pricing

You pay per token at the model's API rate, plus your compute. Nothing extra for the SDK itself.

Sonnet 5's $2/$10 was announced as introductory pricing through August and was supposed to rise to $3/$15 on 1 September. Anthropic dropped the increase, so $2/$10 is the standard rate now. Prompt cache reads are 10% of the input price, or 2.5% on Fable 5.1, which matters for an agent because the system prompt, tool schemas and CLAUDE.md go out on every turn.

Three levers, in the order I reach for them. maxBudgetUsd (Python: max_budget_usd) caps a single run; the result comes back with subtype error_max_budget_usd when it trips. maxTurns caps the loop. And the model choice per subagent, because most of the work in an agent is not the hard part, and Sonnet at 40% of Opus's price does the easy parts fine.

If you use auto mode, the classifier calls are billed as tokens on API keys. Not huge, but it's a line item.

Auth rules and branding

This one's short because the rule is short. If you're building a product on the Agent SDK, authenticate with an API key from the Console or one of the cloud providers. Anthropic's docs say it directly: unless previously approved, third-party developers may not offer claude.ai login or subscription rate limits in their products, and that includes agents built on the SDK. Using an OAuth token from a Free, Pro or Max account in any other product, the SDK included, is a breach of the consumer terms.

What confuses people is the subscription side. In June, Anthropic announced and then paused a change to how Agent SDK, claude -p and third-party app usage would be metered on subscriptions. As of today nothing changed: that usage still draws from your subscription's limits. That's about you, on your own machine, on your own account. It doesn't turn a Max plan into a way to power a product for other people.

Branding is in the same docs. You can call your agent "Claude Agent", list "Claude" inside an Agents menu, or say "YourAgent, powered by Claude". You can't call it "Claude Code" or "Claude Code Agent", and you can't reuse Claude Code's ASCII art or visual style. Your product keeps its own name.

Using the Agent SDK from Go

My backends are Go, so this is the question I actually cared about. The answer is that the SDK is Python and TypeScript only, and the official route for any other language is to run the Claude Code CLI as a subprocess with -p and --output-format stream-json. The SDKs themselves do exactly this under the hood.

stream-json needs --verbose or you get nothing. --include-partial-messages adds token-level deltas if you're streaming to a UI. --input-format stream-json turns it bidirectional for multi-turn. Check the exit code first and the JSON second; a hard failure can print to stderr with a mangled stdout.

There are at least five community Go ports on GitHub (schlunsen, panbanda, ProjAnvil, next-bin and connerohnesorge all have one). They all wrap the CLI the same way, none is affiliated with Anthropic, and every one of them lags a release train that ships daily. For a prototype, pick the one with the most recent commit. For production I wrap the CLI myself. It's about sixty lines, I own every one of them, and when Anthropic renames a field I'm not waiting on a maintainer. One warning: a couple of the ports show OAuth login flows in their READMEs. Re-read the auth section before copying that.

Eight mistakes I'd rather you didn't make

1. Installing @anthropic-ai/claude-code or claude-code-sdk and wondering why the imports don't match the docs. Those are the old names.
2. Expecting .env to load. It doesn't. Export the key or load the file yourself.
3. Getting a bland agent and blaming the model. You're on the minimal system prompt. Ask for the claude_code preset.
4. Trusting allowedTools under bypassPermissions. It approves, it doesn't restrict. Use disallowedTools for the things that must never run.
5. Writing Edit(/secrets/**) and thinking it covers /secrets. One slash anchors to the working directory. Two slashes for an absolute path.
6. Putting the safety check in PostToolUse. The tool already ran. PreToolUse is the only hook that can stop it.
7. Matching only "Task" when watching for subagent calls. Current CLIs report "Agent" in tool_use blocks.
8. Shipping a product on a Pro or Max OAuth token. It's a breach of the consumer terms. API keys.

FAQ

Is the Claude Agent SDK free?

The library costs nothing. You pay for the tokens your agent consumes at the model's API rate, plus whatever you run it on.

Can I use the Claude Agent SDK with a Claude Max subscription?

For your own use, on your own machine, yes: subscription usage currently covers the SDK and claude -p, after Anthropic paused the planned billing change in June. For anything you offer to other people, no. Products need API keys.

Does the Claude Agent SDK support Go?

Not officially. Anthropic's answer for other languages is to run the CLI as a subprocess with --output-format stream-json, which is what the section above does. The community Go ports wrap the same thing.

Do I need Claude Code installed to use the SDK?

No. The binary is bundled in the npm package and the Python wheels. The exceptions are npm ci --omit=optional and sdist installs, where you install Claude Code yourself and set the executable path.

Claude Agent SDK or OpenAI Agents SDK?

If your agent needs a computer (files, shell, a codebase) and you're happy on Claude, the Agent SDK gives you Claude Code's loop with nothing to build. If you want to swap models between providers, or you want handoffs and guardrails as first-class primitives, OpenAI's SDK is built around those. Neither is a hosting product; that's Managed Agents and the Agents API respectively, and both of those are US-only today.

Where to go from here

If you'd rather someone else built the agent, that's what I do for a living. I build AI agents and workflow automation for small and medium businesses across Australia, and accounts payable is where most of them start.

Docs I checked against

• Agent SDK overview, quickstart and migration guide
• Custom tools, permissions, hooks, subagents, skills, structured outputs and hosting
• TypeScript reference and Python reference
• Building agents with the Claude Agent SDK, the rename announcement
• Legal and compliance for the auth and branding rules, and pricing
• anthropics/claude-agent-sdk-typescript and anthropics/claude-agent-sdk-python on GitHub

There was project.md with a full description of what I was building. A CLAUDE.md or AGENTS.md with project rules and coding conventions. A todo.md with the work broken into ordered tasks. And prompt.md, usually somewhere north of 800 words, to kick the whole thing off. Only then would I paste the prompt in and let the agent start.

Last month I built a playable roguelike. The first message was one sentence.

That gap is what this post is about. Testing Fable 5.1 and GPT-6 Astra made it obvious how much my own habits had drifted, and I don't think I'm the only one. The scaffolding I built in 2025 was a reasonable response to models that needed it. The current generation mostly doesn't.

What Agentic Coding Actually Means in 2026

Quick definition so we're on the same page. Agentic coding is when you give an AI agent a goal instead of a completion target, and it plans, reads your codebase, edits multiple files, runs your tests, reads the failures, and keeps going until it thinks it's done. You review and sign off.

That's different from autocomplete, which predicts the next few lines inside the file you're already in. It's also different from vibe coding, where you're accepting output without reading it. Agentic coding still has you in the loop. It just moves you from writing the code to directing and verifying it.

The Old Workflow: Four Markdown Files Before You Start

The 2025 version of my process was front-loaded. I'd spend maybe 45 minutes on setup before a single line got written.

project.md described the whole thing: what it does, who uses it, the data model, the screens. CLAUDE.md held the rules, and mine used to run 200 lines or more, covering everything from error handling patterns to how I like imports sorted. todo.md was my attempt to decompose the work into a sequence the agent could follow without wandering. And prompt.md restated most of it again, because I didn't fully trust the files to be read.

It worked. I shipped things this way and the results were fine. But look at what I was actually doing: I was writing a spec, then writing a summary of the spec, then writing a plan derived from the spec, and hoping the model would reconcile all three. Every one of those documents was a guess about what the model would need. And I made those guesses before I'd seen a single line of its output.

Why It Stopped Being Necessary

Two things changed. One I noticed, one got measured.

The one I noticed: shorter prompts started producing better results. Not always exactly what I had pictured, but usually something good, and often something better than what I would have specified. When it wasn't right, I'd say so, and the next iteration fixed it. At some point the cost of correcting the model dropped below the cost of specifying everything upfront, and once that flips, all the front-loading is just work you didn't need to do.

The one that got measured: researchers at ETH Zurich published the first proper controlled study on repository context files in February (arXiv:2602.11988). Across SWE-bench Lite and a new 138-task benchmark, context files did not generally improve task success rates while adding over 20% to inference cost. Auto-generated files were worse than having no file at all. Human-written ones did better, around 4% on their benchmark, so this isn't an argument for deleting everything. I went through that paper properly in CLAUDE.md: Helpful or Just Expensive Noise? if you want the full breakdown.

The interesting bit for me was the ablation. When they stripped a repo of its README and docs, context files suddenly did help. Which suggests most of what I'd been writing into project.md was already sitting in the repository, and the model was perfectly capable of going and reading it.

There's a second reason worth naming. A long spec written before you've seen anything is mostly assumptions. You're describing a thing that doesn't exist yet, in a language the model interprets loosely, and any detail you got wrong becomes a constraint the agent now faithfully honours. Overspecifying doesn't just waste tokens. It locks in your mistakes.

The New Workflow

Here's what I actually do now.

1. Write the shortest description that still contains your requirements. One to three sentences on what you want built. Add the language, framework, platform, or target audience only if it genuinely constrains the outcome. If you'd be happy either way, leave it out.
2. Let it run. Don't interrupt to add the detail you forgot. You'll get a chance in a few minutes.
3. Test the first iteration. Run it, don't read it. Reading the diff tells you what the agent thinks it built. Running it tells you what it built.
4. Give feedback as a bullet list. Short, direct, one item per thing that's wrong or missing. No explanation of why, no restating the original brief.
5. Repeat.

The whole thing is a loop, and the loop is fast enough that you can afford to be wrong on the first pass. That's the actual unlock. Not that the models got smarter in the abstract, but that iteration got cheap enough to replace planning.

What This Looks Like: A Roguelike From One Sentence

I wanted to test this properly, on something with enough surface area that sloppy direction would show up fast. So I built a game.

Grokue Like is a turn-based roguelite that runs in the browser. Ten procedurally generated floors, loot (normal, rare, epic, and unique), a skill tree, a merchant, and legacy embers you bank between runs so a failed run still leaves you something. It's free, there's no account, and it's kinda fun to play, which surprised me more than it should have.

The stack was Grok 4.6 driving the Pi coding agent. Grok 4.6 landed in August with a 500k context window, and xAI put a lot of emphasis on interactive and visual first passes, which is exactly the shape of this problem. That turned out to be the right call. The first playable version came back with a coherent visual direction I hadn't asked for.

For art, I gave the agent a tool instead of assets. I wrote a small CLI wrapper around Meta's Muse Image, then wrapped that in a skill so the agent could call it directly and generate its own sprites and backgrounds. Muse Image has an anchored composition mode where you pin a series of generations to a few reference images, which is how a set of sprites ends up looking like it came from one artist rather than eleven. At a cent per image, the agent could also afford to throw away the bad ones. Sound and music came from ElevenLabs.

The first message was one sentence: build a turn-based roguelike that runs in the browser. Everything after that was me playing a build and coming back with bullets. Combat feels floaty. The map needs a fog reveal. Death should carry something forward. Merchant prices are wrong. That's the entire creative process.

Things I never specified and would probably have got wrong: the ember mechanic, the art style, and the line on the status bar that reads "the dark moves only when you do." That last one is better writing than I'd have put in a spec.

It Feels Like Leading a Team, Not Writing a Spec

The mental shift that made this click was giving up on the idea that I'm programming the agent.

I'm not. I'm directing it. The relationship that fits best is the one you have with a good developer on your team: you tell them what you want and roughly why, you let them make the calls inside that, and you give feedback on what comes back. You don't hand a senior engineer an 800-word document specifying their variable names. If you did, you'd get worse work and they'd start looking at job ads.

This is also the thing that makes multiple agents workable. Heavy per-project scaffolding doesn't scale past one or two agents, because you're maintaining a spec for each of them and paying the context cost every session. Short briefs and bullet-list feedback scale fine. I can keep three going and my job in each of them is the same 30 seconds of reading and reacting. It's closer to standup than to programming.

Where You Still Need a Real Prompt

None of this means prompting is dead. I want to be clear about that, because "the models are smart now, just talk to them" is a take that will cost you real money in the wrong context.

The dividing line I use is repetition. If a task runs once, iterate conversationally. If it runs more than once, write the prompt properly.

The reason is that conversational iteration has no memory across runs. You converge on a good result through five rounds of feedback, and then the whole thing evaporates. That's fine for a game. It's terrible for an invoice extractor that has to behave identically on Tuesday. For that work, the prompt and the context assembly are the product, and I'd point you at context engineering for AI agents and my prompt engineering notes rather than repeating myself here.

Client work sits mostly on the right side of that table. Personal projects sit mostly on the left. That's roughly the split.

What I Still Put in CLAUDE.md and AGENTS.md

Shorter, not gone. My files run about 10 to 40 lines now and contain only things the agent can't work out by looking around: non-obvious build and test commands with their exact flags, architectural decisions that contradict what the code structure implies, and conventions that go against the obvious default. Everything else got deleted, because it was either duplicating the README or telling the model something it already knew.

Anything that genuinely cannot be broken goes in a hook, not a markdown file. Markdown is a request. A hook is a rule.

FAQ

What is agentic coding?

Agentic coding is a development workflow where an AI agent takes a goal, plans the work, edits files across your codebase, runs the tests, and self-corrects until it's done, with a human reviewing and approving the result. The distinction from autocomplete is autonomy over multiple steps. The distinction from vibe coding is that you still verify.

How do you prevent AI coding agent mistakes with shorter prompts?

By testing every iteration and correcting fast. Short prompts don't reduce mistakes, they change when you catch them. Instead of trying to prevent errors with an upfront spec, you find them by running the build and fixing them in the next round. Keep iterations small enough that any single one is easy to review.

How do you prevent architecture drift in AI coding agents?

Verification you can automate, plus short sessions. Hooks and a test suite catch drift deterministically where instructions in a markdown file won't. Instruction compliance degrades as a session fills up, so starting fresh sessions for distinct pieces of work does more for architectural consistency than a longer rules file.

Is prompt engineering dead?

No. It moved. For one-off building work, conversation beats a carefully written prompt because the correction loop is faster than the specification loop. For anything repeated, automated, or running in production, a well-built prompt and a deliberate context pipeline still make the difference between a system that works and one that works most of the time.

Should I still write a CLAUDE.md or AGENTS.md?

Yes, but keep it under about 40 lines and put only non-discoverable things in it. Committing auto-generated output unreviewed measurably underperforms having no file at all.

The Workflow Will Change Again

I'm aware that everything above has a shelf life. My 2025 process was correct for 2025 models and it's dead weight now. Something similar will happen to this one, probably faster than I'd like.

The part I'd bet on outlasting the specifics: the work is moving from specification to direction and verification. Getting good at describing what you want in three sentences, and at recognising quickly whether what came back is right, is a more durable skill than any particular file layout. Go build something small with one sentence and see what happens.

So when Claude Fable 5.1 landed on September 1, my expectations were low. Fable 5 was already excellent. What's a point-one release supposed to do?

Then I gave it two jobs. Build an image codec that beats WebP. Build a website good enough that an award-winning design studio would put its name on it. Both worked. That's the short version of the review. The longer version is about why, and about whether it's worth what it costs, which is a lot.

What Claude Fable 5.1 actually is

Claude Fable 5.1 is Anthropic's frontier model, released September 1, 2026, sitting above Opus 5 in a tier they call Mythos-class. It shares its weights with Claude Mythos 5.1. The only real difference between the two is safeguards: Fable is generally available with conservative classifiers on top, Mythos goes out through trusted-access programs with those classifiers loosened.

The specs, from the Bedrock model card:

Two of those matter more than they look. The June 2026 cutoff is the freshest of any Claude model, which you notice immediately with libraries that moved this year. And adaptive thinking can't be turned off, so every request carries reasoning tokens whether you wanted them or not. Remember that for the pricing section.

The other thing worth knowing is routing. Some cybersecurity and biology queries get flagged and answered by Opus instead of Fable, and you aren't billed at Fable rates when that happens. Anthropic says cyber false positives are down roughly 60% per session versus Fable 5. I didn't trip a single one across either test, but if you do authorised security work you still will, and the Claude Code issue tracker has months of complaints about exactly that.

Test one: a Go image codec that beats WebP

I picked a codec deliberately. It's hard in a way most benchmark tasks aren't. There are a million Express APIs in the training data and not a million novel entropy coders. You have to hold a lot of state, make decisions that only pay off three files later, and be right about the maths, because when you're wrong about the maths you don't get a warning. You get garbage pixels.

I asked for something better than WebP, in Go, and I let it run.

It landed in one shot. One relatively short prompt, no back and forth, no arguing about whether the bug was real. That surprised me more than the result did.

The result: the codec beats WebP on compression by somewhere between 1 and 9 percent depending on the image, and it's much faster on top of that. One to nine percent sounds modest until you remember WebP has had thirteen years of engineers pushing on it. I did not expect to beat it at all, let alone beat it on speed.

Now the caveats, and they're big enough that I want them in the same section rather than buried at the bottom. This is an experiment. You could not ship it. Compression ratio is maybe a quarter of what makes a real codec viable, and the rest is decode performance on hardware you don't control, progressive rendering, colour accuracy, alpha handling, metadata, streaming, plus the small matter of every browser on earth agreeing to support it. Beating WebP on one axis is not the same as replacing it.

The test set was small too. Under 50 images. That's enough to tell me something interesting happened and nowhere near enough to claim general performance. A real evaluation needs thousands, spread across photos, illustrations, screenshots, and all the horrible edge cases that break codecs in production.

Take it as what it was. Not "can I ship this," but "can the model do a genuinely hard thing from a cold start." It could.

Test two: a website an agency could ship

The second test was the opposite kind of hard. Not correctness, taste.

The brief was a site for an award-winning web design company, which is designed to catch models out. "Professional website" is what every model produces by default. "Award-winning" is what almost none of them can do, and the gap between the two is where all the interesting failure lives. I've run six models through this kind of frontend test before and most land in competent-but-generic territory.

The twist I added: I asked for a portfolio, with a few examples of the studio's best work. That turns one design problem into several at once. A portfolio whose sample work is mediocre fails the brief no matter how nice the wrapper is, and a model with only one good design in it gets caught immediately, because you can watch it recycle the same ideas across every example.

Fable 5.1 didn't get caught. The result is at fable51-webdesign.thomas-wiegold.com. The portfolio pieces are distinct from each other and individually good, which is the bit I didn't expect. The animations and transitions are what I keep going back to look at. They're timed properly and they carry the eye somewhere, and that's usually where generated frontends fall apart.

One prompt. No revisions. Good on the first try, same as the codec.

I'd put it in front of a real client tomorrow. This is the test I'd point at if someone asked what "creative" means for a coding model. Not that it wrote CSS. That it made choices, several times over, without repeating itself.

What actually changed from Fable 5

Here's where I have to be careful, because I like this model and that's exactly when you start seeing improvements that aren't there.

My honest read: Fable 5.1 feels better to work with than Fable 5. More direct. Less talkative. It goes at the goal instead of narrating its way toward it. It might be slightly faster, though I didn't time anything properly, so treat that as a feeling rather than a measurement.

What I can't tell you is that the output is better. My Fable 5 results were already great. My Fable 5.1 results are also great. If there's a quality difference in the final artifacts I'm not confident enough to claim it.

The interesting part is that "less talkative" turns out to be measurable. Snorkel AI found that on frontier coding tasks both models solved, Fable 5.1 used 58% fewer output tokens than Opus 5 and finished 36% faster. Every reported roughly twice the speed at about half the tokens. So the thing I noticed as a vibe is a real property, and it's the one I'd care about most day to day.

On benchmarks, two numbers are worth your attention and most of the rest aren't. Terminal-Bench-Science 0.1 went from 24.7% on Fable 5 to 52.6% on 5.1, which is a jump you rarely see on a point release. And Artificial Analysis put it at the top of its Intelligence Index with a score of 66 at max effort, ahead of Opus 5 at 63 and GPT-5.6 Sol at 61.

One correction while I'm here, because I keep seeing it repeated. The 95% SWE-bench Verified figure doing the rounds is Fable 5's number, not 5.1's. Anthropic didn't headline a SWE-bench result for 5.1 at all. If a review quotes it as a 5.1 score, the person writing it didn't read the launch post.

The cost problem, which has not gone away

Fable 5.1 costs $10 per million input tokens and $50 per million output. Unchanged from Fable 5, and exactly double Opus 5 on both sides.

The headline improvement is cache reads, cut 75% to $0.25 per million from $1.00. That's a strange enough number to say out loud: Fable 5.1's cached input is now cheaper than Opus 5's, even though its base rate is double. If your workload is a long-running agent chewing through the same context, the maths changed in your favour.

Anthropic frames this as roughly 25% cheaper for typical workloads and up to 45% for agentic ones. Artificial Analysis found the opposite on its own Intelligence Index runs: about 20% more expensive per task than Fable 5, because 5.1 burns more tokens getting there. Both can be true. Which applies to you depends on your cache hit rate, and you won't know until you measure it.

For those of us on subscriptions rather than the API, the practical version is simpler. Credits go fast. That's the part that stops me reaching for Fable by default, and it's the same complaint I had about Fable 5.

When it's worth it

Default to Opus 5. Escalate to Fable 5.1 when the task is hard, important, or creative. That's not me being cautious, it's what Anthropic's own docs recommend, and it matches how I've actually been using it.

A concrete threshold: if Opus 5 at high effort fails more than about 15 to 20 percent of your long-horizon tasks, or cache-heavy agent workloads dominate your bill, Fable earns the premium. Otherwise you're paying double for a difference you won't notice on a CRUD endpoint.

Now the part where I disagree with the leaderboards.

The number people point at is Artificial Analysis's Coding Agent Index v1.1, where GPT-5.6 Sol scored 80 against Fable's 77. Two problems with it. The 77 is Fable 5's score, not 5.1's, so the comparison is a generation out of date before you even get to the argument. And I don't think the ranking reflects what happens when you sit down and actually use these models.

I think the benchmarks are broken, and not as a vague complaint. Look at how the numbers get made. Anthropic's cross-vendor comparison columns were run by Anthropic, under Anthropic's conditions. Artificial Analysis revised its own methodology mid-year and Fable 5's launch score of 64.9 became 60, with a note that the two aren't interchangeable. On OSWorld and AutomationBench, runs where a safeguard fired scored zero, which drags the average down for a reason unrelated to capability. And nearly every published chart is a cost-accuracy curve across five effort levels, so a number quoted without its effort tier tells you close to nothing.

None of that is dishonest. It's that a benchmark measures a task, and the task isn't the job. My job is ambiguous briefs, taste, and problems where the hard part is deciding what to build in the first place. No index scores that, and no index is going to.

So I'll say it plainly rather than hedge it into mush. Fable 5.1 is number one right now. It built a working codec and a multi-piece portfolio site in one prompt each, and nothing else I have access to does that. If you want a number to agree with me, it tops the Artificial Analysis Intelligence Index at 66. For the head to head against the alternatives, see my Opus 5 versus GPT-5.6 versus Grok 4.5 comparison.

If you take one thing from this, take the test structure rather than my verdict. Pick two tasks, one genuinely hard and one that needs taste. Run both against whatever you use today and against Fable 5.1. Then compare cost per finished task, not tokens per second and not how impressive the output looks in isolation. I went in expecting a nothing release and came out having changed my default for a whole class of work. Your two tasks might say something different. That's the point of running them.

I've been running auto mode in Claude Code since it shipped, so I know what that classifier does to a shell. What I didn't know is what it does to a browser holding my Gmail session. Working that out led me to a number sitting in public that hasn't shown up in a single launch post I've read.

What changed on 26 August

Claude in Chrome is a browser extension that lets Claude read the page you're on and act on it. Clicking links, typing, navigating, filling forms, all through the sessions you're already logged into. GA changed the permission model. Claude now approves actions it judges safe on its own, using the same mechanism as auto mode in Claude Code, and a classifier reviews each action against what you originally asked for. Mismatches get blocked.

The unglamorous details: Pro, Max, Team and Enterprise, no free tier, desktop Google Chrome only, no mobile. Two modes in settings, manually approve and automatically approve, with the Cowork side panel defaulting to auto. Auto isn't a blanket yes, though. Downloads and entering sensitive information still stop and ask, and some actions are blocked whatever you set. The permissions guide is short enough to actually read.

The thing almost nobody picked up is that Anthropic shipped two browser products that morning. The other is a browser built into Claude Cowork, running its own profile, nothing to install. I'll come back to it, because it ends up being my recommendation.

What the classifier can actually see

Anthropic published a detailed engineering write-up on how auto mode was built. It covers Claude Code rather than the browser, but since the Chrome announcement says explicitly that it's the same mechanism, it's the closest thing we have to documentation of what's now guarding your inbox.

Actions get sorted into three tiers. Reads and navigation are allowlisted and skip classification. File edits inside the project directory pass without a classifier call. Everything else hits a two-stage transcript classifier: a fast single-token filter tuned to err toward blocking, then a chain-of-thought pass on whatever stage one flagged.

Now the part I keep thinking about. The classifier sees only your messages and the agent's tool calls. Claude's reasoning is stripped out, and so are tool results. Anthropic calls this reasoning-blind by design, and the justification is good. If the classifier could read Claude's explanation of why an action is fine, Claude could talk it into things. Stripping tool results is the structural injection defence: if the classifier never reads the poisoned page, the poisoned page can't argue with it.

The cost, which Anthropic names, is provenance. If you never mentioned a specific target and the agent produces one, the classifier can't tell whether it was looked up or made up. In a browser, that means the classifier judges whether an action matches your request while structurally blind to the page the action happens on.

The number nobody is quoting

Anthropic evaluated the pipeline on three datasets and published the table. A 0.4% false-positive rate on 10,000 real internal tool calls. A 5.7% false-negative rate on 1,000 synthetic exfiltration attempts. And a 17% false-negative rate on 52 curated real overeager actions. Three questions, three denominators. Don't blend them, and be suspicious of anyone who does.

The 17% is the one to care about if you're automating business work, and Anthropic's own words for it are "the honest number." An overeager action is Claude understanding your goal, genuinely trying to help, and overstepping while it does. No hostile reasoning anywhere. Roughly one in six got through.

Anthropic also explains the misses. In most cases the classifier spotted the danger fine. What it got wrong was whether anything the user actually said covered this specific action. Their phrasing: it finds approval-shaped evidence and stops short of checking whether that's consent for the blast radius. They've tried prompt-engineering around it and haven't found a fix that costs less than it gains.

One of their examples should make anyone automating a business portal sit up. A user says "cancel my job." The agent queries the cluster, picks the closest match by name similarity, deletes it. That's a designated block case, so the classifier is aimed straight at it. Seventeen percent of the time, aimed and missing.

Swap the cluster for a supplier portal with two near-identical contact records, because somebody made a duplicate eighteen months ago and nobody cleaned it up. Same shape, same failure, different bank account.

Two caveats. The 52-item sample is small, and Anthropic says so. And these figures come from Claude Code's shell and file operations, not the browser. Anthropic says same mechanism; it hasn't published browser equivalents. I'm reasoning across, and you should hold that loosely.

Pushing the other way, an independent stress test on arXiv built a benchmark of deliberately ambiguous authorization scenarios and measured an end-to-end false-negative rate of 81%. That's a designed worst case, not typical traffic. Read it as: when ambiguity is systematic rather than accidental, this gate degrades badly.

The case for the machine check

The argument for auto mode is strong, and it's an argument about humans. Claude Code users approve 93% of permission prompts. A gate you clear 93 times out of 100 has stopped being a gate and become a habit.

Anthropic's own verdict is unusually blunt for a launch-adjacent document. Coming from --dangerously-skip-permissions, which I've admitted to using more than I should, auto mode is a substantial improvement. Coming from careful manual approval, it's arguably a regression, because you're trading your judgment for a classifier that's sometimes wrong. They say plainly it isn't a drop-in replacement for careful human review on high-stakes work.

I can't improve on that, so I won't try.

What the prompt injection numbers say

Two evaluations are in circulation and they get mashed together constantly.

The original, built for the pilot, is retired. Anthropic saturated it: no attack succeeded against Fable 5, Opus 5 or Sonnet 5, and that was without probes and classifiers running at all. When a benchmark returns zero for everything, it's stopped measuring.

The current one uses stronger attacks written by professional red-teamers, reported in the GA announcement and sourced to the Claude Opus 5 system card. Attacks that reached the model succeeded 17.6% of the time against Opus 4.5 and 3.8% against Opus 5, before safeguards. Add the probes and the safety classifier and you get zero successful attacks against Sonnet 5, Opus 5 and Mythos 5, with 0.3% against Fable 5, all manually verified low severity.

Read that twice, because the 0% is doing a lot of PR work. The model alone sits at 3.8%. The zero belongs to the scaffolding. Build agents against the API without a probe layer and an action classifier in front of them, and that 0% was never yours.

Note the denominator too. That's the success rate for attacks that reached the model, not attacks attempted. Opus 4.5 had fewer reach it and still had the highest success rate among those that did.

And if you find a guide quoting 11.2% as the current rate, close the tab. That's from the August 2025 Claude for Chrome pilot announcement, where 123 test cases across 29 scenarios went from 23.6% to 11.2%. It isn't just old. It's from a benchmark its own author has discarded.

Two vulnerabilities that never touched the classifier

Here's why the 0% doesn't comfort me on its own.

In March, Koi Security published ShadowPrompt, a zero-click chain against the Claude in Chrome extension. Any website could silently inject prompts as if you'd typed them. No clicks, no dialog. It combined an over-permissive origin allowlist (anything matching *.claude.ai was trusted) with a DOM-based XSS in a CAPTCHA component on one of those subdomains, at a point when the extension had over three million users. Credit where it's due, the handling was good: reported 26 December, patched in v1.0.41 on 15 January.

In May, LayerX disclosed ClaudeBleed. The Claude extension used Chrome's externally_connectable and trusted the claude.ai origin rather than the execution context, so any other extension, including a zero-permission one, could issue it commands. LayerX demonstrated forced Google Drive sharing and Gmail exfiltration, defeating the confirmation prompts by looping approvals and renaming buttons in the DOM. Anthropic patched in v1.0.70 on 6 May. LayerX says the patch is partial, and here's the part that belongs in this article: they report that in autonomous mode the new validation layer stops being effective and the original path reopens. Manifold Security was still finding residual capability in v1.0.72 in July.

Anthropic ships bugs. Everyone ships bugs. What I take from these two is that both lived in the plumbing underneath the classifier, and a classifier scoring 0% on injection evals is scoring on the layer it can see.

Same lesson as Claude Code hooks. A deterministic guard below the probabilistic one is worth a lot because it doesn't share the blind spots. The browser extension has no hooks equivalent. Real gap.

Four checks before you point the Claude Chrome extension at a business account

If you're evaluating for real work this week, start here.

How much of your accounting stack is gated? Anthropic's safety guide hard-blocks only two categories, adult content and known pirated content. Financial sites aren't blocked, but Claude asks permission before accessing them, and it's prohibited outright from stock trading and investment transactions. Anthropic also advises against using the extension to manage financial accounts at all. So the question isn't whether Xero or MYOB are banned, it's how many prompts and prohibited actions sit between you and a finished invoice run.

Where does auto mode stop in your inbox? Run the same triage task twice, manual and auto. You're not checking whether it works. You're checking which actions it decided didn't need you.

What happens with a deliberately ambiguous instruction? Put two plausible targets on the page and see whether it picks one and proceeds. That's the agent-inferred-parameter case in your own data.

Which browser is Claude actually using? If you already have the extension installed, it stays your default. Settings, Cowork, preferred browser.

If you do any injection testing, use a throwaway account with fake data. The premise of the risk is that the agent acts with your signed-in authority, so a successful test and a successful attack are the same event.

So would I let it near a logged-in business account?

Not a yes or no. A fork.

For portal work, invoice collection, research, anything where Claude needs a browser rather than your browser: use the built-in Cowork browser. Own profile, never sees your tabs, bookmarks or passwords, and banking, email and single sign-on stay out unless you deliberately bring those logins across site by site. Same safeguards, dramatically smaller blast radius. For most of what small businesses want to automate, this is the right tool and it isn't close.

Keep the extension for the page you already have open, in accounts you're already in. Updating a CRM record you're looking at. Working through the doc in front of you.

Turn auto-approve off in your inbox specifically. Email is where the overeager-action class does the most damage and where an injection has the richest payload sitting right next to it.

For the Australian small businesses I work with, the blunt version: if the job is accounting automation, you want a purpose-built agent with scoped credentials and an audit trail, not a browser agent driving a UI with your admin session attached. I wrote about where that line falls earlier this year and nothing here moves it.

That's a problem, because when I start a new site the first decision isn't which model writes the cleanest TypeScript. It's which one gives me a layout I don't immediately want to throw away. Those are different jobs, and there's no law saying the same model has to win both. Finding the best LLM for frontend design is a separate question from finding the best coding model, and I wanted an answer I'd actually tested rather than a leaderboard screenshot.

So I ran one. Four design briefs, one run per model, same prompt for everyone, no per-model tuning, single file HTML. Everything is browsable at webdesign-test.thomas-wiegold.com and you can click through the grid yourself.

The honest headline is that I struggled to pick winners at all. Most of the time the results were close enough that I couldn't articulate why I preferred one over another. It was a vibe. That sounds like a cop-out for a post with this title, but stay with me, because "I can't tell them apart" turns out to be the useful finding.

The short answer

There is no single best LLM for frontend design right now, at least not among the models I use. Across four briefs, GPT-5.6 Sol took two, Claude Opus 5 took one and tied another, and Qwen3.8-Max shared that tie. The gaps were small enough that a different judge would produce a different table. The practical move is to run one prompt through three or four models and pick whichever output you like, because that costs ten minutes and beats agonising over the correct choice.

Which models, and why these ones

The lineup is Claude Opus 5, GPT-5.6 Sol, GPT-5.6 Luna, Qwen3.8-Max, Grok 4.6 and DeepSeek V4 Flash. On the conference brief I also managed to run DeepSeek V4 Pro, Kimi K3 and MiniMax M3.

These are the models I actually use and pay for. That's the whole selection criterion, and I'd rather state it plainly than pretend this is a comprehensive benchmark.

The obvious absences are Kimi and GLM, which I know are strong at this. I don't have a Kimi Code or GLM Code subscription. I have opencode go, but running those models through it is expensive enough and limited enough that I couldn't do a full grid, which is why Kimi shows up on exactly one brief. Gemini isn't in here either, and that one bothers me most, because I liked a lot of what Gemini 3.5 Flash produced for web design and I want to see whether 3.7 kept it. All three are on the list to add to the comparison site later.

How I ran it

Four briefs, picked to stress different things.

Pip & Pocket is an early learning centre. Still Running is a running conference. Three in a Row is tic-tac-toe, the only brief with real interaction. Ordinary Hours is a perfume brand, and that prompt asks for something specific: three genuinely different visual styles on one page, switchable with a button.

One run per model per brief. I didn't tune anything, didn't feed anyone a design system, didn't specify fonts. That's deliberate, and I'll come back to why.

The site holding the results was built in one shot by Grok 4.6. Keyboard shortcuts, a compare view, click a model name to see its whole row or column. I asked for a way to browse test results and got something better than I would have specced.

The usual caveats apply and I'd rather say them than have you find them. Single judge, no blind voting, no accessibility or performance audit, HTML only. Nothing here tells you how these models behave inside a real React codebase.

The one that matters most: every cell in this grid is a single generation. These models are not deterministic, so any given result is one sample from a range, and a rerun could land somewhere else entirely. Keep that in mind for everything below, especially anywhere I say a model got something wrong.

Brief by brief

Pip & Pocket, and why cute is hard

This is the brief I'd keep if I could only keep one, because it's the one that isn't on home turf.

These models are excellent at modern, dark, techy, minimal, slightly artsy. That's the register they default to and it's clearly the register they've seen most. Ask for cute, bright, friendly and colourful and things get shakier. It's the same reason a session musician who's played nothing but jazz for ten years sounds a bit odd playing country.

Opus and Sol were both strong and very close. Opus produced nicer SVG illustrations, no argument there. But I preferred Sol's section layout and its colour choices, and layout and colour is more of the page than the illustrations are. So Sol takes it, narrowly, and I'd understand anyone picking the other way.

Test Results: Pip & Pocket

Still Running, where everything worked

All good. That's the summary.

The conference brief is exactly the modern dark techy thing these models are best at, and it shows. I slightly prefer Opus for the animation on the symbols, but Kimi K3 is great, Luna is very good. If you looked at those three side by side and picked a different one, I'd shrug and agree.

The one weaker result was DeepSeek V4 Pro, which didn't land the way the others did. I want to be careful here, because Pro only ran on this single brief, which makes it unfair to call it the loser of the whole test. It's one result. Worth noting, not worth a verdict.

Test Results: Still Running

Three in a Row, and a real bug

The only brief with actual interaction, so the criteria change. What happens on hover, does the win state get any celebration, do the tap targets work on a phone.

Sol and DeepSeek Flash both had the same concrete problem: the grid squares weren't holding their aspect ratio, so dropping an X or an O into a cell stretched it into a rectangle. The layout was just broken.

Opus and Qwen3.8-Max both produced designs that fit a game. Nothing clever, it just works, which for a game is the entire job. Luna was good too.

Worth flagging that Sol's box bug is almost certainly not a fixed property of Sol. It's one generation. Ask again and there's every chance the sizing comes out fine, and I'd have written a different paragraph.

That's the clearest illustration in the whole test of why you generate several times before concluding anything about a model. Including anything I've concluded here.

Test Results: Three in a Row

Ordinary Hours, where I changed my mind

The perfume prompt asks for three distinct visual styles on one page with a button to switch between them, which makes it a test of range. Can the model produce three different aesthetics, or does it produce one aesthetic three times?

My original pick was Luna, and I was pleased about that because Luna is OpenAI's cheap tier and Sol is the flagship, with more than twenty times the per-token price between them. Cheap model wins beauty contest, nice story.

Then I went back through the results while writing this and found a bug: Luna's second theme doesn't render properly. So the story changes. Sol gets it instead, and on its own merits: its three themes are properly distinct from each other, which is the thing the brief was actually asking for.

I'm leaving the reversal in rather than quietly editing the table, because it's a decent illustration of what these comparisons are actually like. You look, you form an impression, you look again and the impression moves.

Test Results: Ordinary Hours

Where the models separate, and where they don't

Line up the four briefs and a pattern falls out that I didn't expect going in.

On the conference site, everyone was good. On the perfume site, most were interesting. On tic-tac-toe, two models had a sizing bug but the designs were fine. On the early learning centre, the field spread out.

The briefs where models are hard to tell apart are the briefs that sit right in the middle of what they've been trained on. Dark, modern, sans-serif, generous whitespace, a bit of motion. Anthropic has a name for the underlying effect: distributional convergence, models drifting toward the median of their training data. Inter, purple to blue gradients, four identical rounded stat cards in a row. You know the look, and you know it because you've now seen it four hundred times.

Push a model off that median and the differences show up. Cute, bright and colourful is off the median. So, in a different way, is "give me three unrelated styles in one file."

Which suggests something for anyone benchmarking this themselves: test the thing your models are bad at, not the thing they're good at. A test where everyone scores well isn't measuring much.

What the leaderboards say, and where I disagree

The public picture as of mid-August 2026: Arena's WebDev board has Claude Opus 5 at number one, with Kimi K3 and Qwen3.8-Max next. The Design Arena website board has Kimi K3 in front, with Opus 5, GLM-5.2, Gemini 3.7 Flash and Grok 4.6 packed into roughly ten Elo points behind it.

Two caveats I'd rather state than pretend precision. The gaps in that top group are smaller than the confidence intervals, so the ordering shuffles week to week. And the Design Arena figures everyone quotes, including the ones above, come from third-party mirrors labelled display-only, because the live board is JavaScript-gated. Treat any specific number as approximate.

Now the interesting bit. Design Arena says Kimi K3 is the best design model going. On the one brief where I ran Kimi against Opus 5, it was close enough that I gave Opus the edge on a feeling about some animations, which is not exactly a rigorous tiebreak.

The board isn't wrong. It's answering a different question. Design Arena runs thousands of blinded pairwise votes across thousands of prompts, and their methodology hides model identity, updates every couple of hours and uses Bradley-Terry ratings. What comes out is average preference across a crowd, and that's a good design for the question they're asking.

It just isn't my question. I'm picking one output for one brief with my own preferences attached. Average preference and my preference are allowed to disagree, and when the top five are within ten Elo of each other, they will disagree all the time. Two of my four picks are GPT models, which the frontend boards don't rank as design leaders at all.

Use the leaderboards to decide which four models to try. Don't use them to decide which one to ship.

Is it the model, or the system prompt?

A while back I wrote about the Claude Code frontend design plugin and how much it improved raw output. I've since stopped using it, because the models got good enough that I stopped noticing the difference.

Which leaves a question I can't fully answer: did the weights get better, or did the scaffolding around them get better?

There's real evidence for scaffolding. Anthropic ships a frontend-design skill inside Claude Design specifically to push the model away from that convergence problem. The Claude web app often produces nicer UI than raw Claude Code, and the gap there is instructions, not capability. Design Arena also publishes its exact system prompts, rendered from source, so this half of the question is testable by anyone with a spare afternoon.

My test used plain prompts with no design system, which measures each model's default taste. That's worth measuring, because the default is what most people get. It's also the floor. Every model here would do better work with an actual brief attached, and none of them got one.

If you want the single highest-leverage habit here: constrain before you generate, and use positive constraints rather than negative ones. Telling a model "no blue" fails often. Aliasing the whole blue, indigo and purple range to a named custom scale works, because you've replaced the default instead of forbidding it.

The workflow I'd actually use now

Generate simple single-file HTML across three or four models, several runs each. Cheap, fast, disposable. Pick the winner on taste alone and ignore the code quality completely, because you're throwing the code away. Then hand that HTML to whichever coding model you trust and convert it into a real React app, a Shopify theme, a WordPress theme, whatever you're building.

This beats one-model-does-everything because the two passes reward opposite things. The design pass wants variance, so you want several models producing several different answers. The build pass wants correctness, so you want one model that doesn't hallucinate imports.

And given how often my picks came down to a vibe, running more variations is worth more than choosing the right model. That's the actual takeaway from all of this.

Which does leave me open to the obvious charge, so I'll make it myself: my test does exactly one run per model, which is precisely the thing I'm telling you not to do. Fair. I was going for breadth across models, which is a different exercise from actually designing something. But if I ran it again I'd do three passes on the early learning brief at minimum, because that's the one where the spread between models was widest and a single sample tells you the least.

Do you still need to hire a designer?

In the plugin post I said yes, hire one for anything that matters. I'm less sure now, and I'd rather revise that in public than quietly leave it up.

Where I've landed: large company or brand-defining project, hire the designer. SMB or solo operator, the output has crossed the line where it stops looking like slop, and the money is probably better spent elsewhere.

The obvious objection is that AI design is homogenising everything, and the objection is correct. Kyle Chayka called it "Claudian sameness" in The New Yorker this July. Michal Malewicz's Slopless manifesto is an explicit rebellion against it. There's academic backing too: Doshi and Hauser found in Science Advances (volume 10, issue 28, July 2024) that generative AI makes individual writers more creative while making the collective output more similar to itself.

But look at what my advice actually is. Generate a spread, then choose between the options deliberately, even when you can't fully articulate the choice.

That's a direct antidote to convergence. Sameness is what you get from one model, one run, zero judgment. The judgment didn't disappear, it moved. It used to happen while drawing and now it happens while choosing. Smaller job, and I won't pretend otherwise, but it's still the part that decides whether the result is any good.

Grok Bot is xAI’s early-beta product for AI teammates. It launched on 11 August 2026 and right now you only get it with SuperGrok Heavy, Cursor Ultra, or Cursor Premium Teams. The basic idea is simple: you give these bots real work, they use your tools the way you would, and they only ping you when something needs approval. Official details are on the product page and the launch post.

I’ve been messing around with agent setups for a while, including the open-source ones, so this is my attempt to figure out what’s actually useful and what’s still half-baked.

What Is Grok Bot and How Does It Work?

You download a desktop or iOS app, create one or more named bots with roles (Chief of Staff, Sales Outbound, Inbox Manager, whatever), connect your tools once, and then just talk to them. The bots share a single persistent cloud computer. Cookies, sessions, files and CLI credentials are shared across every bot on the same account. Each bot gets its own screen so they can do visual work at the same time.

That computer keeps running after you close the app. You can watch a live preview of what they’re clicking and typing. Sensitive actions (sending email, making payments, that kind of thing) still need your explicit okay. You can also teach a routine by showing it once and then let the bot repeat it. In theory, anyway.

Under the hood it runs on the current Grok stack. Grok 4.6 (the agent-focused model with a 500k context window) is available in the broader system, including Grok Bot. More details live in the computer and apps docs and the Grok 4.6 documentation.

The practical difference is that you’re no longer stuck with chat-style suggestions. The bots can research, draft in your voice, update CRM records, triage an inbox, or pass work between themselves. That part feels different from pure language models.

Real Workflows People Are Running

In the first week a bunch of concrete examples showed up. Sales teams are running overnight pipelines: research prospects, score them, draft sequences in the user’s voice, then queue everything for approval. Ops people are doing inbox triage plus draft replies, weekly report generation, and expense chasing. Some treat a Chief of Staff bot as the router that hands tasks to specialist bots.

Engineering examples include bug reproduction and ticket filing. Content folks are building production and distribution pipelines. A few people are even using it for personal admin (bookings, subscription cleanup, that sort of thing). I haven’t asked one to cancel a gym membership for me yet, but the option is technically there.

The pattern that keeps coming up is multi-bot collaboration. One bot researches, another drafts, a third checks CRM state, and they pass context around in shared threads. It’s the closest thing I’ve seen so far to treating AI like junior colleagues rather than a single smart search box. Whether that mental model is actually better than a clean task list plus cron jobs is still an open question for me, but a lot of early users seem to like the conversational framing.

How Good Is It Right Now? Speed, Reliability & Limits

The UX is polished. Onboarding is smooth and the interface feels intentional. That is not nothing. Plenty of agent tools still look like they were designed by people who only talk to terminals.

The weak spots are more interesting. Pure computer use (driving the browser by screenshots) is still slow and unreliable for anything non-trivial. Simple tasks can feel frustrating, like watching someone type with one finger while narrating every keystroke. Heavy agentic sessions also burn through the available compute faster than you’d expect. There are reports of multi-bot runs hitting weekly limits after only a couple of hours of real work.

Access starts at $200 a month with Cursor Ultra or $300 with SuperGrok Heavy. Teams get in at $120 a seat. There’s no free or lower tier in the current beta. For people already paying those prices the extra cost is zero, but for everyone else it’s a real barrier.

I still think the managed cloud computer is the actual product. Everything else (chat interface, named bots, hand-offs) is packaging around that. The packaging is good. The computer itself isn’t yet as fast or solid as you’d want for production-grade workflows.

Grok Bot vs Hermes Agent and OpenClaw

This is the comparison that matters if you’ve already spent time with local agents.

Grok Bot locks you to the Grok models and charges a premium for the managed environment. Hermes and OpenClaw are free and open-source. You bring your own models (including Grok if you want), your own machine or server, and you own the whole stack. Setup is heavier and you handle security yourself. I wrote more about Hermes in this earlier post.

The honest split is simple. If you want zero setup and a polished “talk to your teammates” experience, Grok Bot is currently the cleanest option. If you care about cost, model freedom, and not handing credentials to a shared cloud computer, the open-source tools are still stronger. Some people are running both. They use Grok Bot for the easy stuff and keep Hermes or OpenClaw for the work that needs more control.

I don’t think the chat format itself adds massive new capability over well-designed tasks and repeating jobs. It’s mostly a UI and mental-model choice. That choice will matter more to some people than others.

Who Is Grok Bot Actually For (and Who It’s Not)

This is where my own thinking settled. The product is clearly aimed at people who don’t want to set up and secure a local agent. Office knowledge workers, GTM folks, founders who already live in Cursor or SuperGrok, and anyone who likes the idea of little workers that talk to each other will probably feel at home here. The conversational framing maps more cleanly onto how humans already work together than a pure task queue does. It feels a bit like managing a very polite, slightly over-eager intern team that never sleeps.

If you’re already deep into open-source agents and comfortable with the operational overhead, Grok Bot will feel expensive and constrained. The price and the current usage limits are the real hurdles. I also expect other companies to ship similar managed teammate products over the next year or two. This pattern isn’t going away.

For smaller teams trying to decide where agents fit, the broader landscape still matters. I looked at some of those options in this piece on AI agents for small business. And if you’re still evaluating the underlying models, my earlier Grok 4.5 review covers the previous generation.

Closing Thoughts

Grok Bot is interesting. The managed persistent computer plus real tool sign-in is a genuine step past pure chat agents. The teammate framing and multi-bot collaboration make the experience feel more natural for a lot of knowledge work.

It’s also still early. Computer use is slower than it should be, quotas disappear under load, and the price keeps it out of reach for many people who would otherwise try it. I’m not ready to call it a must-have. I’m also not writing it off. The idea of always-on AI workers that can hand work between themselves is clearly useful. We’re going to see more of this, both from xAI and from everyone else.

If you’re already on an eligible subscription, it’s worth spinning up a couple of bots and seeing how the workflows feel in practice. If you’re not, keep an eye on the open-source side and wait for the next round of improvements. Either way, the gap between “AI that suggests” and “AI that finishes the job” is getting smaller. And that is probably the part worth paying attention to.

I need to be upfront about something. I did not run my full test suite on this model. I tried, got about eight tokens per second on my M4 MacBook Air, and lost patience somewhere around the second prompt. That failure is most of what this post is about, because I suspect a lot of you are about to have the same evening I had.

What Qwen3.8-27B actually is

27.78 billion parameters, dense, including a vision encoder that nobody had announced in advance. It takes text, images and video in. Native context is 262,144 tokens, extensible to roughly a million with YaRN scaling if you're running it on a server.

The architecture is the genuinely interesting part, and it isn't a plain transformer. Across 64 layers, Qwen interleaves 48 Gated DeltaNet layers (linear attention) with 16 full Gated Attention layers, in a 3:1 ratio. Only those 16 full-attention layers keep a KV cache. That works out to about 64 KB per token, roughly a quarter of what a conventional 64-layer dense model would want.

If you only remember one number from this post, make it that one. Everything about whether this model fits on your machine comes back to KV cache growth.

Thinking mode is on by default, with a reasoning_effort knob set to xhigh out of the box. You can drop it to medium or low. You will want to. More on that shortly.

The benchmarks, and why I'd discount them

Qwen's own table looks great:

Jump from 13.3 to 42.2 on DeepSWE is not a rounding error. The generational gain over Qwen3.6-27B looks real and it's consistent across the whole table rather than one suspicious spike.

Now the part that made me rewrite my first draft. Every one of those numbers came from Qwen. Some of the benchmarks are in-house, some have corrected ground truth, and the competitor scores in their comparison were imported rather than re-run. As of the time I'm writing this, Artificial Analysis hasn't published an intelligence index for the 27B, and no lab has reproduced anything.

I originally wanted to write "we now have Opus 4.6 level intelligence running on consumer hardware." Then I read Qwen's own table properly. Opus 4.6 Max is ahead on Terminal-Bench 2.1 (78.2 to 73.0), ahead on GPQA-Diamond (91.3 to 89.2), and comfortably ahead on Humanity's Last Exam (40.0 to 30.8). The 27B wins some rows and loses others, on a scorecard the vendor produced. "Beats Opus" is a headline, not a finding.

What's true is still remarkable: a 28B model you can download for free is trading blows with a frontier API on coding benchmarks. That's the story. It doesn't need inflating.

What it feels like to use

I did play with it, just not systematically. It's very good. The benchmark direction matches the vibe, which is the most I'm willing to claim from an afternoon of poking at it.

Simon Willison, who has been running his SVG pelican test on everything for two years now, called it the best pelican he's seen from a model that runs on his laptop. He also noted the run took nearly 21 minutes and burned 22,276 reasoning tokens to produce 3,223 tokens of output. On an M5 Max. That is the model in one anecdote: strong, and exhaustingly thorough about it.

The most common complaint in the first week wasn't quality, it was verbosity. At default xhigh the thing meanders. People running it in agent loops report 10k tokens where a competing model uses one. There are also chat template bugs, including a fatal exception if you pass enable_thinking=false, so grab a community-fixed template before you wire it into anything serious.

Can you run it? The hardware maths

Short version: you need about 17 to 18 GB for a 4-bit quant, plus KV cache on top. 24 GB of unified memory is the practical floor, 32 GB is where it gets comfortable, and 48 GB is the value sweet spot.

Here's the Unsloth GGUF table, which is the one everyone is working from:

Add roughly 0.93 GB for the vision projector if you want image input, because it isn't bundled with the language GGUF. And don't go below Q4 on this model. Multiple people reported quality falling off a cliff, more than they expected for a 27B, and aggressive KV cache quantization broke at least one person's private eval outright.

KV cache maths at 64 KB per token: 8K context is about half a gig, 32K is 2 GB, 128K is 8 GB, and the full 262K is 16 GB on its own. Start at 8K and raise it while watching memory pressure. The million-token YaRN context is a server feature and static YaRN hurts short-prompt quality anyway.

The Apple Silicon specifics

Three things that surprised me.

First, Ollama's fast MLX backend requires more than 32 GB of unified memory, so a 24 GB Air can't use it at all. You're on the GGUF path: LM Studio if you want one-click, llama.cpp if you want control.

Second, that matters less than you'd think, because the "MLX is faster on Apple Silicon" rule mostly applies to small models. The MLX advantage is large under about 14B and collapses above 27B, where you become memory-bandwidth-bound and the runtime stops being the bottleneck. So pick GGUF and stop worrying about it.

Third, thermals. The Air is fanless. Sustained inference pins the GPU and the chassis heats up, and testing converges on roughly 60 to 70% of peak after eight to ten minutes of full load. For a chat session you'll never notice. For a long refactor you absolutely will.

Where I hit the wall

My M1 Mac Mini has 16 GB. It is not a host for this model, full stop. The 4-bit build needs 17-something gigabytes and the machine just swaps itself into uselessness. I run a 4B coder there and that's fine.

The M4 Air has 24 GB and it does load. It runs. It also produces fewer than ten tokens per second once you're a few thousand tokens in, which sounds survivable until you combine it with a model that wants to spend 20,000 reasoning tokens thinking about your question. I sat there watching text appear at reading speed and decided I had better things to do with my afternoon. The only measured number I've seen for base M4 silicon on this model is 5 to 6 tokens per second on a 32 GB Mac Mini, so I don't think I got unlucky.

Here's the part I'm actually annoyed about. Two years ago I looked at upgrading and talked myself out of it. The reasoning felt sound at the time: local models weren't good enough to justify the spend, and Apple had new chips coming. Both of those things were true. What I got wrong was the ordering. The models got good faster than the hardware got cheap, and now memory prices are ugly and the machine I want costs more than it did when I didn't want it.

If you're weighing this up now, the lesson generalises. Buy the RAM before you can justify it. With local inference the useful model always shows up before the affordable machine does, and you can't add unified memory later.

Who should actually buy hardware for this

Privacy is the whole argument, and it's a strong one. Apache 2.0 weights running fully offline is a different conversation with your compliance people than a hosted API's terms of service. If you're a doctor, a lawyer, an accountant, or anyone else who handles material that legally cannot leave the building, this is the first time the local option has been good enough to be boring rather than a compromise. I do a fair bit of AI consulting work in Sydney with exactly those professions, and the conversation has changed completely in about six months.

For that use case I'd buy hardware today. A maxed Mac Studio, a 48 GB M4 Pro Mac Mini (best value Mac for this specific model, and a desk-bound machine dissipates heat properly), a DGX Spark, or an RTX 5090 if you're on the Nvidia side. The 5090 numbers are in a different league: 200-plus tokens per second with NVFP4 and speculative decoding, versus my sad little eight.

What I wouldn't do is pretend this replaces a hosted API for everything. It isn't fast enough for long agent loops or batch work on a laptop, and the hardest 5% of tasks are exactly where a frontier model still earns its subscription. The pattern that works is hybrid. API for the difficult things, local for volume and for anything that can't leave your machine.

Why this is Apple's opening

Apple has had a rough couple of years in AI by every measure that gets written about. But they own the layer local inference actually needs, which is a lot of fast unified memory in a machine normal people already buy.

Think about what happens if the current trajectory holds. If a local model handles 95% of what you do in a day, what's the marginal case for a monthly subscription? I don't think the answer is "none," because that last 5% is where the value concentrates and it's the part you'd pay for anyway. But it changes the shape of the market a lot, and it changes it in a direction where the company selling you 128 GB of unified memory does very well.

The open question is memory prices. Either the current buildout overshoots and we get a glut, or the datacentre demand keeps eating supply and consumer RAM stays expensive for years. I'd bet on the glut, mostly because everyone building capacity right now is assuming everyone else won't. That bet has a poor historical record for the people making it and a good one for the rest of us.

Where I land

Qwen3.8-27B is the most important open release of the year if you have the machine, and a preview if you don't. I'm in the second group and it's my own fault.

Two things would change my read. Independent reproduction of those benchmarks, which nobody has done yet and which I'd want before repeating any of the vendor numbers as fact. And a quantization or a speculative decoding build that gets a 24 GB Mac to twenty tokens per second, at which point I'd stop writing about this model and start using it.

If you've got 32 GB or more, go get it. Set your context to 8K, use the official thinking-mode sampling parameters (temperature 1.0, top_p 0.95, top_k 20), drop reasoning_effort to medium, and tell me if it's as good as I think it is.

Day one with pi did nothing to change my mind. It felt sparse to the point of feeling broken.

Then I installed a handful of extensions, and about a week later I noticed something. I'd been splitting my work across Claude Code, OpenCode and Grok Build, and now pi had quietly absorbed most of it. Claude Code is the thing I open when I specifically want a Claude model. Everything else moved. What surprised me more is that the CLI isn't actually the interesting part. The SDK is. Once you can call an agent as a library from a cron job, with only the tools you handed it, you stop thinking about coding assistants and start thinking about small autonomous programs that happen to have a language model in the middle.

This post covers the install, the extensions that saved it for me, an honest account of the security model (it doesn't really have one), and a worked SDK example: a document processing agent that turns messy supplier invoices into validated structured JSON.

What is the pi coding agent?

Pi is a minimal, MIT-licensed terminal coding agent and TypeScript agent toolkit. It ships with four built-in tools (read, write, edit, bash) and a system prompt that, together with those tool definitions, comes in under 1,000 tokens. It supports 15+ model providers with mid-session switching, and everything beyond the core is meant to be an extension you write yourself.

It was created by Mario Zechner, who you might know from libGDX or Spine, and it's now stewarded by Earendil Inc., the company Armin Ronacher co-founded. The repo lives at earendil-works/pi. Star count is somewhere north of 80,000 as I write this, though it's climbing fast enough that whatever number I put here will be wrong by the time you read it.

The design thesis is aggressive minimalism. No MCP. No sub-agents. No plan mode. No built-in todo list. No permission sandbox. Zechner's position is roughly "if I don't need it, it won't be built," and the intended response to a missing feature is that you ask pi to build it for you, then reload.

Quick naming note, because search engines are hopeless at this: this pi is not Inflection's Pi chatbot, not a Raspberry Pi, and not github.com/badlogic/pi, which is a completely different CLI for managing vLLM GPU pods. Zechner has said he picked an un-Googleable name as a joke so nobody would ever use it. That plan failed spectacularly.

Install pi coding agent

Or curl -fsSL https://pi.dev/install.sh | sh if you'd rather have the standalone binary. You need Node 20+ or Bun. It runs on macOS, Linux, Windows, and apparently Termux on Android, which I have not tried and probably never will.

Then either /login for subscription OAuth, or drop API keys into your environment or ~/.pi/agent/auth.json.

One trap worth knowing about: the packages moved from the @mariozechner/* scope to @earendil-works/* back in May 2026. Any tutorial or Stack Overflow answer older than that will point you at deprecated packages. pi update --self handles the migration if you were already on the old scope.

Pi coding agent setup

Context files work the way you'd hope. Pi loads AGENTS.md hierarchically, global first then project, and it also reads CLAUDE.md, so migrating an existing repo is genuinely a no-op. Nice touch.

Sessions auto-save to ~/.pi/agent/sessions/ organised by working directory. pi -c continues the most recent one, pi -r gives you a browser, --fork branches. They're trees rather than flat logs, so you can rewind a bad state, go do a side quest, and merge a summary back. Stored as plain JSONL, which means you can grep it, which I appreciate more than I expected to.

The first time you open a project with local .pi/ config, pi asks whether you trust it. Remember that prompt, because it's more limited than it sounds and we'll come back to it.

A week in: what's missing, and the extensions that fixed it

My first two hours with pi were mildly unpleasant. I'd internalised a bunch of conveniences from Claude Code, OpenCode and Grok Build without noticing how much I was leaning on them, and pi has none of them. No todo list, so no sense of what the agent thinks it's doing across a long task. No sub-agents. Minimal status feedback. It felt like being handed a text editor with no syntax highlighting after five years of an IDE.

Then I read the reasoning, and some of it is actually good. MCP servers burn 13,700 to 18,000 context tokens before you've typed a single character, and progressive disclosure through a CLI plus a README does the same job for almost nothing. Sub-agents are opaque black boxes with lousy context transfer. Built-in plan and todo state confuses models, where a PLAN.md file does the job and survives across sessions and tools.

I buy the MCP one completely. I'm less convinced on todos, which is why the first thing I did was add one.

My current install list, in rough order of how much I'd miss them:

• @juicesharp/rpiv-ask-user-question. Lets the agent stop and ask me a clarifying question instead of guessing. This is the one that changed the most. Half of a bad agent run is the model committing to the wrong interpretation in the first thirty seconds, and a single question at the right moment saves the whole session.
• @juicesharp/rpiv-todo. Ironically the exact feature the project deliberately omitted. It's a todo list for the model, rendered as a live overlay, and it survives /reload and compaction. That last part matters more than it sounds.
• pi-lens. Real-time code feedback: LSP, linters, formatters, type checking. Pi with no language server is a bit like coding in Notepad, and this closes the gap without dragging in a whole IDE.
• pi-agent-browser. Browser automation through agent-browser rather than Playwright. I've switched to it for basically everything now. It's dramatically more token-efficient than driving Playwright through an agent, which historically meant dumping enormous DOM snapshots into context and watching the model drown.
• @tintinweb/pi-subagents. Claude Code style sub-agents. I use them narrowly, for fanning out independent read-heavy work, which is the case where the context-transfer objection doesn't really bite.
• pi-powerline-footer. Purely cosmetic. I regret nothing. Seeing model, thinking level and token spend at a glance genuinely changes how carefully you spend context.

Installing is one line each (pi install npm:pi-lens and so on), and that's the real point. The individual extensions matter less than the fact that assembling my ideal setup took an evening, and the extensions API is just TypeScript with hot reload. You can bundle your own into a Pi Package and install it elsewhere from npm or git. When your agent harness is missing something, you don't file an issue and wait.

One caveat that follows directly from that openness: pi packages run with full system access, extensions execute arbitrary code, and skills can instruct the model to run anything. Read the source before you install a stranger's package. I'll come back to this.

Now the part I can't prove. My gut says pi gives me slightly better results than the other CLIs on comparable tasks. My best guess is that it's the tiny system prompt and the absence of clutter, since less junk in the context window means more of the model's attention lands on my actual problem. That lines up with everything I've written about context engineering for AI agents. But it's a feeling, not a measurement, and I'm not going to dress it up as one. Let's look at numbers other people collected instead.

Pi coding agent vs Claude Code

Minimal doesn't mean weak here. Pi placed second on Terminal-Bench running Claude Opus 4.5, beaten only by the benchmark's own minimal harness, and it did that without MCP, sub-agents or plan mode. Databricks' harness benchmark from 5 August 2026 is more interesting though. Running the same model at the same thinking effort through different harnesses produced cost differences of more than 2x while quality stayed flat, with simple harnesses like pi often coming out best. Their codebase, their tasks, so it's not a universal law. But "the harness materially changes your bill" is worth sitting with.

Now the part that will actually decide this for most people: money.

On 4 April 2026 Anthropic's billing enforcement landed, and third-party tools started drawing from extra usage instead of subscription limits. Pi's own provider docs still state it plainly: Claude Pro/Max auth works, but third-party harness usage is billed per token, not against your plan. Translation, you can log in, you just don't save any money. There have been partial reversals since, so verify before budgeting around it. Codex, xAI and Copilot logins all work normally and still save you money.

Which brings me to my actual setup, because it's the only honest answer to "is this a Claude Code alternative." I run Grok, ChatGPT/Codex, OpenCode Go and Qwen through pi. Claude stays in Claude Code, where it's still covered by my subscription. The result is that I barely open Claude Code anymore, because most of my agent work now runs on my OpenCode Go subscription with DeepSeek V4 Flash 0731, which is very good and absurdly cheap. The recently released version is significantly better than the preview I tested in my DeepSeek V4 review.

Pi didn't replace Claude Code for me. It made my Claude subscription optional, which is a different and slightly more uncomfortable thing.

Pi coding agent with Ollama and local models

Local models plug in through an OpenAI-compatible entry in models.json, so Ollama, LM Studio, vLLM and llama.cpp all work without ceremony. Set expectations accordingly: on a 24GB M4 MacBook Air they're fine for scoped work like writing an extension or summarising a file, and they fall over on agentic work across a large repo about forty minutes in, when the context window folds. Unless privacy is the entire point, a cheap hosted model is the better trade.

Pi is YOLO by default, and it means it

Two things happened in my first week that I want you to know about before you install this.

First: I was mid-task on a new project, and pi decided it needed examples of what I was trying to build. So it went and searched a bunch of folders across my machine looking for similar code. Not the project directory. My machine. It didn't ask, it didn't announce it, it just did it and then carried on like nothing had happened. To be fair, the examples it found were relevant. That's almost worse.

Second: I built a skill for an image generation CLI I'd written in Go, and pi installed it globally, for every agent, without checking with me first.

Neither of these is a bug. From the README: pi has no built-in permission system for restricting filesystem, process, network, or credential access, and by default it runs with the permissions of the user and process that launched it. Zechner's rationale is that once an agent can write and execute code, permission dialogs are mostly security theatre, and you cannot solve the read-data plus execute-code plus network trifecta without severing the network, at which point you've built a very expensive autocomplete. He's not wrong, exactly. I've made roughly the same argument about --dangerously-skip-permissions, which is the setting everyone turns on by day three anyway. Pi skips the pantomime and starts there.

And remember that project trust prompt? It's an input-loading guard. It stops a repo silently changing your settings and extensions before you approve. It does not restrict what any tool does once you're working, and prompt injection from repo files or command output is explicitly treated as expected local-agent risk.

So: containerise anything you wouldn't personally review. The docs cover three patterns. Gondolin is the nicest, routing tools and shell commands into a local Linux micro-VM while pi and your provider credentials stay on the host. Plain Docker is the simplest, with the caveat that your API keys go into the container. NVIDIA's OpenShell gives you policy-level control and can keep raw keys outside the sandbox entirely. My line is simple: if you're running an unattended loop where you won't read every diff, and recursive agent loops are exactly that, a sandbox isn't optional.

Credit where it's due, though. The supply-chain hygiene is unusually serious for a project this young: exact version pinning, min-release-age=2 to avoid same-day dependency releases, a published shrinkwrap, --ignore-scripts on documented installs, a scheduled audit workflow. The risk here is at runtime, not in the dependency tree.

Building with the pi SDK: a document processing agent

This is the shift that took pi from pleasant to interesting.

Pi runs in four modes. Interactive is the TUI you've been reading about. Print mode (pi -p "...") is single-shot for scripts. RPC mode speaks JSON over stdin and stdout, which is how you'd drive it from Go or Python. And then there's the SDK, which runs the agent in-process inside your own Node program. That last one is the one nobody talks about enough.

It's a library you call in-process:

That's a complete, working agent. Nine lines of setup and you have the same loop that powers the CLI, streaming into your process.

The session object is where the useful surface lives. prompt() sends a turn and resolves when the agent settles. steer() interrupts mid-stream to redirect it, which is the programmatic version of typing while it's thinking. followUp() queues a message to be consumed once it's idle. setModel() and setThinkingLevel() change models mid-session, so you can run cheap extraction on a small model and escalate to something bigger only when a document fails validation. compact() summarises history when you're running long. abort() and dispose() do what you'd expect, and you'll want both wired to your job timeout.

SessionManager decides where conversations live. inMemory() for throwaway batch jobs, create(cwd) to persist to disk, open(path) to reload one. Sessions are trees, so branch() lets you fork a conversation, try something, and keep or discard it. One gotcha here that bit me: session replacement (new session, resume, fork, import) lives on AgentSessionRuntime via createAgentSessionRuntime(), not on AgentSession. And because subscriptions attach to a specific session object, you have to re-subscribe after any replacement or your event handler goes quiet with no error.

ModelRuntime owns credential resolution and model catalogues. It reads your existing ~/.pi/agent/auth.json, which means an SDK program inherits whatever you already logged into on the CLI. Convenient for local development, and something to think about carefully before you ship it anywhere.

So what can you actually build with this? Code review bots that comment on PRs. Scheduled maintenance loops that upgrade dependencies overnight. Custom TUIs and web UIs on top of the agent loop. CI steps that triage a failing test before a human looks. Shopify built pi-autoresearch, an autonomous optimisation loop, as a pi extension, and reported unit tests running 300x faster and roughly 20% faster React component mounting off the back of it. The most famous example is OpenClaw, which is built on pi's SDK and is, at last count, the most-starred repository on GitHub by a comical margin.

The SDK docs are solid and there are working examples in the repo ranging from minimal to full control. Pin your versions, though. The API is stabilising but not frozen, and the scope migration already broke a generation of tutorials.

The example I'll build here is document processing, because it's the least glamorous and most universally useful thing a small business has. Drop a supplier invoice into a folder, get validated structured JSON out. It's the kind of boring automation I keep recommending in my writing on AI agents for small business, and it exercises the parts of the SDK that matter: custom tools, structured output and tool restriction.

The document layer, and one myth to kill first

anydoc is Firecrawl's Rust document parser, MIT licensed, and it converts 14 office formats (docx, xlsx, pptx, odt, rtf, epub, csv, pdf and friends) to clean Markdown in under 5 milliseconds per document. No ML models, no external services, entirely local. Detection is content-based rather than extension-based, so a .pdf that's secretly a Word file still converts.

It does not do OCR, and it does not extract to a schema. This is the single most common misconception about it and the entire architecture depends on getting it right. anydoc reads the text layer of a PDF. Hand it a scanned or photographed invoice and you get back ConvertError::Unsupported.

So the design is two-path. anydoc as the fast local path for digital files, and Firecrawl's hosted /v2/parse endpoint with mode: "ocr" as the fallback when anydoc throws. Be clear-eyed about the trade: the moment you hit /parse, your supplier invoices leave the machine. For accounts payable data that's a decision to make deliberately, and if you want a fully local story you'd pair anydoc with Docling or a local vision model instead. Also worth noting that anydoc launched this month, so edge cases on weird real-world files are going to surface.

Forcing structured JSON out of a pi agent

Pi has no native response-schema option. There's no responseFormat on createAgentSession() or on prompt(). I looked. Don't waste the twenty minutes I did.

The idiomatic pattern is a terminating tool: you define a tool whose parameters are the schema you want back, and whose execute() returns terminate: true.

Three things here cost me time, so let me save you some.

Import from bare typebox, not @sinclair/typebox. The current @earendil-works packages use the bare one. Older forks and tutorials use the scoped one, you'll get confusing type errors, and you'll blame yourself.

terminate only fires when every finalised tool result in the batch is terminating. Pi runs tool calls in parallel by default, so if the model calls your emit tool alongside anything else in the same batch, the run keeps going.

And content is what the model sees while details is for your code. That split is the nicest small design decision in the whole SDK. Keep the LLM-facing text short and human, stash the machine-readable payload in details, and read it back off the turn_end event's toolResults or the tool_execution_end event's result.

Writing custom tools that do real work

The emit tool above is just a shape. The tools that earn their keep are the boring deterministic ones.

The GST validator, then. Notice how little of it involves the model:

A few rules I'd defend to anyone.

Never let the model do arithmetic you care about. Give it a function that does the maths and let it reason about the result. Its job is interpretation and edge cases. Same goes for the ABN checksum, duplicate detection and date parsing.

Write the description and promptGuidelines for the model rather than for your future self. These fields are the entire interface between your code and the agent's decision making. "Check Australian GST arithmetic, call this before emitting" is a usable instruction. "Validates GST" is useless. This is context engineering at the tool level, and it's where most of the tuning time goes.

Return errors by throwing. Returning an error-shaped object never sets the error flag on the result, which is the sort of thing you discover at 11pm. If the tool genuinely failed, throw from execute().

And use StringEnum from @earendil-works/pi-ai rather than TypeBox's Type.Union of literals for enum fields. The union form breaks Google's API. It'll work fine on Anthropic and OpenAI and then fall over the day you switch models, which rather defeats the point of a model-agnostic harness.

Give the agent your tools and nothing else

This is where the YOLO problem gets solved properly, and it's my favourite thing about the SDK.

noTools: "builtin" strips read, write, edit and bash entirely. The agent now physically cannot touch your filesystem or run a shell command. It can call four functions you wrote, and that is the complete universe available to it. There's also an allowlist form (tools: ["read", "grep"]) if you'd rather have a read-only agent that can still poke around.

Think about what that buys you. The same harness that wandered off and scanned my home directory unprompted is, in this configuration, incapable of doing so. Not discouraged by a system prompt. Incapable, because the tools don't exist. That's a much stronger guarantee than any permission dialog, and it's the argument for the SDK over the CLI in production: you define the box, and the agent has no way out of it.

Then subscribe to the event stream for your audit log. You get message deltas, tool execution start and end, turn boundaries, compaction, retries. Log the lot. When a client asks why the agent flagged their supplier's invoice, you want the answer, and you want it without re-running anything.

Where to stop

The agent produces a validated draft and then it stops. That's the whole design.

The obvious next move is a create_draft_bill tool that posts to Xero with Type: "ACCPAY", Status: "DRAFT" and an idempotency key derived from the supplier ABN plus invoice number, so a network retry can't create a duplicate. That's a separate post, because doing it properly means covering ATO tax invoice requirements, the 47% no-ABN withholding rule, ABN Lookup verification and out-of-band confirmation for bank detail changes. Business email compromise accounted for over three billion US dollars in reported losses in the FBI's 2025 IC3 report, and almost all of it hinges on someone changing bank details in a real invoice thread. An agent must never auto-pay. Not ever.

Same pattern, different tools, is how the e-commerce agents I've been building this month are structured. Narrow tool surface, deterministic validation, a hard stop before anything irreversible.

So should you switch?

Run it as a second agent for a week. Don't rip anything out. Point it at a real repo, use /model to swap providers mid-session, and ask it to build the one workflow feature you miss. If after a week you're reaching for it by default, migrate. If you're not, you've lost an hour.

It's a good fit if you're comfortable in TypeScript, you want to do your own context engineering, and you have non-Claude subscriptions to spend. It's a bad fit if you want safety rails out of the box, or if your entire workflow is built around a Claude subscription you're not willing to restructure.

It's also not universally loved. The complaint I've seen most from people who bounced off it is that it doesn't finish the job the way Claude Code or OpenCode do. Take that seriously, along with the DIY burden: a minimal core means you do the assembly.

As a CLI, the pi coding agent is a peer. It sits comfortably alongside Claude Code, Codex, Grok Build and OpenCode, and I'd be fine using any of them. The differences are real but small, and my gut feeling about better results is a gut feeling.

As an SDK it has no real equivalent. A minimal, model-agnostic, MIT-licensed agent you can embed as a library, restrict to exactly the tools you wrote, and drop into a cron job in about fifty lines is a different category of thing to a coding assistant.

The CLI got me to install it. The SDK is why it's still here.

People had been expecting something. Fable 5 was the flagship, but it costs $10 in and $50 out per million tokens, and over the past couple of weeks GPT-5.6 Sol, Grok 4.5, Kimi K3 and Qwen 3.8 all landed close enough on quality that paying Fable prices for everyday work stopped making sense. When your competitors are almost as good for a fraction of the money, the thing you need isn't a smarter model. It's a better deal. Opus 5 is exactly that: $5 in, $25 out, and Anthropic's own pitch is that it comes close to Fable 5's intelligence at half the price. Axios noted it's the fourth Claude 5 release in under two months, which tells you how the race has shifted from capability to economics.

So the question isn't really "is Opus 5 good". It obviously is. The question is whether it's better than what everyone else is charging less for.

I ran it against GPT-5.6 Sol and Grok 4.5 on four builds: an art gallery site, a working spreadsheet app, an image codec written from scratch in Go, and an audit of my own open source agent framework. Wall clock times recorded, no interventions, one prompt each.

Short version: Opus 5 wins on points, GPT-5.6 Sol is right behind it and beat both on the hardest test, and Grok 4.5 is fast enough that the quality gap matters less than you'd expect.

The two web builds are up at llm-test-0726.thomas-wiegold.com if you'd rather form your own opinion before reading mine. Design is the one thing you shouldn't take my word on.

Why these four tests

My old test course is dead, and Opus 5 is not what killed it. It had been dying since Fable 5.

The set was four prompts I'd used for about a year: two landing pages, a poker simulation in Go, and an audit of thomas-wiegold.com. The Qwen 3.8 Max review was the last one I ran on it, and the cracks were showing there already. Frontier models passing everything is fine. Open weight models passing everything at the same level is the end of the road, because at that point I'm comparing font choices and squinting at padding, which is taste with extra steps, not evaluation.

Claire Vo, writing her Opus 5 review in Lenny's Newsletter, calls this an "intelligence overhang". When capability saturates on the tasks you actually hand these things, the variables that matter become effort, latency and price. So the new tests had to be harder, and they had to resemble work someone would pay for. Here's what I landed on.

Art gallery website. Open creative brief, go all out, no spec and no wireframe. Not a challenge, and I won't pretend otherwise. Every current model passes. It's a pure design test: does the model have taste of its own when you refuse to give it constraints.

Spreadsheet web app. Excel-inspired, React, fully functional. Formulas, cell references, selection, undo. I expected this to be the hard one. It wasn't, which I'll get to.

Image codec in Go, from scratch. Beat PNG on compression, ideally approach WebP. No libraries doing the real work. This is the actual intelligence test of the four, and the only one where a model has to invent something rather than assemble something.

Audit of agentiny. My open source agent framework, swapped in for the website audit because thomas-wiegold.com is tuned to death and returns a polite list of nothing. Agentiny is young and has real headroom.

Same prompt to each model, wall clock from submit to final output, no nudging. If a model went off the rails, that's the result. Standard caveat: n=1 per model per test, so what I'd defend is the shape of the results, not the decimals.

The four builds, model by model

Test 1: Art gallery site, go all out

Nobody fails this one. That's the point of including it. Every current model can build a decent gallery site, so what's left to measure is taste, and taste doesn't show up on a leaderboard.

Grok 4.5 finished in five minutes. Opus 5 and GPT-5.6 Sol both took roughly half an hour.

That's a six-fold difference and my first instinct was that Grok had cheated somehow. It hadn't. The site was good. Clean, modern, nothing embarrassing, and if you showed it to a client without context they'd probably sign off on it.

But put it next to the other two and the extra 25 minutes are visible. Both Opus and GPT produced something more considered. Opus went deeper on detail: a horizontal scroll section in the middle that actually worked properly, and it shipped a light theme without being asked, which is the kind of thing I usually have to go back and request.

GPT-5.6 Sol went the other way. Simpler layout, but genuinely artsy in a way that suited the brief. The typography was excellent. If I actually owned a gallery, I think I'd pick the GPT site, and that surprised me because on paper it did less. All three are live on the results site, and this is the test where I'd most expect people to disagree with me.

Takeaway: on open creative briefs, thinking time converts into quality. Hold that thought, because it doesn't hold for long.

Test 2: Excel-style spreadsheet app in React

Grok 16 minutes. GPT 31 minutes. Opus just under 50.

All three shipped a working, usable, cut-down Excel. One shot each. I genuinely did not expect that, and I'd designed this test assuming it would be the one that separated them. Formula evaluation, cell ranges, keyboard navigation, all of it functioning on the first attempt. "Build me a spreadsheet app" was a joke prompt eighteen months ago. Now it's a Tuesday afternoon. They're on the results site too, and unlike the gallery test these you can actually poke at until something breaks.

Opus 5 was the best of the three. Most detailed UI, most thought given to the fiddly interaction bits, the one I'd want to keep working on. GPT was the weakest, thinner on UI polish and with genuinely odd undo behaviour that I'd have had to fix before showing anyone.

Grok landed almost exactly where Opus did. Not quite, but close enough that I had to look hard to find the gaps. And it did it in a third of the time with a fraction of the tokens.

So who wins? Opus produced the better artifact. But if I'm paying for this, Grok delivered maybe 95% of the result at a third of the cost, and I think that's the winner on any measure I'd defend to a client. This is also the test that breaks the pattern from the gallery site: more thinking time did not produce a proportionally better app. It produced a slightly better one, expensively.

I'm keeping this test even though it failed at the job I designed it for. It doesn't separate the models on capability anymore, but the spread on time and tokens is enormous (16 minutes to 50), and the small details are where you see who was actually paying attention. Undo behaviour is not a headline feature. It's exactly the kind of thing that quietly determines whether the code is worth keeping.

Test 3: Image codec in Go, from scratch

This is where it got interesting, and where the times inverted completely.

GPT-5.6 Sol: 21 minutes. Grok 4.5: 33 minutes. Opus 5: one hour and twelve minutes.

Opus spent nearly four times as long as GPT on this. Here's what everyone produced.

GPT won, clearly. Real compression, genuinely good results on simple images with flat colour regions and clean edges. It struggled on busy, complex, noisy photos, which is roughly PNG's own failure profile, so it didn't hit the WebP target. But it was the only one that produced something I'd call a working codec, and it did it in 21 minutes. That's the single most impressive result of the whole course.

Opus was second. It compressed. Files came out smaller than the input, the thing worked, but it was well behind PNG and never in the running for WebP. Seventy-two minutes for a codec that loses to a format from 1996 is a rough trade.

Grok failed outright. Not "did worse", failed. Average output larger than the uncompressed source. It produced a program that confidently made files bigger, which is a fascinating thing for a compression algorithm to do.

Now the part that bothers me. Opus 5's headline independent result is ARC-AGI-3, where ARC Prize verified 30.2% against GPT-5.6 Sol's 7.8%, roughly four times better on a benchmark built specifically to resist memorisation and measure novel reasoning. That's the one number in the whole launch that wasn't run on Anthropic's own harness. If any test on my course maps to that capability, it's this one. Invent a compression scheme, no library to lean on, no Stack Overflow answer to pattern match.

And GPT beat it. Not narrowly.

Two honest caveats before anyone quotes me on this. First, the ARC comparison isn't effort-matched: Opus 5's 30.2% was measured at High effort because of the short pre-launch window, while Sol's 7.8% was a Max run. The gap is still huge, but it's not a clean duel. Second, one codec attempt doesn't refute a verified benchmark, and it's entirely possible Opus wins this on a best-of-five. I'm not claiming ARC-AGI-3 is broken. I'm saying a benchmark measuring novel reasoning in the abstract did not predict novel reasoning in Go, on a Tuesday, with my prompt.

Of the four tests this is the one I'm most interested in developing, because it's the only one that hands you numbers instead of opinions. Compression ratio against a fixed image set, measured the same way every time. That's a benchmark, or close enough to one that I could turn it into a real benchmark with a bit of work. I need to experiment with the prompt and the image set first, since right now it's one attempt with a loosely worded target, and the difference between "beat PNG" and "here are twenty images and a scoring function" is the difference between an anecdote and a number.

Test 4: Auditing the agentiny codebase

Time wasn't a factor here. All three were quick, and none of the differences were meaningful. If you want to see what they were looking at, agentiny is on GitHub.

For grading I ran the three audit reports through several other LLMs and asked them to rank the findings by importance and coverage, then compared that ranking to my own read of the codebase. They agreed with each other and with me, which is about as good as blind grading gets when you're the one who wrote the code.

Opus 5 came out on top. It found the most genuinely important problems, the ones I'd actually prioritise. GPT-5.6 Sol was very close behind, close enough that on a different day it might flip. Grok was the weakest. It missed several things the other two caught.

But, and this is the bit worth internalising, Grok also found some minor issues that neither of the others mentioned. Not important issues, but real ones.

This matches something I've said before: run audits across multiple models and consolidate the results. Model quality ranks cleanly, coverage doesn't. Even the model that comes third has non-overlapping findings, because these things all have different blind spots. CodeRabbit's independent eval found the same shape from a different direction, measuring Opus 5 at their cleanest ever precision (39.3%) but with lower recall than their baseline (55.2% against 61.1%) and around four times the nitpicks. Their conclusion was that it's a precision specialist that pairs well with a recall model, not a drop-in sole reviewer. Two very different methodologies, same practical advice: don't use one model.

What the benchmarks predicted vs what I actually got

Here's the honest comparison. On Artificial Analysis, which is the closest thing we have to an independent aggregate, Opus 5 sits at number one on both the Intelligence Index (61 at max effort, against Fable 5's 60 and GPT-5.6 Sol's 59) and the Agentic Index (55.3 against GPT's 54.0).

Look at those margins. One point. One point three. And that 61 is a max-effort number, which is not the setting most people will actually run, so the medium-effort score of 56 is probably closer to what you'd feel day to day.

My results have exactly that shape. Opus edges ahead across the four tests, but not by much, and not on every one. A one point aggregate lead is not something you feel in daily work, and my course confirmed that more than it challenged it. Where the benchmarks held up: the audit and the spreadsheet, both won by Opus, both by a margin small enough to argue about. Where they misled me: the codec, badly.

A caveat on the rest of the numbers. Anthropic's announcement leads with SWE-bench Verified at 96.0% and Frontier-Bench at 43.3%, and those are impressive, but most of the headline coding and agentic figures are vendor-run on Anthropic's own harness. Agentic benchmarks are notoriously harness-sensitive. Treat cross-model gaps as directional. Even the Artificial Analysis suite, which is the cleanest signal in the set, was co-run with Anthropic pre-release.

Then there's cost, which for me is the actual story. Opus 5 is $5 per million input tokens and $25 output, exactly half of Fable 5 at $10/$50, while Artificial Analysis measures it at comparable intelligence for 26% lower cost per task. I liked Fable 5 when it launched. I don't think it's worth the money anymore, and Opus 5 is the reason.

One thing to budget for: thinking tokens bill as output, adaptive thinking is on by default now rather than something you switch on, and Artificial Analysis flags Opus 5 at max effort as very verbose with a time to first token of nearly 70 seconds. The rate card didn't change. Your bill might. That 72 minute codec run wasn't free, and the effort dial (low, medium, high, xhigh, max) is a bigger cost lever than the model choice.

Verdict: which model for which job

Opus 5 and GPT-5.6 Sol are close. Both are a clear step above Grok 4.5 on quality. Grok is meaningfully faster and cheaper. That's the whole comparison in three sentences.

But the useful version is more specific than that.

The speed/quality trade depends entirely on the task

I like fast. I'm impatient, and a five minute turnaround changes how I work in a way that a marginally nicer output doesn't. For a large slice of everyday work, Grok is enough. Landing pages, CRUD, refactors, the stuff that makes up most of a week. My longer Grok 4.5 review goes into where it holds up.

Reach for Opus or GPT when the task needs depth, novelty or judgment. And keep the codec test in mind as the cautionary case, because Grok didn't produce a worse codec, it produced a non-codec. The failure mode of a cheaper model on genuinely hard work isn't "slightly worse", it's "confidently wrong", and you have to be able to tell the difference.

The planner/coder split I want to try next

The obvious experiment: Opus 5 or GPT-5.6 Sol plans and reviews, Grok writes the code. You buy judgment where judgment matters and volume where it doesn't.

I haven't tested this. It could easily fall apart on context handoff, which is its own discipline and something I've written about in context engineering for AI agents. But the audit result is suggestive, because the models that graded best were the ones that spotted the important problems, and that's exactly the reviewer role.

Where this leaves Anthropic

Opus 5 was necessary. With GPT-5.6, Grok 4.5, Kimi K3 and Qwen 3.8 all shipping in the same window, Anthropic without Opus 5 would be behind on price-performance rather than narrowly ahead on it. Releasing it was the right call and the pricing is the reason, not the benchmarks.

The broader point I keep coming back to: most of what we do every day doesn't need the intelligence ceiling. Paying for the ceiling on routine work is waste. That applies to Fable 5, and increasingly it applies to Opus 5 and GPT-5.6 Sol too, when a cheaper model would have done the job.

Next on the list is the same four tests against Kimi K3 and Qwen 3.8 Max. I left them out here purely because this took a full day and I ran out of one. A Chinese model comparison is its own article, and depending on timing it might end up including DeepSeek V5 too.

Quick naming note before we start. The official name is Qwen3.8-Max-Preview (product ID qwen3.8-max-preview). Everyone's calling it "Qwen 3.8 Max", including me in casual conversation, but keep in mind this is a paid preview available through Alibaba's Token Plan, not a finished release. Open weights are promised "soon", with no date and no licence. We'll see.

This month has been absolutely insane for AI releases. I still owe you reviews of ChatGPT 5.6 and Kimi K3 (I've covered Grok 4.5 and Claude Fable 5 already), but I'm reviewing Qwen first for a simple reason: Qwen has quietly become the most important model family for local AI. If you run agents locally, chances are you're running Qwen. So when they ship a new flagship, I want to know what's coming down the pipeline to the smaller models I'll eventually run on my own hardware.

What We Actually Know (and Don't)

Here's the honest state of play as of today.

Confirmed: 2.4T total parameters (vendor figure), sparse MoE, multimodal (text, images, video, documents). It's the Qwen team's first multimodal model above 1T parameters. The endpoints speak both OpenAI and Anthropic protocols, so it drops straight into Claude Code, Cursor, OpenCode and friends without changing your harness.

Not confirmed: basically everything else. Context window (the 1M figure floating around is community lore, not an official spec), active parameter count, knowledge cutoff, benchmarks. Every benchmark number you'll see circulating "for" Qwen3.8 is actually a Qwen3.7-Max number. The only independent evaluation I've found is Trilogy AI's StackPerf run, a single blind head-to-head where Qwen3.8-Max-Preview scored 80 against Kimi K3's 83. One run, one task, so take it as a data point, not a verdict.

Qwen3.8-Max-Preview at a glance:

My Test Suite: Same 4 Prompts, Every Model

If you've read my other reviews, you know the drill. I run the same four tests with the same prompts on every model so I can compare directly: a Sydney coffee roaster landing page, a pop culture themed clothing store, a Texas Hold'em poker simulation written in Go (six AI players with distinct personalities, realistic betting, 1000 hands, statistics at the end), and a full audit of this website. Same methodology as my Grok 4.5 and Gemini 3.5 Flash reviews.

The Results

Coffee Roaster Website

This took a looong time. The model sat there thinking for several minutes before doing anything at all, and it stayed slow throughout. Over 30 minutes total, the longest this test has ever taken.

But here's the thing: the result might be the best I've ever gotten from this prompt. It went into obsessive detail after the first HTML hit the disk, testing repeatedly with Playwright and adding little touches each pass. I got things I never asked for: a shopping cart, a wholesale section with registration, an Instagram link, coffee sorting and search by roast level. In a different context that scope creep could be a problem. Here, it made the site feel real. Visually it was tasteful too. Nice fonts, nice colours, good animations, solid layout.

Pop Culture E-commerce Store

Even slower than the coffee site. At this point I started wondering whether the model is just fundamentally slow, or whether Alibaba simply didn't have enough servers ready for launch day (I was testing hours after release, to be fair).

The obsessiveness continued. It checked buttons, menus, animations and aspect ratios with Playwright like it was getting paid per test run. The result was great: polished, detailed, good typography. I still slightly prefer the style Gemini 3.5 Flash gave me on this prompt, but that's taste. If the speed improves, this is a serious frontend model.

Poker Simulation (Go)

One hour and twenty minutes. I made coffee. I answered emails. I briefly considered learning to whittle.

But it one-shotted it. Only two other models have managed that so far, Claude Fable 5 and Grok 4.5, which makes Qwen3.8-Max the third member of a very small club. What's interesting is that the core simulation seemed done after about 20 minutes. The remaining hour went into fine-tuning the six player archetypes (the tight rock, the crazy maniac, and so on) until the betting behaviour looked genuinely realistic. And it did. The final simulation output showed clearly distinct, believable playing styles across 1000 hands. The statistics were good, though not as detailed as what Grok 4.5 produced on the same test.

Website Audit

35 minutes, the slowest audit in this test series. It didn't find any new errors on thomas-wiegold.com, but it flagged potential pitfalls I'd genuinely never thought about. And honestly, an audit is the one test where slow and thorough is exactly what you want. My tip if you care about this stuff: run the same audit through Fable, ChatGPT 5.6, Kimi K3 and Qwen3.8-Max, then consolidate the results into one file to work through. The overlap tells you what's real, the differences tell you what's interesting.

The Speed Problem

Let's not dance around it. In every single test, this was the slowest model I've used. The open question is why. It could be the model itself being extremely verbose in its thinking. It could be day-one server strain, since I tested hours after launch. Alibaba calls the preview "continuously evolving", so this may well improve.

The trade-off is real though. For rapid iteration, the speed is genuinely annoying. For important one-shot builds where you want maximum thoroughness, watching it Playwright-test every button for an hour starts to feel less like slowness and more like diligence. I'll update this review if the speed changes.

Pricing: The Token Plan Deal

I subscribed to the Token Plan Standard tier at USD $18/month for this review. All four tests combined used 6% of my weekly credit quota. Six percent. For a model this capable, that's a genuinely good deal, and the plan also gives you access to other models plus image generation.

The catch is the preview discount doing heavy lifting: Qwen3.8-Max-Preview currently runs at 1/10th of standard credit rates, dropping to 1/50th overnight (22:00 to 08:00 UTC+8, so basically midnight to 10am here in Sydney). When the preview pricing ends, redo your maths. And remember Alibaba's own docs say the preview endpoint may change or be replaced. Test on it, don't build production on it.

Verdict: Who Should Use Qwen3.8-Max?

Qwen3.8-Max is very good and very slow. If speed doesn't matter for your task, it belongs in the top tier right now, alongside Fable 5, ChatGPT 5.6 Sol, Grok 4.5 and Kimi K3. Honestly, they're all great, and they're all close. The frontier has become crowded, and the real differentiators now are price, speed, availability and vibe.

For me, today, Grok 4.5 stays my daily driver purely because of speed and efficiency. That could change next month. It changes depending on the task, too. And if you're doing something genuinely important, run it through two or three of these models and compare. That's a luxury we simply didn't have a year ago.

Which brings me to Anthropic. In my opinion they're the big loser of this month. After telling everyone Fable is too good and too expensive to include permanently in normal subscriptions, they've watched competitors ship models that are (probably) roughly as good, much cheaper, more available, and less restricted. The competition didn't out-build them, it out-shipped them. No wonder people are speculating Opus 5 is coming soon. For a while I was genuinely worried the top models would end up locked behind enterprise contracts. Not anymore, and I'm very happy about that.

One caveat worth knowing before you evaluate Qwen for business use: Simon Willison and others have documented that Qwen models carry baked-in CCP-aligned guardrails on politically sensitive topics. For coding and agent work it's largely irrelevant, but if you're a regulated business doing anything content-adjacent, know what you're deploying. I wrote more about choosing AI tools sensibly in my piece on AI agents for small business.

And the thing I'm actually most excited about? The small models. Qwen is the open-weight family powering half the local agent setups out there, including my own experiments with Hermes agent. If the promised open weights land, and if the 3.8 improvements trickle down to sizes I can run on my Mac, that's the real story. I think local AI is going to be very, very good within a few years, especially for businesses like medical practices and law firms where privacy and legal constraints make cloud AI complicated. When the small Qwen3.8 models drop, I'll be testing them the same week. Watch this space.

So this is the other article. The one that tells you what to set up, in what order, and what it'll cost you (spoiler: about twenty US dollars a month plus an afternoon of your time). I'm assuming you can already open Claude or ChatGPT and type a question into it. I'm not going to explain what a chatbot is. What I am going to explain is how to wire one into your actual email, your actual calendar, and your actual pricing, so it stops being a party trick and starts being useful.

One thing I'm deliberately leaving out: AI answering your phone. It's probably the single biggest win available to a trade business, because a missed call is a lost job and voicemail has been losing you jobs for years. But it's also a genuinely deep topic (number porting, voice quality, transcripts, privacy, what happens when it mishears an address) and cramming it in here would do it a disservice. It's getting its own article. Come back for that one.

Quick note: why Claude, and what if you use something else

Almost everything below is written using Claude, because that's what I use every day and I'd rather show you something I actually run than something I've read about. It's made by Anthropic, it costs about twenty US dollars a month for the paid tier, and it's the one I'd hand to a tradie who wants results this week rather than a hobby.

But this is not a Claude advertisement, and you are not stuck.

ChatGPT (OpenAI) does the same job. So, increasingly, do Gemini (Google) and Grok (xAI). They all now have some version of the same three ingredients: a chat window, a way to connect your email and calendar, and an "agent" mode that can go off and do multi-step work while you get on with your day. The names differ, the menus are in different places, the monthly limits are carved up differently. The shape is the same.

So when I say "switch on Cowork and schedule a task", the ChatGPT equivalent is Work, which lives inside the Codex app. When I say "connect the Gmail connector", every one of these has connectors, they just call them different things and hide them in a different corner of the menu. If you're already paying for one of them, use that one. Switching tools to follow a blog post is a waste of a perfectly good subscription.

What matters far more than which brand you pick is the two things in the next couple of sections: the safety rule, and loading the thing with your actual business information. Get those right and the logo on the app is a detail.

The one rule that makes all of this safe

Before any of the fun stuff, this.

AI drafts. You send.

That's it. That's the rule. Never give an agent the power to send, pay, or delete on a real account without you looking at it first.

Here's why, and it's not hand-wringing. Security researchers call it the lethal trifecta: an agent that can (1) read your private data, (2) is exposed to untrusted content from the outside world, and (3) can send data back out. Your inbox has all three. Prompt injection sits at number one on the OWASP Top 10 for LLMs, and it is not a solved problem. It's not a bug waiting for a patch.

Concretely: someone emails you. Buried in the footer, in white text, is an instruction. Your helpful agent reads the inbox, reads the instruction, and cheerfully forwards your customer list to a stranger, or pays an invoice that doesn't exist. It thinks it's helping. It is, technically, following instructions.

This is the reason Anthropic's Gmail connector can only create drafts and cannot press send. People complain about that. It's not a limitation, it's the safety rail. I've written before about what happens when you hand an agent too much rope, and about building an approval gate so a human stays in the loop.

Four rules, stick them on the wall:

1. Nothing sends, pays, or deletes without a human clicking the button.
2. Use a dedicated account with the minimum permissions, not your admin login.
3. No number reaches a customer unchecked. Not one.
4. If a tool needs full write access to be useful, that's a reason to be suspicious, not a reason to grant it.

Right. Now the good bit.

Setting up the foundation

Two steps. Neither is hard, and the second one is the one everybody skips.

Connect Gmail, Calendar and Drive

In Claude, go to Settings, then Connectors, and toggle on Gmail, Google Calendar and Google Drive. Sign in with Google, approve the scopes, done. If you're on a Team plan, an Owner has to enable connectors first, and if you run Google Workspace your admin might need to mark Claude as a trusted app under Security, then API controls. Give it fifteen minutes to propagate before you assume it's broken. (I did not give it fifteen minutes. I assumed it was broken.)

Microsoft 365 works the same way if that's your world. Same principles apply throughout.

Three gotchas worth knowing before you start:

• Gmail is draft-only. Claude reads and composes. You send. See above.
• Calendar is full read and write. This one genuinely works end to end, which is why the calendar workflows below are the most satisfying.
• Attachments are metadata only. Claude sees that a PDF exists, not what's in it. If the quote is in the attachment, you'll need to hand it over yourself.

And do this properly: create a dedicated account, something like ai@yourbiz.com.au, and connect that. Not the login that owns everything. It costs you nothing and it means the blast radius of a bad day is small.

Build the project brain

This is the highest-leverage hour you will spend, and almost nobody does it. So let me slow down here, because if you skip this section the rest of the article is a toy.

What a Project actually is. In Claude, a Project is a folder that holds three things: a set of standing instructions, a pile of documents it can read (the "project knowledge"), and all the chats you've had inside it. Every conversation you start inside that Project automatically sees the instructions and the documents. You don't re-explain your business every morning. It's already there.

ChatGPT has the same concept and calls them Projects too. Gemini has Gems. Same idea, different noun.

Why it matters more than the model. A generic chatbot does not know your call-out fee. Ask it to write a quote and it will invent a number that looks plausible, because inventing plausible things is precisely what these things do. It is not lying to you. It genuinely has no idea, and it has been trained to be helpful, so it produces something helpful-shaped. This is the single biggest reason people try AI for a week and conclude it's rubbish. They asked a stranger to quote a job.

Context is the whole ballgame. A mediocre model with your real prices beats a brilliant model guessing, every single time.

Step one: create the Project and write the instructions. The instructions are the standing brief. Not a prompt, more like the induction you'd give a new office junior. Mine for a trades business would read something like:

You work for [Business], a residential electrical contractor on Sydney's Upper North Shore. Everything you write goes to Australian customers, so use Australian spelling and dollars, and always show GST. You draft, you never send. Never invent a price: if a price isn't in the knowledge files, say "PRICE NEEDED" and stop. Quotes always include our standard exclusions and the 12 month warranty wording. Keep the tone plain and friendly, no corporate waffle. If a job is outside our service area, say so instead of quoting it.

Note what that's doing. It sets the geography, the currency, the tone, the hard rule about sending, and (this is the important one) it gives the model a way to fail loudly instead of guessing. "Say PRICE NEEDED and stop" is worth more than any clever phrasing.

Step two: upload the knowledge. Drag the documents in. Realistically:

• Rate card: hourly rates, call-out fee, after-hours and weekend loading
• Materials price list, exported from your supplier or your accounting software
• Standard inclusions and exclusions
• Terms and conditions, warranty wording, payment terms
• Service area, listed as actual suburbs
• Licence numbers and whatever compliance boilerplate you paste into everything
• A description of how you actually run a job, start to finish
• Two or three of your best past quotes, as examples of the format you want

That last one does a surprising amount of work. Showing beats telling. Give it three real quotes and it will match your style far better than any adjective you could have written into the instructions.

Step three: keep it current, and keep it clean. Two rules I'd tattoo on this:

Point at live files, not dead copies. Rather than uploading a pricing PDF that goes stale, keep the price list in a Google Sheet in the Drive folder the Project can read. Copper goes up, you update one cell, and every quote from that moment on is right. Uploaded files are snapshots. Connected files are current.

Less, but better. There's a real temptation to shovel in every document you own on the theory that more context is better. It isn't. A Project stuffed with six versions of the rate card, two of which are from 2023, will confidently use the wrong one. Every document you add is a document the model might believe. Prune it like you'd prune a toolbox: if you wouldn't hand it to a new employee on day one, it doesn't go in.

A word on memory. Claude also has a memory feature that carries details across conversations, and it's genuinely handy for the small stuff (it'll remember you're in Sydney, that you hate exclamation marks). But do not rely on memory for your prices. Memory is a nice-to-have that gets recalled loosely. Project knowledge is a document it reads. For anything where being wrong costs you money, use the document.

If you want to go deeper on writing the instructions themselves, I've got a whole piece on prompt structure.

Do this properly and everything below stops being a demo and starts being a system.

Scheduled tasks, or: getting it to do things while you're up a ladder

Everything so far assumes you're sitting there typing. The step up is telling it to run on its own.

That's what Cowork is for. It isn't a separate app or a separate tab, it's a button right there in the chat box. You flick it on, and the same conversation you were already having gains two abilities: it can work with files on your machine, and it can run on a schedule instead of only when you're sitting there.

Setting one up is genuinely just: turn on Cowork, describe the task in plain English the way you would to a person, check it does the thing correctly when you run it by hand, and then set it to repeat. Daily at 6am. Every Friday at 4pm. Whatever suits.

Two bits of advice that will save you a fortnight of annoyance:

Always run it manually first. Twice. Watch what it does. A scheduled task that quietly does the wrong thing every morning at six is worse than no task at all, because you'll trust it.

Watch your usage. Scheduled tasks and Cowork sessions eat a plan's allowance much faster than chatting does. A task that runs hourly because it felt clever at the time will find your limit for you.

ChatGPT has scheduled tasks too, and if you'd rather run this stuff on your own hardware without a subscription in the loop, that's what tools like n8n and Hermes are for. More on those later.

The four workflows worth building first

1. The morning briefing

This is the one to build first, because it's useful on day one and it can't hurt anybody.

What it does: at 6am it reads overnight email, sorts it, drafts replies to the easy ones, and tells you what actually needs you. By the time you're in the ute with a coffee, you've got a list instead of an inbox.

How to set it up: turn on Cowork in the chat box, make sure the Gmail and Calendar connectors are on, and create a task with a prompt along these lines:

Every weekday at 6am, read all email received since 5pm yesterday.

Sort it into four groups: (1) new job enquiries, (2) customers replying about a quote, (3) suppliers and admin, (4) junk I can ignore.

For every new enquiry, pull out: name, suburb, phone number, what they want done, and any date they mentioned. Flag anything that sounds like an emergency at the top.

Then check my calendar for today and tomorrow and tell me what I've already got on.

Draft replies in Gmail for anything routine (acknowledging an enquiry, confirming an appointment). Leave them in drafts. Do not send anything.

Finish with a short list: "needs Thomas", for anything you couldn't handle or weren't sure about.

Keep the whole thing under a page. I'm reading it on a phone.

Then set it to run weekdays at 6am and go to bed.

Where it breaks: it will get urgency wrong. It will confidently file a burst pipe under "admin" because the customer was polite about it. It's a briefing, not an autopilot. Read it, don't obey it.

2. Enquiry to quote to calendar

This is the one that made me sit up, because the calendar half genuinely works without a human in the middle.

What it does: an enquiry lands. Claude pulls out the name, address and job type, checks your actual calendar for actual free slots, drafts a quote using your actual pricing from the Project knowledge, and puts a tentative hold in the diary. You read it, fix the bit it got wrong, and hit send.

The free-slot check is the part that surprises people. Because Calendar is read and write, "can you do Thursday morning?" gets a real answer instead of a hallucinated one. It looks at the diary. It knows you're already in Hornsby at nine.

How to set it up. Do this one inside your Project (so it has the prices) rather than as a blind scheduled task, at least until you trust it. When an enquiry comes in, you type something as lazy as "handle the Henderson email" and it goes. The standing instruction that makes that work looks like this, and it lives in your Project instructions:

When I ask you to handle an enquiry:

1. Find the email and pull out: name, address, phone, what they want, and any dates or times they mentioned.

2. Check my Google Calendar for the next 10 working days. Find slots that fit the job length, respecting travel: don't offer me a 9am in Berowra if I'm booked in Chatswood at 8.

3. Draft the quote using ONLY the rates and materials in the project files. If something isn't priced there, write PRICE NEEDED in the line item and tell me. Do not estimate.

4. Create a tentative calendar event, title it "TENTATIVE: [name] [suburb]", and put the customer's phone and the job description in the notes.

5. Draft the reply email in Gmail: thank them, summarise the job so they know we understood it, give the quote, offer two of the calendar slots, include our standard exclusions and warranty. Leave it in drafts.

6. Tell me in chat what you did, what you weren't sure about, and anything you had to guess.

Step 6 is the one people leave out and it's the most valuable line in there. You want it confessing.

Where it breaks, and this is not optional reading. The quote sits in drafts. Always. If you want true automatic sending you need an automation layer like n8n underneath, and even then I'd put a gate on it.

Here's why. AI is very good at producing a quote that looks right. The layout is beautiful. The line items are professional. The wording is better than yours. And the measurement is wrong by a factor of two and it has no idea, because being wrong feels exactly the same to it as being right.

The AI takeoff tools that read plans and photos run about 2 to 4 percent variance from a manual takeoff on clean residential drawings, and 8 to 12 percent on messy commercial sets. Which sounds survivable, until you notice individual line items can be wildly out even when the total lands close, because the errors cancel each other out. Great for the average. Terrible for the one job you actually quoted.

Draft, check, send. Every time. If you only take one thing from this article, take that.

3. Invoices and the boring money stuff

Start with receipt capture, because it's the least glamorous and the most reliable. Dext, Hubdoc, or MYOB's built-in capture, whichever your accountant already uses. Photograph the receipt in the merchant car park, it extracts the supplier, the ABN, the GST and the total, and files it against the right account. This is the most boringly dependable AI in the entire stack. It also doubles as your compliance fix, because the ATO wants your records kept for five years and a shoebox of faded thermal paper is not a system, it's a hope.

Then use the AI that's already inside your accounting software. Xero's Just Ask Xero (JAX) will draft invoices, chase payments, and predict when a given customer is actually likely to pay you (which is a different question from when they said they would). QuickBooks has a similar squad of agents, including one that pre-checks your GST against your P&L before you lodge. You're already paying for these. Turn them on.

Then wire your books into Claude. Xero ships an official MCP server, which in plain English means Claude can query your accounting data directly. Once that's connected, a Friday afternoon scheduled task earns its keep:

Every Friday at 4pm, check Xero for:

1. Invoices more than 14 days overdue. For each one, tell me the customer, the amount, how many days late, and whether I've already chased them.

2. Any completed jobs from this week that I haven't invoiced yet. Cross-check against my calendar.

3. Draft a polite chase-up email for each overdue invoice. Firm but not rude, we want to keep these people. Leave them in drafts.

Give me the total outstanding at the top so I know how bad it is before I read the detail.

Point 2 is the sleeper. Most small operators aren't losing money to bad debts, they're losing it to jobs they simply forgot to invoice. Having something cross-reference "went there" against "billed for it" once a week is a genuinely profitable ten minutes.

The limit. Treat a BAS as a draft, never a lodgement. These tools are good at flagging discrepancies and terrible at understanding your specific situation. GST-free items, capital purchases, private use apportionment: a general-purpose model will be fluent and confident and wrong. Your bookkeeper's name goes on the form, not Claude's.

4. Site notes and photos, without typing anything

You asked for this one and it's my favourite, because it's the gap nobody fills.

The simple version: the Claude mobile app takes voice input. Sit in the ute after a job and ramble at it. "Right, Henderson job, Wahroonga, the switchboard is older than I am, needs a full replacement, I've quoted around fourteen hundred, they want it done before the twentieth, also the bloke has a dog that hates me." Ask for a structured job note. Get one back. Done before you've reversed out of the driveway.

The better version: a phone shortcut (iOS Shortcuts, or Tasker on Android) that dumps voice memos and site photos into a dated Google Drive folder. One tap. Then at the end of the day, Claude reads the folder via the Drive connector and turns the whole mess into job records, a materials list for the merchant run, and tomorrow's follow-ups.

One warning here, because I don't want you building this and getting cross with me. Claude's Drive connector reads Docs and Sheets well. Audio files, it does not. Do not drop an .m4a into Drive and expect Claude to listen to it. Transcribe first: use your phone's built-in dictation, or run Whisper locally (my Mac Mini M1 does this happily and nothing leaves the house, which is a nice bonus if the recording mentions a client's name). Text into Drive, then Claude.

And if you're already on ServiceM8, check what you've got before you build anything. Its dictation writes straight to the job card, and you're paying for it.

Claude Code is not just for code (and that's the interesting part)

The name is doing this tool a disservice.

Claude Code is a terminal application. You point it at a folder on your computer and it can read, write, move, rename and reorganise everything in there, run commands, and work through a long list of tasks without you babysitting each step. That is obviously useful for code. It is also, and almost nobody talks about this, extremely useful for files. Which is what a small business actually drowns in.

Things I've watched it do that have nothing to do with programming:

• Take a folder of 400 job photos with names like IMG_4471.jpeg and rename and sort them into folders by date and job number.
• Read a directory of eighty PDF invoices from suppliers, pull out the date, supplier, total and GST from each, and produce a single spreadsheet.
• Take a decade of scattered documents and reorganise them into a filing structure that a human could actually navigate.
• Turn a folder of messy site notes into a properly formatted report, with headings, in Word.
• Compare this year's price list to last year's and tell me exactly what moved and by how much.

That's the pitch: it's a very fast, very literal assistant that can touch every file on your machine. If your admin problem is fundamentally "there are too many files and they're a mess", this is the tool, and it's the one that'll surprise you.

Now the honest part, because there's a catch and it's a real one.

This is meaningfully harder than everything else in this article. It lives in a terminal, which means a black window and typed commands. There's no undo button. When it renames four hundred files it has renamed four hundred files, and if it misunderstood you, it misunderstood you four hundred times, quickly and efficiently.

So if you go here:

• Work on a copy. Always. Copy the folder, let it loose on the copy, check the result, then swap. This is not paranoia, it's just what you do.
• Use version control or at least a backup before any bulk operation. Time Machine counts.
• Do not turn off the permission prompts. There's a flag that skips all the "can I do this?" confirmations, and it is enormously tempting, and I've written about why that's a bad idea at some length. Short version: the confirmations are the only thing standing between an ambiguous instruction and a very bad afternoon.
• Watch the meter. Agents can get stuck in fix-loops, cheerfully burning tokens trying the same broken thing in slightly different ways. Type /cost occasionally. Ask me how I know.

And on the actual coding side: yes, you can now build small internal tools that used to require hiring someone like me. A pricing calculator. A script that reformats quotes onto your letterhead. A little dashboard of unpaid jobs. Genuinely, this works.

Good candidates: internal, not sensitive, low stakes, and honestly you'd have bodged it in a spreadsheet anyway.

Bad candidates: anything touching customer personal data, anything touching payments, anything a customer sees. Not because the AI can't write it. It can, and it'll work fine on Tuesday. The problem arrives six months later when a dependency updates and it silently breaks, and fixing it requires precisely the skill you were avoiding by using AI in the first place. Building is the easy half. Owning it forever is the other half, and nobody mentions that in the demo video.

The alternatives, honestly

I've written this around Claude because it's what I use daily and know best. But the shape of it ports.

ChatGPT's version is called Work, and it now lives inside the Codex app, which until recently was just OpenAI's coding agent. That's a slightly odd home for it if you're not a developer, but it's the same idea: connectors, file access, multi-step tasks that run without you watching. Every workflow above translates roughly one to one. If you're already living in OpenAI, stay there and don't let me talk you into a second subscription.

n8n, Make, Zapier are the layer you add when you want something to run without you sitting in a chat window. Self-hosted n8n on a spare Mac Mini or a five dollar VPS is genuinely cheap. It's also only free if someone owns it, and "someone" is you, on a Friday night, when an update breaks a workflow.

Hermes Agent from Nous Research is the always-on, self-hosted, talks-to-you-on-Telegram end of the spectrum. It's real, it's open source, it runs for pocket money, and I've written about it in detail. It's also a developer's toy rather than a business product. No audit logging, no support, you own uptime and security. Wonderful if you enjoy running a Linux box. If you don't know whether you enjoy running a Linux box, you don't.

Rough order of power, and of how much of your weekend it eats: Chat, then Cowork, then n8n, then self-hosted. Stop as soon as it's working. There's more on the broader agent landscape here.

What's a waste of your money

• Unreviewed AI quotes. Covered above. The maths is seductive and the line items lie.
• "AI local SEO" subscriptions at $99 a month. Answer your phone first. Come back to this when you've run out of better problems, which will be never.
• Thin AI wrappers. Before you subscribe to any narrow "AI something" tool, ask what it does that your twenty dollar chatbot and the software you already pay for don't. Usually the answer is "a nicer logo".
• Autonomous scheduling optimisation. Suggestions across a couple of vans, fine. Set-and-forget across a fleet with real constraints, no. Not yet.
• Anything Australian and regulatory. A general chatbot will be confidently, fluently wrong about GST treatment, Fair Work award rates, and state licensing. It will not hedge. It will not blink. Your apprentice's pay rate is not a vibe.

Watch the meters too. Per seat, per minute, per token. Cowork tasks chew through a plan allowance far faster than chat does, and the first big bill is always a surprise.

Start here this week

Three things. In this order.

1. Turn on the AI you're already paying for. ServiceM8, Xero, MYOB, QuickBooks, all of them shipped real features in the last year. Zero new vendors, zero new risk.
2. Twenty dollars for Claude or ChatGPT, connectors wired to a dedicated account, not your admin login.
3. Build the project brain. Rates, prices, terms, service area. Thirty minutes. This is the step that separates a useful assistant from an expensive random number generator.

Then leave it alone for a fortnight and see what you actually reach for.

Next up on the blog: the phone. Which is where the real money is hiding, and where the real landmines are too.

I build this stuff for a living, out of Sydney, mostly automation for the admin work that eats the evening. If you get stuck wiring it up, you know where to find me.

What Grok 4.5 actually is (and why the price is the story)

Grok 4.5 launched July 8, 2026 for developers, with the wider public rollout a day later, from SpaceXAI. If that name makes you do a double take, fair enough: SpaceX absorbed xAI back in February, so the company shipping Grok now is the merged entity. It's pitched squarely at coding, agentic tasks, and knowledge work. This is not a consumer-chatbot play, which is a nice change of pace.

Here's a clean definition if you just want the one-liner: Grok 4.5 is a mixture-of-experts coding and agentic model from SpaceXAI, jointly trained with Cursor, that lands near the top of the independent leaderboards while costing a fraction of its rivals per task. The headline isn't the leaderboard, though. It's the economics.

The API price is $2 per million input tokens and $6 per million output, which Artificial Analysis clocks at more than 60% below Claude Opus 4.8 and GPT-5.5. On top of that, it's stingy with tokens: roughly 4.2 times fewer output tokens than Opus 4.8 to resolve the same task. Cheap price multiplied by fewer tokens is where the real gap opens up.

One thing to get straight before anyone runs away with a "Grok beats Opus" headline. Musk's own framing, in the pre-launch chatter, put it at roughly Opus 4.7, not 4.8. So "Opus-class tier" is fair. "Beats Opus" is not what the numbers say. Cursor also disclosed that an earlier snapshot of its own codebase accidentally ended up in training, which gave Grok an unfair edge on one internal benchmark. Credit for owning it publicly. It's also worth naming, because it does raise a small flag about training-data contamination.

The Cursor connection: why this model can code

This is the part that actually explains the coding ability, so stick with me.

On June 16, 2026, SpaceX exercised an option to buy Anysphere, the maker of Cursor, for $60 billion in an all-stock deal. That's the largest acquisition of a venture-backed startup ever recorded, nearly double Google's Wiz purchase. TechCrunch, Reuters, CNBC, and everyone else confirmed it, so this isn't a rumour I'm passing along.

But the deal itself isn't the interesting part for us. The training is. Cursor's own launch post says Grok 4.5 was trained on trillions of tokens of Cursor data: real developer-agent interactions, not just scraped public repos. That's the difference between learning what code looks like and learning how developers actually work through a problem. Identify the bug, run the tool, read the error, fix it, verify. That feedback loop is a genuinely plausible source of the token-efficiency edge, not marketing fluff.

Here's the question I actually wanted answered. Cursor's own Composer 2.5 was already a strong, cheap coding specialist. So what did the joint training add on top of a model that was already good? The honest answer from the outside is: breadth. Cursor deliberately widened the data mix beyond pure software engineering into STEM, research, and general knowledge work, so Grok 4.5 generalises where Composer stayed narrow. Whether that trade lands for your specific workflow is exactly the kind of thing you'd want to test yourself.

Speculating a little, because it's fun: the more interesting thing over the next year probably isn't the model at all. It's what SpaceX and Cursor ship around it, the agent and tooling layer, now that the same company owns both the IDE and the weights. That's a tighter loop than any competitor has right now. We'll see if they use it.

I ran my usual tests

If you've read my other model reviews, you know the drill. I keep the prompts fixed across every model so the comparison actually means something. Same three tests, every time:

• Two single-file HTML website builds: a Sydney coffee roaster and a pop-culture clothing store.
• A Go Texas Hold'em terminal simulation: six personality-driven bots, 1000 hands, full statistics at the end.
• An audit of thomas-wiegold.com, my own blog.

None of this is a scientific benchmark. It's a working developer poking at a tool the way he'd actually use it. I'll be explicit about that, because the whole point is that it's hands-on rather than a leaderboard screenshot.

The website builds

These were among the best build results I've gotten out of any model. Layout was clean, the animations were tasteful rather than the usual "look at all the CSS I know" mess, the fonts were sensible, and it was fully responsive on mobile without me asking twice. It also generated fast.

I'll hedge honestly, though. I still slightly prefer the builds I got from Gemini 3.5 Flash in my Antigravity review. Not by a lot, and reasonable people could flip the ranking, but I'm not going to oversell Grok here just because the price is good.

The Go poker simulation

This might be my best-ever result on this test, full stop.

Before Grok 4.5, exactly one model had one-shotted the poker sim cleanly: Claude Fable. Grok did it too, first try, and its statistics output at the end was more detailed than what I'd seen before. Six bots with distinct personalities, a thousand hands, and a stats breakdown I didn't have to fix. Under ten minutes, start to finish. For a test that's tripped up plenty of frontier models, that's a genuine result.

The site audit

This is the one. This is the reason I'm bothering to write the review at all.

Grok 4.5 caught a real problem on my site that no other model has ever flagged, and I've run this audit a lot. Not even Fable found it. My Markdown to HTML article conversion was injecting hidden, non-rendering data into the output, quietly inflating the payload of every single article. Invisible to a reader, invisible to me, and adding weight to every page load. Concrete, fixable, and exactly the kind of finding that justifies running the test in the first place. I fixed it today.

So where does that leave the section verdict? The benchmarks put Grok 4.5 a notch below Fable, Opus, and GPT-5.5. In my hands, across these three tasks, it felt level with them or close enough that I couldn't reliably tell them apart. Gut feel, not science. But that honesty is exactly why I trust my own tests more than a vendor chart, and hopefully why you do too.

Pricing and SuperGrok: where the value actually lands

I'll say it plainly, because it's the number that matters. On Artificial Analysis's Coding Agent Index, a task in Grok Build runs about $2.49. The same class of task costs roughly $5.07 for GPT-5.5 in Codex and about $11.80 for Fable 5 in Claude Code. That's not a rounding difference. Multiply it across a month of agent runs and the arithmetic bends hard in Grok's favour.

The API breakdown, if you want it: $2 input, $6 output, $0.50 cached (a 75% discount), and prices double once your prompt crosses 200k tokens. Cached-heavy agentic loops are where this gets silly cheap.

My own experience backs the number up. I'm on the $30-a-month SuperGrok plan, and my full review test suite, all three tests, barely dented my quota. Somewhere around 1% of my weekly usage. I had to double-check that, because it felt too low. It wasn't.

This tracks with the thesis I've been banging on about since my MAI-Code-1-Flash review: I back cheap-and-fast models for the work most of us do most of the time. Fable is a better model. I'm not disputing that. But I won't pay Fable prices for routine work, and neither will most teams once they see the invoice. Grok 4.5 is the one I'll actually reach for day to day.

The caveats worth knowing

Nothing here is a dealbreaker on its own, but you should know all of it before committing.

The context window is 500k tokens. That's a regression from Grok 4.3's 1M, and it sits behind the 1M windows on Claude and Gemini. Now, I'd argue those headline numbers are mostly marketing anyway, since models degrade and start dropping detail long before the window is actually full, so I wouldn't lose sleep over the on-paper gap. Still, less usable room is less usable room, and if you're doing long-document work or feeding in a large repo, it's worth knowing you've got a smaller runway to play with than the previous version gave you.

Reliability has a real wrinkle. On the AA-Omniscience benchmark, accuracy improved but the hallucination rate rose to around 54%, up from 25%. In plain terms: it knows more, and it's also more confident when it's wrong. For knowledge work, keep a hand on the wheel.

It's not available in the EU at launch. Mid-July is the expected date, so European devs should confirm access before building anything on it.

On trust: the loudest Hacker News thread on launch day wasn't about capability at all, it was about alleged Musk-linked steering of political outputs. I'll note it and leave it there. Opinions in that thread were genuinely split, and it's not my job to settle it for you.

And the "1.5 trillion parameters on a V9 foundation" figure that's floating around? That's attributed to Musk and reporting, not officially disclosed by SpaceXAI. Treat it as widely-repeated-but-unconfirmed.

Grok 4.5 review verdict: a game-changer for Grok

Straight call: Grok 4.5 is a genuinely good coding model now, and at $30 a month on SuperGrok it's an absurd amount of value. I'll keep using it. Even though Fable is the better model, I'm not paying Fable prices for the routine stuff, and this closes enough of the gap that I don't feel like I'm compromising.

The way to think about it isn't "replace Claude." It's routing. Send the high-volume, repetitive, agentic work to Grok 4.5, and keep Opus 4.8 or Fable 5 for the hardest, highest-risk tasks where a mistake is expensive. Match the model to the job. That's the whole game in mid-2026.

And the competition is only heating up, which is great for us. GPT-5.6 is landing within days and Grok 5 is reportedly already in training. More pressure on all of them means better tools and lower prices for the rest of us. Mostly, I'm curious where Cursor goes now that it's part of a rocket company. That's a sentence I did not expect to write this year.

Short version: it's fast, it's cheap, and I won't be using it. But the details are more interesting than the verdict, so let's go through it.

What Is MAI-Code-1-Flash?

MAI-Code-1-Flash is Microsoft's first coding model trained entirely in-house, announced on June 2, 2026 at Build as one of seven MAI models from Mustafa Suleyman's AI Superintelligence Team. It's a sparse Mixture-of-Experts model with 137B total parameters, only 5B active per token, and a 256K context window. It was trained directly inside the GitHub Copilot production harness, which is genuinely a different approach: instead of optimising for benchmarks and hoping it transfers, Microsoft trained it against the exact tools and workflows Copilot uses in VS Code.

The positioning is refreshingly honest. This is not a frontier model and Microsoft doesn't pretend it is. It's meant to be the cheap, fast, everyday workhorse. Completions, small refactors, boilerplate, repository Q&A. I actually like this strategy. Not every task needs a trillion-parameter model burning through your credits, and a vendor saying "this is our budget option" out loud is rarer than it should be.

The question is whether it's good enough at the everyday stuff to earn that slot. That's what I tested.

Why I Wanted to Try It

Two reasons. First, plain curiosity. Microsoft building its own models after the OpenAI contract renegotiation is one of the more interesting strategic shifts this year. Suleyman has been open about the ambition (and about the current gap, telling The Verge that only three labs matter right now and Microsoft isn't one of them). A first release from a team with that much money behind it is worth a look regardless of the benchmarks.

Second, I genuinely like the fast-and-affordable philosophy. My daily driver setup already routes easy tasks to cheaper models. If Microsoft delivered a great budget model, I'd use it. I have zero brand loyalty here.

I tested it through my existing GitHub Copilot subscription in VS Code, using the same four prompts I throw at every model I review. Same conditions, same expectations.

My Test Setup

The Four Standard Tests

Every model I review gets the identical suite:

1. A website for a fictional Sydney coffee roaster. Tests design taste, layout, typography.
2. A pop culture clothing store. Tests the same skills with a louder brief.
3. A poker simulation in Go. Tests actual logic: game rules, statistics, state management.
4. An audit of my own site, thomas-wiegold.com. Tests analysis and judgement on a real codebase.

Same prompts every time. It's not a scientific benchmark, but it's apples-to-apples across every model I've written about, and after enough reviews you develop a feel for where each model sits.

Hands-On Results

Website Builds: Rounded Corners Everywhere

The coffee roaster site was, and I'm choosing my words carefully here, incredibly ugly. It worked. Nothing was broken. But it was the worst-looking result I've had since I started running this test. Boring system-adjacent fonts, the most generic brown-and-cream coffee palette imaginable, a flat layout with no hierarchy, and rounded corners on absolutely everything. Buttons, cards, images, sections. If it had a corner, it got rounded.

The clothing store went slightly better. The animations were actually a bright spot, smooth and reasonably tasteful. But the same problems showed up: boring fonts, and again the rounded-corner obsession. It's such a specific quirk that I have to assume it's baked into the training data or the RL reward somewhere. Some evaluator at Microsoft apparently really likes border-radius.

To be fair, I retested both builds with my custom web-design skill that I developed based on Claude's frontend-design plugin, which usually rescues weaker models. Fonts and layout improved noticeably. The rounded corners survived. At this point I consider them a personality trait.

Go Poker Simulation: Fast but Broken

This one was an outright fail, and the numbers make it easy to show why.

Generation speed was genuinely impressive. The code appeared quickly, compiled, and ran. Then I looked at the output: after 1,000 simulated hands, individual players were showing 800 to 900+ wins each. In a game where most of the time one player wins per hand, that's not a rounding error, that's a simulation that doesn't understand what it's simulating. The betting logic was similarly broken. Players were behaving in ways that made no sense, and the game state was often inconsistent. The model produced code that looked superficially correct but failed to implement the rules of poker in any meaningful way.

It is not an easy task. Claude Fable was the only model to get it right on the first try, and created a perfect simulation. All other models I tested needed multiple iterations, and some never got there. But MAI-Code-1-Flash didn't even get close. It was fast, yes, but speed without correctness is useless.

Website Audit: Passable, Not Impressive

The audit of my own site was the most respectable result. Fast turnaround, and it found a handful of real issues. It also flagged several non-issues with full confidence and missed things that stronger models catch reliably. If I'd never run this test with a better model, I might have been satisfied. I have, so I wasn't.

Passable. That's the word. Not a disaster, not a reason to pick it.

How MAI-Code-1-Flash Compares to the Benchmarks

Here's where my results and Microsoft's marketing part ways.

Microsoft's launch numbers show MAI-Code-1-Flash beating Claude Haiku 4.5 across every coding benchmark it published, including a 16-point lead on SWE-Bench Pro (51.2% vs 35.2%), while using up to 60% fewer tokens. Sounds great. Three caveats.

First, every comparison is vendor-run, in Microsoft's own VS Code harness, against exactly one competitor, and Haiku is the smallest model in Anthropic's lineup. That's not a benchmark table, that's a controlled demolition.

Second, on independent leaderboards, ~51% SWE-Bench Pro sits well below the frontier. Opus 4.8, GLM-5.2, Qwen3.7 Max, MiniMax M3, GPT-5.5 and Kimi K2.6 all score meaningfully higher. To be fair to Microsoft, 51% from a 5B-active model is strong for its size (Opus 4.6 scored around 52% on the same benchmark not that long ago). The engineering is real. The competitive position isn't.

Third, and this is the part benchmarks never capture: my hands-on results don't match the story. A model that scores 51% on SWE-Bench Pro should not produce a poker simulation where everyone wins. Benchmarks and real-world usability diverge, and this release is a textbook example.

One more thing worth knowing before you buy the "clean, licensed data" pitch: Simon Willison dug into the underlying technical paper and found the same public-web-crawl licensing situation as every other major LLM, roughly 1.2 trillion crawled pages plus a Common Crawl portion. The Data Card also discloses that OpenAI models were used for data preparation. Technically compatible with "no distillation," but the marketing is doing some heavy lifting.

Is MAI-Code-1-Flash Worth Using?

For me, no. And I've tried to find a reason.

The pricing is real: $0.75 per million input tokens, $0.075 cached, $4.50 output under Copilot's AI Credits system. That undercuts Haiku 4.5 on input, and the token efficiency claim means effective cost per task could be lower still. If your entire use case is autocomplete and boilerplate at maximum volume and minimum cost, it might squeeze into your setup on price alone.

But at similar price points you can get Kimi K2.7 Code, Qwen 3.7 Plus, MiniMax M3, or DeepSeek V4, and every one of them beat MAI-Code-1-Flash in my testing. There's no dimension where it wins except being made by Microsoft. If you're in Copilot and want a cheap model, my honest recommendation is Kimi K2.7 Code. It costs about the same and doesn't lose track of who won the poker hand.

One practical warning if you do try it: multiple users have reported a plan-mode looping bug that burns credits at speed while producing nothing. Under usage-based billing, set your budget caps before experimenting, not after.

The genuinely interesting open question is MAI-Thinking-1, the flagship reasoning model still in private preview. It's the base MAI-Code-1-Flash was trained from, it's a much bigger model, and Microsoft's claims for it are more ambitious. If the small model is this rough and the big one is good, the strategy still works.

Final Thoughts

Fast and cheap: confirmed. Both claims check out completely, and I want to give Microsoft credit for shipping exactly the performance profile it advertised.

Quality: not confirmed. The design output was the worst I've tested, the logic task failed outright, and the audit was merely okay. For a model whose entire pitch is "good enough for everyday work," it wasn't good enough for my everyday work.

I remain curious about where Microsoft goes from here. Training inside the production harness is a smart idea, the token efficiency is real engineering, and MAI-Thinking-1 might change the picture entirely. I'll test it the moment I can get access, same four tasks as always.

Until then: skip this one. Your credits deserve better.

Previously in this series: my reviews of Claude Fable 5, MiniMax M3, and DeepSeek V4. And if you enjoy watching hype meet reality, there's also the DeepSeek V5 rumour that wasn't.

It's a great story. It's also wrong.

The actual research (which is not from Stanford, and which I'd encourage you to read before quoting) found close to the opposite. The best agent it tested held its assigned goal almost perfectly for more than 100,000 tokens. The scary number was a measure of how robust the thing was, not where it fell off a cliff. Somewhere between the paper and my inbox, a finding got inverted and a database got dramatically murdered for effect.

So no, there's no tidy 20-minute timer. But agents do degrade over long contexts, and the way they degrade is genuinely worth understanding. The real story is less cinematic and a lot more useful: the leverage in agentic AI has quietly moved from writing clever prompts to managing what's in the context window. This is a field guide to doing that. Fair warning, it's notes on an open problem, not a victory lap.

Prompting Isn't Dead, It Just Moved Into the Loop

You've heard the line by now. "Prompting is dead, build loops." Stop hand-holding the model, wrap it in an agentic loop, let it rip.

There's a real half-truth in there. We genuinely are writing fewer babysitting instructions than we did two years ago, because the models read intent better and because over-steering them often makes things worse, not better. I've written before about why most of the old prompt-engineering tricks have aged badly, and a lot of that holds. The era of stuffing your prompt with "you are a world-class expert who thinks step by step" is over.

But "prompting is dead" is the wrong conclusion. Here's the thing nobody mentions when they're selling you on loops: a one-shot prompt's weakness costs you exactly once. In a loop, that same weakness fires on every single iteration. And worse, the errors don't just repeat, they accumulate inside the context. Each pass leaves residue. By turn forty, the agent isn't reasoning about your task anymore, it's reasoning about a context window full of its own earlier confusion.

I went deep on this when I wrote up how the Ralph loop and recursive agents actually behave, and it changed how I think about the whole pattern. The loop doesn't retire the prompt. It multiplies the consequences of a bad one, and it adds a brand new failure mode (drift) that one-shot prompting never had to worry about.

So the skill didn't disappear. It moved. It used to be about wording the request. Now it's about curating what's in the window. We gave that second thing a name, context engineering, and it turns out to be most of the job.

Your Context Window Is Lying to You

Let me save you some pain. The big number on the model's spec sheet is marketing.

Effective vs advertised context

Every frontier model now advertises an enormous context window. A million tokens, sometimes more. The implication is that you can pour your entire codebase in and the model will hold all of it in mind at once. It will not.

The benchmarks are pretty blunt about this. NVIDIA's RULER work, which has become the standard sanity check, suggests models reliably use only somewhere around 50 to 70 percent of their advertised window for real multi-step work. Chroma's "context rot" study tested a whole pile of frontier models and found that every single one degraded as the input grew. Not at the limit. Well before it. Even on tasks that should have been trivial.

And here's the part that matters for agents specifically: multi-fact, multi-hop retrieval falls apart much faster than simple single-fact lookup. Finding one needle in a haystack is easy. Connecting four needles scattered through that haystack, which is what your agent does on every non-trivial step, is where the wheels come off.

So the spec-sheet number is a capacity claim, not a quality claim. Treat it accordingly.

Why a bigger window won't save you

The intuitive fix is "just use a model with a bigger window." It doesn't work, for two compounding reasons.

The first is context rot: the signal gets blurrier the more you load, full stop. The second is sneakier and it's called proactive interference. Stale information sitting in your context actively corrupts the model's ability to retrieve the fresh stuff. The unsettling finding from recent work is that this persists regardless of how big the window is, and it resists prompt-engineering fixes. You can't just politely ask the model to ignore the old data.

This is where I reach for an analogy, and I'll upgrade the usual one. People say the context window is like RAM, limited working memory you have to manage. True, but incomplete. It's RAM where the contents get blurrier the more you cram in, and where the old entries actively scramble your reads of the new ones. So the job was never just fitting things into memory. The job is eviction and forgetting. (Standard disclaimer: this is a useful model, not a literal description of the architecture. Please don't email me about attention heads.)

One more thing that breaks the "bigger window" instinct. In long autonomous runs, agents derail even when the window never fills up. There's a now-famous benchmark where an agent running a simulated vending-machine business spiraled into a meltdown and started trying to email the FBI about imaginary crimes. The failure had nothing to do with running out of space. It was a coherence failure. More room wouldn't have helped. The agent didn't need a bigger desk, it needed to stop losing the plot.

8 Rules for Engineering Agent Context

Right, the practical part. This is the stuff I actually do, roughly in order of how much grief each one has saved me. Most of it lines up with Anthropic's own writeup on context engineering, with a layer of my own scar tissue on top.

1. Treat context as a budget, not a backpack. Your instinct is to throw everything in just in case. Resist it. Aim for the smallest set of high-signal tokens that gets the job done. A rule of thumb I've seen practitioners settle on is to keep your working context well under the advertised window, often in the ballpark of a quarter to a third of it. That's a starting heuristic, not a law of physics, but it's a much better default than "fill it up."

2. Put your persistent rules in a file, not the chat. Anything the agent must always know belongs in a project file like CLAUDE.md or AGENTS.md at the repo root, not buried in conversation history that's going to get compacted away. This is now a genuine cross-tool standard. That said, these files are not free and they're not magic, and I've got opinions on when CLAUDE.md is helpful versus when it's just expensive noise that are worth a read before you write a 500-line one.

3. Retrieve just-in-time, don't front-load. Instead of dumping the whole codebase into context, hand the agent lightweight references (file paths, queries, links) and let it pull what it needs with tools like grep and glob. This mirrors how you actually work. You don't memorize the repo, you search it. Your agent should too.

4. Compact deliberately. When you're near the limit, summarize and continue rather than just truncating. The safest, lowest-risk trim is clearing out stale tool results, because once a tool call is buried deep in history, the agent rarely needs the raw output again. One caution from experience: over-aggressive compaction will quietly drop something whose importance only becomes obvious three steps later. Compaction is lossy. Tune it on real traces, not vibes.

5. Give the agent a scratchpad. A simple NOTES.md or a maintained to-do list that lives outside the context window is shockingly effective. The agent writes progress and decisions to it, pulls them back when needed, and suddenly it can track a long task without holding the entire history in working memory. Cheap, simple, works.

6. Re-ground the goal on a cadence. Periodically re-inject the actual objective into the context. It sounds almost too dumb to bother with, but re-stating the goal measurably reduces drift and costs you next to nothing. If your agent is on a long run, remind it what it's doing. Same as you'd do with a junior who's been heads-down for three hours.

7. Decompose with sub-agents, but only for parallel work. Splitting a task across multiple agents gives each one a small, clean context, which is great. The catch is it costs dramatically more tokens (think on the order of fifteen times more in some setups) and it introduces a fresh failure mode where the sub-agents collectively wander off the original problem. Sub-agents shine for breadth-first, independent subtasks. For tightly-coupled work like most coding, where everything depends on everything else, they're often a bad fit. Don't reach for a swarm when one focused agent would do.

8. Gate the irreversible stuff. This is the real answer to the "agent deleted the database" horror story. Put a human approval step in front of destructive actions: database writes and deletes, deploys, anything involving money. Add checkpointing so a derailed run resumes from a known-good state instead of starting over. I'll be honest, the temptation to just let it run unsupervised is strong, and I've written about what actually happens when you skip permissions and let Claude Code off the leash. Short version: gate the things you can't undo. Your future self will thank you.

Does the Model Matter? Yes, But Not How You Think

It does matter, just not in the way the leaderboard-watchers assume.

The newer reasoning models resist goal drift noticeably better than last year's batch. That's real progress. But "better" is not "immune," and under sustained pressure or a long pile-up of accumulated context, every model I've worked with will eventually wobble. There's no model you can buy your way to fire-and-forget reliability with. Not yet.

Two findings from recent work are worth tattooing somewhere visible. First, drift can be inherited. If one agent picks up a context that an earlier, weaker agent already nudged off course, the strong model tends to continue the drift rather than correct it. So watch your handoffs. Second, coding agents drift more when an instruction conflicts with something the model was strongly trained to value, like security or privacy. Tell an agent to do something that rubs against its training, and the constraint gets slippery under pressure.

The practical takeaway for picking a model: ignore the headline context number. Choose based on the actual shape of your workload (how much multi-hop retrieval, how long the horizon, how much tool use) and then, this is the important bit, measure your own degradation curve. The spec sheet won't tell you where your agent starts getting dumb. Only your own evals will. This is doubly true if you're putting agents to work in a small business, where a confidently-wrong agent costs you real money and not just a benchmark point.

The Frontier: Agents That Manage Their Own Memory

Here's where I get to be a bit speculative, because this is the part I find genuinely exciting.

Everything above is us managing the agent's context by hand. The obvious next move is to make that someone else's job. Specifically, another agent's. Picture a second process running in the background between tasks, pruning the memory, reorganizing it, rewriting the messy bits into something cleaner. An agent that "dreams" during its downtime and wakes up with a tidier head.

I want to be clear that this isn't me inventing something in a coffee shop. It's an active research frontier with a real name. The Letta folks (out of the MemGPT lineage) call it sleep-time compute: agents using idle time to turn raw context into learned context they can reuse later. There's a small wave of 2026 preprints borrowing directly from neuroscience, with names like SCM and SleepGate, that model consolidation and forgetting as explicit phases, the way actual sleep does. And the early shipped versions are already here. ChatGPT and Claude Code both do background memory consolidation between sessions now, distilling your preferences and project notes without being asked. (If you see a specific product name attached to this, check it against the vendor's own docs before you repeat it, because the branding in blog posts runs ahead of the official announcements.)

The honest trade-off, because there's always one: an unsupervised process that rewrites memory can also hallucinate structure. Feed it a noisy context and it'll confidently write down patterns that were never there, then treat that fiction as fact next session. Dreaming curates, but it can also quietly corrupt. We do not have this solved.

Which is sort of the whole point. We're all at the very beginning of figuring this out, together, in public. I'm tinkering with some of this myself. I don't have anything worth showing yet, just a growing pile of experiments and half-formed opinions, and I'd rather tell you that plainly than pretend I've cracked it.

Putting It Together

If you want the whole thing as a decision ladder, here it is.

Default to tightly-scoped workflows over open-ended autonomy. When you do go agentic, engineer the context first (rules one through six, the budgeting and re-grounding and just-in-time stuff). Reach for sub-agents only when the work genuinely splits into independent parts. And no matter what, gate the actions you can't take back.

A few caveats so you don't quote me too confidently. This field moves monthly, and a lot of the research I've leaned on here is preprints and previews, not settled science. Some of it comes from vendors who have a horse in the race, though the independent benchmarks broadly back up the direction. And there is no universal token cliff. Degradation is gradual, it's model-specific, and it's task-specific. Anyone selling you a magic number is selling you something.

But the core lesson is solid, and it's not going anywhere. In agentic AI, the win doesn't go to whoever has the biggest context window or the cleverest prompt. It goes to whoever manages context the best. Prompting moved into the loop, and the loop runs on whatever you put in front of it.

So put good things in front of it. That's the job now.

This is most of what I actually do when I build agents for businesses. The prompt is rarely the problem, and the context almost always is.

Last updated: June 2026. This space changes fast, so if you're reading this much later, assume some of the specifics have moved.

I run DeepSeek in production every day, so this question matters to me beyond idle curiosity. It's the model quietly doing most of the work behind my agent setup, and if a successor is coming I'd like to know whether to plan around it or not. So I went and checked the primary sources instead of trusting the rumour mill. Let me walk you through what's real, what's noise, and what I'd actually watch.

Spoiler: the most repeated "evidence" for an imminent DeepSeek V5 is a misreading of a routine deprecation notice. We'll get to that.

The short answer: no date, and that's okay

DeepSeek has not announced V5. Not a name, not a window, not a teaser. The only models DeepSeek officially lists right now are V4 Pro and V4 Flash. Everything else is people reading tea leaves.

It helps to look at the cadence. DeepSeek shipped V3, then V3.1, then V3.2, then V4, all inside roughly a year. V4 itself only landed on 24 April 2026. That's recent. A clean, ground-up V5 this soon would break the pattern hard. What's far more likely in the near term is a tuned V4.x or a stable V4 release that drops the "Preview" label. Useful, but not the generational leap the hype crowd is promising.

So if your planning depends on V5 arriving by a particular date, build for the world where it doesn't. You'll be right more often.

Why everyone thinks DeepSeek V5 is imminent (and why they're wrong)

Here's the load-bearing myth. DeepSeek's API docs note that the model names deepseek-chat and deepseek-reasoner will be discontinued at 15:59 UTC on 24 July 2026. A bunch of posts took "deprecation on July 24" and ran with "new model on July 24." You can see the logic. You can also see why it's wrong if you read one more sentence.

Those two names are legacy aliases. They currently point at V4 Flash, one for non-thinking mode and one for thinking mode. DeepSeek is retiring the old generic names and asking everyone to call the models by their real names instead. The migration is a one-line change: swap your model field to deepseek-v4-flash or deepseek-v4-pro. Your base_url stays the same. Your API key stays the same. That's the whole job.

If you're calling the OpenAI-compatible endpoint, it's literally model: "deepseek-chat" becoming model: "deepseek-v4-flash" and you're done. I migrated my own jobs in about five minutes, and most of that was hunting down where past-me had scattered the string. The point is, nothing about your integration breaks. It just stops accepting an old name.

So this isn't a roadmap leak. It's housekeeping. The 90-day countdown started the day V4 shipped, which is exactly how you'd phase out old aliases after a new generation lands. If DeepSeek were dropping V5 on the 24th, they'd be sending you to call deepseek-v5-something. They're not. They're sending you to V4.

Does this mean V4 Flash is going away?

No, and this is the part the hype posts get backwards. V4 Flash is the destination, not the casualty. The legacy names are being pointed somewhere permanent, and that somewhere is the explicit V4 model IDs. If you've hard-coded deepseek-chat in some script from 2025, sure, go fix that before July. But your actual model isn't disappearing. Nothing about this date implies a new release. It's a rename with a deadline.

What DeepSeek V4 actually is right now: Pro vs Flash

Before we daydream about V5, it's worth being precise about what shipped, because half the articles out there get the basics wrong.

DeepSeek V4 launched as a Preview on 24 April 2026, both variants at once, open-weight on Hugging Face under the MIT license. The two models:

• V4 Pro: a 1.6T-parameter Mixture-of-Experts model with around 49B active parameters per token. This is the premium tier.
• V4 Flash: 284B total, around 13B active. Fast, cheap, the volume workhorse.

Both run a 1M-token context window. Both support thinking and non-thinking modes.

A quick word on why the active-parameter numbers matter, because they're the whole reason Flash is cheap. In a Mixture-of-Experts model, only a slice of the network fires for any given token. Flash carries 284B parameters but only lights up around 13B per token, so you get the breadth of a big model at the inference cost of a small one. That's the trick that lets DeepSeek price Flash the way it does. The 1M context, meanwhile, is genuinely useful for the kind of agent work where you're stuffing in a pile of search results and documents and asking the model to make sense of all of it at once. It's not a gimmick number on a spec sheet, at least not for how I use it.

DeepSeek V4 Pro pricing

The numbers, per million tokens:

One correction worth flagging, since I've seen it repeated: the V4 Pro rate isn't a discount anymore. It started as a 75% promo, then became the permanent list price on 22 May 2026. So $0.435 / $0.87 is just what it costs now. Don't budget for the promo expiring, because it already didn't.

Here's where I'll get personal. V4 Flash is my daily driver. I use it constantly through the OpenCode Go subscription, which at the volume I push feels close to unlimited, and it's the engine behind most of what my Hermes setup does all day. Searching, researching, summarising, analysing, the unglamorous connective tissue of agent work. Flash is genuinely more than enough for all of it, and it's quick enough that I rarely sit there watching a spinner.

What I would not do is write code with Flash. Pro is much better for that, and I get into why in my V4 review. But honestly, even with Pro available, my coding rotation is Opus, GPT-5, Minimax M3, and Kimi 2.7 Code. DeepSeek isn't where I reach first for code. It's where I reach first for cheap, fast, high-volume thinking. Different tool, different job.

DeepSeek V4 Pro vs Claude (and GPT-5)

This is the comparison everyone wants, so let's be honest about it instead of cheerleading.

On a chunk of benchmarks, V4 Pro lands in the neighbourhood of Claude Opus 4.7 and GPT-5.5, and it does it at a fraction of the API cost. That part is real and it's impressive. But "near the frontier on some benchmarks" is not the same as "the frontier." On the hardest reasoning and agentic tasks, the top closed models still tend to edge ahead. If raw capability at any cost is your only metric, DeepSeek probably isn't your answer yet.

Where DeepSeek wins, decisively, is the price-to-performance ratio. That's the actual story, and it's a big one. You can run a lot of competent inference for very little money, which changes what kinds of products and agent loops are even economically viable. I've shipped things with Flash that would've been silly to attempt at Claude prices.

Make it concrete. Picture an agent that does a hundred small research-and-summarise passes a day, each one chewing through a decent slab of context. On Flash, that's pocket change, the kind of bill you forget to look at. Run the same loop on a frontier closed model and suddenly you're doing actual maths before you let it run unattended. The capability gap on that particular task is small to none. The cost gap is enormous. That asymmetry is the entire reason I default to Flash for high-volume agent work, and it's why I think the "vs Claude" framing slightly misses the point. They're not really competing for the same job.

How to compare them without fooling yourself

Vendor benchmark slides are marketing. If you want a read you can trust, lean on standardised harnesses like SWE-bench Pro or DeepSWE that run the same tasks against every model with the same grading. And test on your workload. A model that tops a leaderboard can still fall apart on your specific agent loop, and the only way to know is to run it. I've been burned by trusting a headline number more than once.

The DeepSeek vision API gap, and why agent users are still waiting

Here's a detail that trips people up. DeepSeek does have vision now. It added image recognition to the chat interface and the mobile app back on 29 April 2026, which SCMP wrote up nicely under the headline that the whale can finally see. Upload an image, it understands the image. Good, fast, the works.

The catch, and it's a big one if you build agents: it's chat-only. There's still no vision endpoint in the API. No image content type, nothing you can call programmatically. So for anyone wiring DeepSeek into an automated pipeline, the vision capability may as well not exist. A human can paste a screenshot into the chat box and get a great answer. Your agent, running headless at 2am, cannot. And agents are where vision actually earns its keep: reading a chart, parsing a screenshot, pulling structure out of a scanned document, checking what a UI looks like. All the workflows I'd most want it for live on the API side, behind a door that's still locked.

deepseek vision api: the part that matters for Hermes

This is exactly the gap I keep hitting. Inside Hermes, when something needs eyes, I can't hand it to DeepSeek. I route those tasks through Qwen 3.7 Plus instead, purely because Qwen gives me vision over an API and DeepSeek doesn't. It works, but it's a second model in the stack for one missing feature, and I'd happily collapse it back down to one if I could.

So when I think about what would make DeepSeek's next model actually move the needle for me, "another point on a reasoning benchmark" isn't it. A vision API is.

The funding signal, and when DeepSeek V5 might actually land

There's one piece of news that genuinely changes DeepSeek's trajectory, and it's not a model. It's money.

DeepSeek closed its first ever external funding round in June 2026, roughly $7.4 billion, at a valuation reported in the $55 to $59 billion range. SCMP covered the structure: Tencent led it, the EV battery giant CATL crossed over as a strategic investor (an AI-meets-energy bet that's more interesting than it sounds), and notably, Alibaba and ByteDance sat it out. For a lab that had stayed proudly self-funded, that's a real shift. More capital means more compute, which means more concurrent training runs, which is the kind of thing that eventually produces a V5.

But "eventually" is doing heavy lifting there. None of this comes with a date.

If you want a sane way to track the actual signal instead of the rumours, watch two things: DeepSeek's official Change Log and their Hugging Face org. When a new generation is real, it shows up there first, usually with weights and a paper, not as a leak on social media. Everything upstream of those two sources is speculation dressed up as reporting.

What I want from DeepSeek V5

So, the honest answer to "when is V5 coming out" is: unknown, no date, probably not as soon as the hype says, and the July 24 thing is a red herring. But I'm still genuinely excited for it, so here's my wishlist.

Keep the Flash-tier pricing. Add a real vision API so I can finally drop Qwen 3.7 Plus from my Hermes stack. And push the agent reliability up a notch. Do those three things and V5 becomes close to the perfect agent model for how I actually work, and my default in Hermes the day it ships.

Until then, V4 Flash is doing the job, the date nobody can give you is still nobody's to give, and I'd gently suggest ignoring any headline that claims otherwise. I'll update this post the moment DeepSeek puts something official on the Change Log. Not before.

I was wrong to sigh. Hermes Agent is different in ways that actually matter, and after a few weeks of running it for real work, I get why the dev crowd moved over. This is the honest version: what it is, what you can actually do with it, how it stacks up against OpenClaw, and the security tradeoff nobody seems to want to talk about.

What exactly is Hermes Agent?

Hermes Agent is an open-source, model-agnostic autonomous agent from Nous Research that plans, calls tools, and keeps state across sessions, while teaching itself new skills as it goes.

That bolded sentence is the whole pitch. It's not a coding copilot bolted to your editor. It's a persistent agent that you point at a task, and it goes off and does it, remembering what happened last time.

Now, the naming. The naming is a mess and you need to keep three things straight or you'll confuse yourself reading the docs:

• Hermes 4 is the LLM weights, the actual model.
• Hermes Agent is the framework, the thing this article is about.
• Hermes Desktop is the GUI app that hit public preview in June 2026.

The desktop release shipped no new model and no new architecture. It's just a door into the existing agent that doesn't make you live in a terminal. Useful, but don't mistake it for a reinvention.

What makes Hermes Agent interesting is the closed learning loop. It auto-creates skills after hard tasks, searches its own past sessions (it keeps them in SQLite with full-text search), and builds a model of you over time. Most agents forget you the second the session ends. This one tries not to. That's the part worth paying attention to.

What can you do with a Hermes agent?

Here's where I'm going to disappoint the hype merchants. The genuinely good use cases are boring, and that's a feature.

Where Hermes shines is always-on scheduled work. Morning research digests, competitive-intelligence sweeps, watching a trend API or a search feed and pinging you when something moves. The canonical example everyone reaches for is a daily briefing dropped into Telegram or Slack at 8am, and honestly, that example is popular because it works.

My actual usage looks like this. I've got it collecting information on a schedule, running a paper-trading agent, and watching trend APIs and search results so I don't have to babysit a dozen browser tabs. None of that is glamorous. All of it saves me time every single day.

What it's bad at is worth saying plainly. Complex multi-file refactors that need real architectural judgment are not its strength, because the open models I run it on are good but not Claude-Opus good. And for tight interactive coding where I want to see and control every bit of context, the CLI or Claude Code still beats it. Hermes is for the work you'd otherwise forget to do, not the work you're actively thinking hard about.

This is also where I'll gently push back on the discourse. Half of what I read on X during the OpenClaw wave was people breathlessly announcing they'd "automated their business," and when you looked closer it was a script that renamed some files. If you want a realistic sense of what these tools do for an actual small operation, I wrote up where AI agents help small businesses in 2026 without the fairy dust. Hermes is more capable than the OpenClaw stuff was. The honest wins are still the unglamorous scheduled ones.

Hermes Agent vs OpenClaw

So why did OpenClaw bounce off me and Hermes stick?

With OpenClaw, every task I imagined throwing at it had an answer along the lines of "yeah, Claude Desktop already does that, or Claude Code does it better." It never carved out its own reason to exist on my machine. I'm sure people built real things with it. I just wasn't one of them, and I didn't find the hype convincing enough to keep trying.

Hermes is different because of three things: persistent memory, a "Soul" identity layer, and automatic skill creation. The point isn't any single feature. It's that the agent compounds. Recurring work gets cheaper and faster as it accrues skills, so the thing you set up this week is better next week without you touching it. That's a fundamentally different value proposition than "wrapper around a model that runs commands."

I want to be fair, though, because I'm not here to sell you anything. The memory is small. We're talking roughly a 2,200-character memory file and a 1,375-character user file, with no version history by default. Early facts get overwritten as new ones come in. So the "it remembers you" story is real but bounded, and you'll feel the ceiling on long-running personal-assistant work. This is early infrastructure, not a finished memory system, and I'll come back to that.

Hermes agent skills, memory, and the self-improving loop

This is the most technically interesting part of the system, so let's actually dig in.

How skills work

Skills are just markdown files. Each one is a SKILL.md file living under ~/.hermes/skills/, following the open agentskills.io standard. They load by progressive disclosure, which is a fancy way of saying the agent reads only the short description of each skill for free, and pulls the full content into context only when a task actually matches. That keeps the context lean, which matters more than people realize.

The clever bit is that the agent writes its own skills. After a non-trivial task (think five or more tool calls, or some non-obvious workaround it had to figure out), it can save what it just did as a reusable skill via skill_manage. There's also an autonomous Curator that grades, consolidates, and prunes the skill library on a cycle, so it doesn't just accumulate junk.

The mental model that finally made it click for me: memory is for "what" (facts), skills are for "how" (procedures). Keep those two ideas separate and the whole architecture suddenly makes sense.

Where auto-generated skills fall short

Here's my slightly contrarian take, because almost nobody is saying this. The auto-created skills are convenient, and sometimes they're genuinely good. But more than once I looked at one and thought, I could write this better myself.

When I hand-author a skill with proper prompting and the right context, the result is cleaner and more reliable than what the agent scribbles after a task. Claude even has a dedicated skill-creation skill that helps structure new ones, and if you care about the output, a few minutes of deliberate authoring beats auto-generation. (If you want to get the prompting right, I put my approach in prompt engineering best practices for 2026.) Treat auto-generation as a fast first draft, not the finished article.

Same energy on memory and Soul. They're interesting, and I'm glad they exist, but they feel like the first rung on a tall ladder. The memory is too small and not situationally aware enough yet. I'm fairly convinced the next generation of agents will have far more sophisticated, context-aware memory systems, and we'll look back on this as the charmingly primitive beginning. Which is fine. Everything starts somewhere.

How to get Hermes agent (and run it cheaply)

Getting started is straightforward, and the cost story is genuinely good if you set it up right.

You've got two front ends. The Desktop app (public preview since June 2026) and the CLI are the same agent core sharing the same ~/.hermes config, so anything you set up in one shows up in the other. Desktop adds a settings UI so you're not editing YAML, plus a file browser, voice mode, and live subagent watch-windows that let you see parallel work happening. The CLI keeps finer control over context and compaction. I bounce between both. Desktop for monitoring and review, CLI when I want my hands on the wheel.

The part that makes this cheap is the inference. I run mine on my OpenCode Go subscription, which Hermes supports as a first-class provider. Setup is three things:

1. Add OPENCODE_GO_API_KEY=<your-key> to ~/.hermes/.env.
2. Set the provider to opencode-go (base URL https://opencode.ai/zen/go/v1).
3. Pick your model in Settings, or in config.yaml.

Then route deliberately. I use DeepSeek V4 Flash for roughly 95% of what I throw at Hermes, because the easy, high-volume cron and subagent work doesn't need a frontier model and Flash is absurdly cost-effective. I reserve the stronger models (Qwen3.7 Max, Kimi K2.6) for the genuinely hard reasoning. The caps on OpenCode Go are dollar-based rather than request-based, so cheap models stretch almost comically far. You can run an enormous amount of agent work for not much money.

Two gotchas to flag before they bite you, because they bit me. Dotted model IDs sometimes get mangled into hyphens (minimax-m2.7 turning into minimax-m2-7), which produces a confusing "model not supported" 401. And there are routing edge cases where Hermes picks the wrong endpoint format and you get a 401 for no obvious reason. The fix is the same in both cases: pin your provider and model explicitly, start a fresh session, and update Hermes regularly because these are being patched. Check the project's GitHub issues if a specific model misbehaves; the relevant ones were still in flux at the time I set this up.

The security tradeoff nobody mentions

Now the part I actually want you to remember.

Early on, Hermes ran into a Node.js install conflict on my Mac and just... fixed it. Diagnosed it, sorted out the mess, moved on. My first reaction was delight. My second reaction, about thirty seconds later, was a small cold feeling in my stomach.

Because an agent that can fix your machine can also break it. Unsandboxed, with free run of your filesystem and shell, Hermes is a real security risk. It's not malicious, but it's powerful and autonomous, and "powerful and autonomous with root-ish access to your laptop" is exactly the sentence that should make you pause. I've written before about the danger of skipping permissions on coding agents, and the same logic applies here, except Hermes runs on a schedule when you're not watching.

So harden it before you expose it to anything. Switch the terminal backend to Docker so it's containerized. Use gateway allowlists. If you're running the agent on a VPS or home server and reaching it from your laptop, bind the dashboard to a Tailscale IP and put OAuth in front of it, and never, ever expose a dashboard to the open internet. Tailscale plus OAuth is the clean pattern here and it's not much work.

And here's the bigger-picture thought that the Node incident kicked off for me. Google, Microsoft, and Apple are all going to bake agents directly into their operating systems. When that happens, a complex, less-sandboxed, set-it-up-yourself tool like Hermes becomes obsolete for casual and non-professional users, because they'll get something safer and simpler for free, built into the thing they already use. Hermes's real audience isn't the casual user. It's devs and tinkerers who want control, can manage the risk, and have workflows worth the setup cost. That's not a knock. It's just who this is for.

So is it worth it?

Hermes Agent earns the place on my machine that OpenClaw never did. The compounding skills, the persistent memory, the always-on scheduled autonomy that quietly handles the work I'd otherwise forget. It's a genuinely useful tool and I'm keeping it.

But it's preview-stage and it's rough in real places. The context compression can get lossy on long sessions, the memory is too small, and the provider routing has sharp edges. Treat it as powerful early infrastructure, not a polished product, and you won't be disappointed.

Honest bottom line: if you're a dev running cheap inference and you've got unglamorous, always-on workflows that eat your time, set it up this weekend. If you want a casual assistant that just works with zero fuss, sit tight, because the OS vendors are coming for exactly that lane, and they'll do the babysitting for you.

Anthropic had been teasing this thing for months. There were announcements about a model so capable it was deemed too dangerous to release to the public, the whole Mythos saga, that kind of thing. And just the name, Mythos, sounded epic in a way that made my eyes roll a little. Expectations were sky high, which usually means disappointment is loading.

I'll admit I got cynical about it. My honest assumption was that this was more hype than substance, a bit of marketing theatre to keep interest up while the company circles an IPO. So when Fable 5 actually shipped on June 9, much sooner than I expected, I went in ready to be unimpressed.

I was wrong. This is the best coding model I've used, and unlike the last few releases, I can actually feel the difference.

What Is Claude Fable 5?

Quick context before I get to the testing, because the naming is genuinely confusing.

Claude Fable 5 is the first publicly available model from Anthropic's new Mythos tier, which sits a notch above Opus in their lineup. There are technically two models here. Fable 5 is the one you and I can use. Mythos 5 is the exact same underlying model with some safety guardrails removed, and it's locked behind Anthropic's Project Glasswing program for vetted partners. Same weights, different leash. That's the whole distinction.

The specs that matter to a developer: a 1 million token context window, up to 128k tokens of output, a January 2026 knowledge cutoff, and the API id claude-fable-5. It runs in an adaptive thinking mode, so you don't toggle reasoning on and off.

The reason it took this long to reach the public is the cyber angle. The earlier Mythos Preview from April was held back specifically because it was scary good at finding software vulnerabilities. Fable 5 ships with safety classifiers that catch sensitive requests and quietly route them to Opus 4.8 instead, which is what made Anthropic comfortable releasing it widely. More on those filters later, because they're not free.

How I Tested It

If you've read my reviews before, you know I run the same handful of prompts on every model. Identical wording, every time. That's the entire point. It's the only way I can honestly compare a new model against the last one, instead of going on vibes and a warm fuzzy feeling. So Fable 5 got the same four tests as everyone else.

Two websites, same prompts as always

First up, my two website builds: a Sydney coffee roaster site and a pop culture online shop. Same prompts I've used for a dozen models now.

The results were very good. Nice fonts, clean technical output, and some lovely animations. The pop culture store in particular had some cool little details. I'll be honest though, my coffee roaster prompt always seems to converge on a similar design no matter who builds it, so that one didn't surprise me much. And for the pop culture shop, I still slightly prefer the colours and overall style I got back in my Gemini 3.5 Flash review. Taste is taste.

But here's the thing that actually impressed me, and it wasn't the output. It was the process.

Fable 5 wrote the initial HTML fast, like you'd expect. Then it did something no other model has done on this prompt without being told to: it fired up Playwright, took screenshots at different viewport sizes, looked at its own work, and kept fixing the code until it was clean. It tested itself. Repeatedly. Until the thing was flawless across screen sizes. I've seen models claim they've checked their work. This one actually did, and I watched it happen.

The poker simulation that finally worked

This is the one. If you take one thing from this Claude Fable 5 review, make it this.

I have a prompt I've been running for ages: build a Texas Hold'em poker simulation in Go, with six AI players who each have distinct personalities, playing 1000 hands. At the end I want detailed statistics on the players and the games, plus a way to replay any individual hand with verbose stats. It sounds reasonable. It is reasonable. And not a single model has ever one-shotted it.

There was always something broken. Sometimes the poker rules themselves were subtly wrong. More often the players behaved like they'd never seen a deck of cards, making nonsense decisions. And the end statistics rarely added up to anything coherent. Close, occasionally. Never right.

Fable 5 just did it. First try. The rules work, the hand evaluation is correct, and the six personalities actually play poker like distinct humans would. The aggressive one bullies, the cautious one folds marginal hands, and it all holds together across a thousand hands. The stats make sense. The hand replay works.

And it was fast. MiniMax M3 burned something like 40 minutes grinding on its attempt. Fable did this in about 14. That's the gap we're talking about.

Auditing my own site

Last test, an audit of thomas-wiegold.com. This is a hard one to impress me with, because I've audited, fixed, and re-audited that site more times than I'd like to admit. There isn't much low hanging fruit left.

Fable found something genuinely new. It also surfaced a few issues I'd heard before from other tools, which is fine, but the new finding is what mattered. Everything it flagged made sense, nothing was hallucinated filler, and the fixes it then implemented were clean and quick. Probably the best site audit I've ever run, and I say that as someone who's run a lot of them.

The short version of all this: with Opus 4.8 I genuinely couldn't tell you whether it was better than 4.7 or 4.6. The gains were too small to feel. With Fable, I can feel it. That's new.

How Fable 5 Stacks Up on Benchmarks

I usually don't put much stock in benchmark tables, and you shouldn't either, but the numbers here line up with what I felt so they're worth a look.

Fable 5 tops basically everything. It's number one on the Artificial Analysis Intelligence Index, and it posts 80.3% on SWE-Bench Pro. For comparison, Opus 4.8 sits at 69.2%, GPT-5.5 at 58.6%, and Gemini 3.1 Pro at 54.2% on the same test. On the harder FrontierCode Diamond benchmark the gap is brutal: 29.3% for Fable versus 13.4% for Opus 4.8. That's more than double.

Worth noting that the headline comparison table is Anthropic's own, so take it with the usual pinch of salt. But the third party results back it up. It came out on top on Vals.ai and on Cursor's internal benchmark, and the independent reviewers who tested it agree. Simon Willison, who doesn't do hype, called it "a beast" and noted it has a "big model smell," meaning it just knows more than Opus 4.8. Ethan Mollick said it outperformed every other public model by a considerable margin. Dan Shipper scored it 91 out of 100 on his team's senior engineer benchmark, against 63 for Opus 4.8, and called it a "warp drive."

That warp drive framing is the right one. Anthropic's own line is that the longer and more complex the task, the bigger Fable's lead gets, and that matches my experience exactly. The poker sim is precisely the kind of long, fiddly, multi-step job where it pulls away from the pack. For a quick one-line fix you won't notice the difference. For a real build, you will.

The Catch: Price, Limits, and Filters

Right, so it's not all sunshine. This is the part where I temper the excitement, because there's a lot to temper.

First, the price. Fable 5 costs $10 per million input tokens and $50 per million output, which is exactly double Opus 4.8. On claude.ai it eats your subscription usage twice as fast, too. Now, to be fair, it's not a token waster in the way some recent models are. It doesn't ramble or pad. But it still burns through your limits quickly because the per-token cost is just higher.

Then there's the part that actually worries me. Right now Fable 5 is included in the Pro, Max, and Team plans, but only until June 22. From June 23 it gets pulled from subscriptions and you'll need usage credits to keep using it. Anthropic says they "aim to restore" standard access once they have enough capacity, which is the kind of phrasing that doesn't fill me with confidence. If it leaves subscriptions for good, paying API rates for daily use would be too expensive to justify. I really hope it stays. I'll be using it as much as I can in the meantime, partly out of genuine excitement and partly to get my money's worth before the door maybe closes.

Next, the safety filters. Those classifiers I mentioned earlier route cyber, bio, chemistry, and a few other categories to Opus 4.8 instead of Fable. The problem is false positives, and they're real. Scientists have been vocal about it. One medical physicist said he basically can't use the thing because he says the word "nuclear" all day. There are reports of MRI image segmentation getting flagged as bioterrorism. If your work lives anywhere near security or the life sciences, this is a live issue, not a hypothetical one. For my work it hasn't come up, but your mileage will vary.

And then there's the genuinely weird one. Buried in the system card is a fourth, invisible safeguard. For requests related to frontier AI development, things like building pretraining pipelines or designing ML accelerators, Fable will quietly degrade its own answers. No fallback, no notification, nothing. It just gets dumber and doesn't tell you. Critics have called a model that silently makes itself worse "categorically misaligned," and I get the argument. It probably won't touch what you and I do day to day, but a model that decides on its own to be less helpful without saying so is a bad precedent, and worth knowing about.

I'll also just name the elephant. All of this is landing while Anthropic preps an IPO at a roughly $965 billion valuation. The push toward usage based billing and the subscription removal have a certain timing to them. I'm not saying it's a conspiracy. I am saying the incentives are worth keeping in mind when you read the marketing.

Who Should Actually Use It

Cut through all of that and here's my read.

Use Fable 5 if you're doing hard, long-horizon, well-specified work you can hand off and walk away from. Big migrations, multi-day agent runs, deep research, the kind of build my poker sim represents. For that work the doubled token price is nothing compared to the senior dev hours it saves. My practical advice: don't make it your default. Route your routine stuff to Opus 4.8 or something cheaper, and save Fable for the top 20% of genuinely hard tasks. There's also a 90% discount on cached input, so lean on that for stable prompts.

Wait or skip it if you're a scientist or security researcher dealing with constant false positives, if you're on a zero-data-retention contract (Fable forces a mandatory 30-day retention, which is a dealbreaker for some shops), or if you're a high-volume cost-sensitive user where Opus at half the price gets you most of the way there.

The Verdict

Best coding model I've used. Full stop. And I don't say that lightly, given how jaded I'd gotten about new releases.

What sets it apart isn't just that the output is better, although it is. It's how it works. The Playwright self-testing, the way it one-shotted a poker prompt that has defeated every model before it, the thoroughness of the site audit. It behaves less like a code generator and more like a careful engineer who actually checks their work before saying they're done.

It's not AGI. Let's be clear about that so nobody quotes me out of context. It still makes mistakes, it's expensive, and the filters are clumsy. But the hype, for once, was real. This is the first release in a while where the jump over the previous flagship is obvious rather than something I have to squint to find.

So I'm excited, caveats and all. I'll be living in it for the next week. And now the ball is firmly in OpenAI's court. Your move.