AI Agent Development

Model selection, and why not the biggest

Most steps in a real agent are small: classify this, extract that, decide whether this looks finished. Those run perfectly well on a small, fast model, and the difference in latency is the difference between an agent that feels instant and one that feels like waiting. I reserve the large models for the steps that genuinely need judgement, and I'll happily run part of a pipeline on a local model when the data shouldn't leave the building at all.

Context engineering

Most agents that behave badly aren't badly prompted. They're badly supplied. The model is being asked to decide something while missing a fact it had no way to know, or drowning in twenty pages of context where two paragraphs mattered. Getting the right information in front of the model at the right step is the majority of the work, and I've written a field guide on exactly this.

Tool design

Tools are the agent's entire interface to your business, and their descriptions are instructions, not documentation. A tool called escalate whose description explains when to reach for it will get used correctly far more often than one that just says what it does.

const tools = [
  {
    name: "get_ticket",
    description: "Fetch a support ticket by id, with its full message history.",
    input_schema: {
      type: "object",
      properties: { id: { type: "string" } },
      required: ["id"],
    },
  },
  {
    name: "escalate",
    description:
      "Hand the ticket to a human. Call this when you are not confident, " +
      "or when the customer is asking for something you cannot verify.",
    input_schema: {
      type: "object",
      properties: { id: { type: "string" }, reason: { type: "string" } },
      required: ["id", "reason"],
    },
  },
];

Evals before it ships

An agent that works on the three examples you tried by hand is not an agent that works. Before anything goes live it runs against a fixture set built from your real cases, including the awkward ones, and the tests assert on the actions taken, not on the wording of the output.

test("escalates instead of replying when a chargeback is threatened", async () => {
  const result = await runAgent(fixture("chargeback-threat.json"));

  expect(result.actions).toContainEqual(
    expect.objectContaining({ tool: "escalate" }),
  );
  expect(result.actions).not.toContainEqual(
    expect.objectContaining({ tool: "send_reply" }),
  );
});

Guardrails and human-in-the-loop

Every agent I build has an explicit answer to “what happens when it's wrong?” Usually that means confidence thresholds with a review queue behind them, hard limits on what any tool can do without a human signing off, and an irreversible-action list that always requires approval. An agent that quietly does the wrong thing is much worse than one that stops and asks.

Observability

Every run is logged with the full decision trail: which tools were called, in what order, and what came back. Usage tracking too, so you can see what the thing costs to operate before the invoice tells you. When an agent does something surprising in month four, that trail is the difference between a fix and a rewrite.

$ agent extract --queue invoices
→ 12 documents waiting
→ read fields from 12 of 12
→ matched 11 against known suppliers
! 1 total did not reconcile
→ queued that one for review
✓ 11 written to the ledger

$ agent extract --queue invoices
→ 12 documents waiting
→ read fields from 12 of 12
→ matched 11 against known suppliers
! 1 total did not reconcile
→ queued that one for review
✓ 11 written to the ledger

One run of a document extraction agent. The line that matters is the one that did not reconcile, because that is the one a person needs to see.