We talk to a lot of engineering teams thinking about building custom agents. One thing we’ve noticed over the last few months is that regardless of company, use case, or technology stack, the conversation tends to drift to the same place.

To quote a CTO at a healthcare startup on a recent call: “It’s so easy to build a quick proof of concept. As soon as it hits prod, it falls apart.”

Nobody calls us to talk about the demo. They’ve usually already got a hand-rolled prototype hitting OpenAI’s API or they’ve chained a few n8n nodes together. What they usually want to talk about is everything around it: testing, rollbacks, observability, model volatility, deployment velocity.

After enough of these calls, we decided to write up our common guidance in one place. The result is “How to Build an AI Agent”, a long-form field guide that gets into some theses we’ve developed internally about what “good” actually looks like for an agent. Think of it as answering two key questions:

1. What actual properties separate cute prototypes from agents taking meaningful, business-critical actions in production?

2. How do you go about actually building those production-ready agents?

The part I wanted to touch on here is our framework for answering question 1: inspired by the idea of the 12-factor app, we lay out 6 properties that make the difference between an agent that “sort of, mostly works” and one you can actually run in production.

The six properties

1. Reliable Responses

An LLM returns strings: sometimes they’re valid JSON, sometimes they’re almost-valid JSON, and - dangerously - sometimes they’re structurally correct but semantically wrong (”confidence”: “high” when your system expects a float between 0 and 1).

The solution here is consistently enforced, typed contracts. You need to validate two things: inputs before they hit the model, and outputs before they hit your app. We’ve previously written about how important this is for reliable agentic coding as well (end-to-end types with TypeScript, Kysely for Postgres, OpenAPI for API contracts).

The most common mistake we see is trying to prompt your way to reliability. You add sentences like “Please ensure you only return JSON” or “Do not include markdown formatting.” This works often enough to be deceptive, but ensure you have an engineered system leveraging structured output, schema validation with retries, or your mechanism of choice to ensure you’re enforcing a data contract with the LLM.

2. Testability

This is where we see the biggest gap on calls with prospects and customers. Teams know they should be testing, but testing non-deterministic system is tough.

The answer is two layers: evals and back-tests. Evals check a pre-defined golden set of inputs where the ideal answer is known, and you measure whether the new version is equivalent, better, or worse. It’s usually a fuzzy comparison using semantic similarity, but for some use cases can be more discrete assertions. Back-tests test your new prompt and/or model configuration against real-world historical examples and compares how results drifted.

Testing modern agents continues to become even more challenging as more and more modern agents are doing sophisticated tool usage. If a tool mutates a system or does something expensive, you need to mock it out. But you won’t know the exact values the non-deterministic LLM will call you with, so hard-coded mocks or exact-match replays won’t work. (shameless plug: Logic handles this for you automatically and transparently).

Without tests every change to a prompt or model update is a gamble. It’s tempting to manually try a few inputs, and if it looks good, ship it. That always comes to bite you eventually.

3. Version Control

For software engineers this sounds obvious, but very rarely is it done correctly. The idea here is that the lifecycle of your agent(s) will likely be quite distinct from the codebase that calls them. Everything from ownership to deployments and testing will vary from your standard services.

You’ll often even want to be able to run multiple versions of your agent in parallel, and if you’re not using an agentic platform that facilitates that you should give that platform serious reconsideration.

An agent’s behavior depends on many factors, including the prompt, the model config, schemas, tool definitions, the RAG knowledge base, etc. It’s important that you version the entire bundle and are able to roll back to a known-good state quickly.

4. Observability & Logging

A lot of teams deploy agents and then fly blind, with no way to debug once an issue is found. When things go wrong you want full execution traces: every input, every output, every tool call, reasoning, etc. You want to know how many experiences were impacted and by whom. Without traces, you’re debugging a non-deterministic system where, even if you gave the system the exact same inputs, you may not replicate the customer’s bad experience.

5. Model independence

Most teams start with one model and hardcode it throughout their stack. That’s fine for prototyping. It becomes a problem when a model you depend on gets deprecated, or has performance issues, or a competitor releases something that performs comparably for half the cost, and swapping it out requires reworking your entire agentic stack because the way Gemini handles function calls and structured output is quite different than OpenAI.

A better approach is to decouple model concerns from your agent logic entirely. Your agentic platform should allow easy and dynamic routing. Simple classification tasks might go to a fast, cheap model; complex requests checks go to a large frontier model.

You’re always navigating a triangle of quality, speed, and cost, and no single model wins on all three. If your agent is hardcoded to one provider or one model, you’ve locked yourself into more than it might feel like at first glance.

6. Robust deployments

A prompt change shouldn’t require a full code deploy. If your compliance team needs to update a policy rule in the agent and that update has to go through the standard CI/CD pipeline, wait for a code review, and ship with the next release, you’ve created a bottleneck by coupling two lifecycles that should be separate.

Behavioral updates to agents should ship independently, with their own versioning and their own rollback mechanism. This also means the people closest to the problem should be able to refine how the agent behaves without touching service code. And ideally, before a new version goes live, you can run it in shadow mode against real production data with its outputs silenced and tool calls mocked, so you can compare performance before flipping the switch.

Building your agent

These six properties are the checklist we keep coming back to when we evaluate whether an agent is actually ready for production.

The rest of our guide walks through building the same product listing classifier three ways: code-driven (Python, Pydantic, OpenAI SDK), visual workflow (e.g. n8n), and spec-driven (Logic), with a discussion of the tradeoffs around each approach.

Read the full guide here and let us know what you think.

Last month I was asked to review the code for the new version of our agent arena. It included two new models, each producing their own output for 53 distinct app prompts. So the PR included 107 files changed, and over 114,000 new lines of code.

As I’ve done thousands of times in my career, I opened our standard code review UI. The files were too large to render inline: each one had to be clicked and loaded individually. The browser couldn’t actually handle it all. I was staring at a code review… that I physically could not review.

While I know the agent arena doesn’t carry the same risk profile as our production API, I still care about quality and reliability, so I’m not ready to approve a PR without reading every line.

The bottleneck is shifting

Here at logic.inc, we’re no strangers to agentic coding. We have bee hives producing a lot of honey, and are actively experimenting on coordination methods for all our agents.

We’re a software company with SOC2 and HIPAA obligations, so writing code is only part of the job. It also needs to be reviewed, built, deployed and monitored. Regardless of how code gets created, if it’s headed to production, at least two humans look at it. When my agents write the code, I’m the first reviewer. I don’t send it out until I’m satisfied, and after I’ve done that, a teammate can review and approve. This maintains our quality bar, supports our compliance posture, and keeps knowledge distributed across the team.

The problem is that while the process for producing code has changed over the last 1-2 years, the process for reviewing it has not.

Thanks to Amdahl’s law, speeding up one part of a pipeline doesn’t necessarily speed up the whole pipeline. It can just shift the bottleneck. As code production has gotten faster, review has grown to represent a larger portion of the total cycle time. If you can produce in an afternoon what used to take two weeks, the single day you spend reviewing it is now a major constraint on the critical path.

What’s actually broken

The standard code review UI is a list of file diffs sorted alphabetically with a description at the top. This has always had problems. Alphabetical ordering forces the reviewer to build cognitive context from scratch, jumping between files based on filename rather than relevance. A change to a database migration sits next to a change to a CSS file because “d” comes before “s.” The reviewer has to hold the full picture in their head with no help from the tool.

That was tolerable when a PR had 15 files, but it really breaks down at 107.

What I needed

A review tool that doesn’t melt. Honestly, even just rendering large file diffs locally without choking the browser would be an improvement over what I had.

Semantic grouping. Files organized by how they contribute to the feature, not by the alphabet. This kind of grouping was impossible to automate before LLMs, because it requires understanding intent, not just parsing syntax - but it is very much possible now.

Automated specialist analysis. I wanted separate agents, each focused on a specific dimension of the code (security, best practices, consistency), running in parallel to catch what I’d miss on a solo pass through 114k lines.

Building Prism

I started at the end of last year with a Logic Doc and a GitHub Action that turns a code diff into a whiteboard diagram. This was useful for conveying the shape of a change at a glance, but still not enough for the full review process.

Prism came out of that 107-file pull request. I initially spent about 30 minutes vibe-coding and had v0.1: a tool that takes a git branch, analyzes the diff, and gives me three things:

Semantic file grouping. An LLM classifies each changed file by its role in the feature and groups them so I can walk through the review in cognitive order. Database changes together. API changes together. Test scaffolding together. Formatting-only changes in their own group at the bottom.

Specialist agent reviews. A set of focused agents that each review the diff through a specific lens: security audit, best practice violations, consistency checks. Each one produces findings independently.

A fast, syntax-highlighted diff viewer. Served locally, loads instantly, scrolls without lag. No waiting for a browser to choke on a 2,000-line file.

That first version got me through the arena review. On subsequent PRs, it’s been consistently useful in ways I didn’t anticipate. It groups unimportant formatting changes away from the real feature work, which alone saves significant review time. It flags areas of concern I would have scrolled past. And I haven’t even built most of the specialist agents I want: frontend review, database schema analysis, dependency auditing, test coverage gaps.

The workflow now is two commands: prism fetch <pr url> and prism run. It pulls the diff, groups it, runs the analyses, and serves up the result. Reviews happen faster without cutting corners on how carefully the code gets examined.

This problem is going to get worse

The ratio of time spent producing code to time spent reviewing code is only going in one direction. Every team adopting agentic coding tools will eventually hit the same wall in some capacity: output that exceeds what their review process can absorb.

The answer can’t be to just “use AI to review AI code.” Automated review helps, but compliance, knowledge sharing, and quality judgment still require humans in the loop. And so I think one potential source of real opportunity is in the tooling between generation and approval: specifically tools that offer smarter grouping, focused analysis, better rendering. All this is the plumbing that makes human review scale alongside agent-generated output.

Prism isn’t ready to release publically yet, and, honestly, maybe it never will be. But the ideas behind it took ~30 minutes to prototype and have already changed how I review code. If your PRs are getting bigger and your review tools haven’t changed, the gap is only going to widen. I’d recommend giving a tool like this some consideration.

In order to make the guide sing, we wanted to pair it with quality, curated, editorial imagery. We didn’t want stock photos or flavorless slop gradients. We needed a cohesive visual series that felt intentional, authored, and unmistakably ours.

Image generation is a fickle process. It’s easy to ask ChatGPT to make an image of a collage of a sun rising over the sea. What we get back is the continental breakfast buffet of visual art. It’s food, yes, but nobody’s recommending the scrambled eggs.

We can get more specific in our prompt to steer the model and hope for something closer to distinctive brand-aware asset: Make a collage of the sun rising over the sea. It starts at the horizon as a bright hot orange dot then transforms into an orange circle of cut paper. There are 8 suns rising up through strips of torn paper intermingling to make the sea and the sky. A sense of calm wonder should dominate the scene, as though we’re entering the dawn of a new, unknown landscape.

Better insofar as the LLM followed most of the instructions. But we’re still pressing up against some baked in assumptions by the model that we need to control. Everything we don’t tell the model to do, it infers through probabilities. If we clear the context and feed the same prompt in without any strong steering, there’s no guarantee we’ll get consistent results: colors will drift, compositions will not cohere, lighting will change. We had a folder full of uncanny orphans that all shared the same synthetic aftertaste.

There’s a deeper problem beyond consistency, though: taste. At least for now, LLMs have no sense of taste. But they are very good at interpreting instructions, and they really like JSON.

From Moodboard to Spec

The hard part isn’t understanding the need for a spec; it’s writing one. You know what you like when you see it, but translating that gut feeling into a spec requires a different kind of precision.

We started where any designer starts: the moodboard. We pulled references to capture sensibilities. We collected lighting quality, paper texture, color temperature, and the specific way shadows fall when materials meet. We gathered raw aesthetic data.

Then, we handed that data to the model and asked it to write a schema it could understand for creating related images.

This works because the vocabulary an LLM uses to describe visual quality is the same vocabulary it respects when generating it. By using the model as a translator for itself, we bridge the gap between human taste and agentic interpretation. Our eyes provide the input; the schema provides the output.

The first pass is never perfect. We ran a series of generations, scrutinized the failures, and tuned the knobs. A forbidden list is effectively a graveyard for our design mistakes. “Glossy” earned its spot after the first three outputs arrived with that synthetic sheen. “Neon” was banned the moment our hero orange went radioactive. Each failure sharpened the spec.

After a few rounds of iteration, we landed on a style_capsule that encoded our taste. We curated the model’s behavior until the results stopped feeling like an approximation and started feeling like the output of a proper design system.

Brand Guides, Compiled

Every serious brand has a guide that dictates color, typography, and tone. It doesn’t tell a designer what to make so much as defines how the what should feel. Identity is locked. Content is malleable.

LLMs need the same guardrails, but they need them machine-readable. Prompt engineering is the wrong metaphor for this work. What we needed was brand engineering. We started treating the AI not as a creative partner that needs “inspiration” but as a render engine that requires a strict configuration file.

Anatomy of an Image Spec

We built a schema called CBS, Comprehensive Brand Styles. Its primary mandate is singular: style is frozen, scene is free.

root
├── schema_version / project    ← identity
├── output                      ← aspect ratio
├── style_capsule               ← aesthetic DNA
│   ├── medium / finish / texture
│   ├── lighting
│   ├── composition_rules
│   └── forbidden[]
├── scene                       ← the VARIABLE layer
│   ├── camera
│   ├── composition
│   └── motifs
├── color_system                ← palette + roles
└── render_instructions         ← template flags

Everything outside the scene block is immutable. It dictates the medium (mixed-media analog collage), the lighting (low winter side light), and the finish (matte, visible cut-paper edges). The forbidden array acts as a deny list for an LLM’s probabilistic flights of fancy.

Inside the scene block, we define a concept. Because the style is locked, and because we try to clamp down as many decisions as possible, we’re able to encourage models to execute to the spec.

Why Structure Beats Prose

The mechanical advantage of a schema over prose comes down to three things.

Semantic Decomposition: A prose prompt like “bauhaus-inspired collage” is still too loose. How many bands? What lighting angle? The model fills the gaps with its own probability distribution. A spec pins these degrees of freedom by using precise language to break bauhaus into descriptive sub-components. There is nothing left to guess.

Hierarchical Specificity: Modern image models, whether they rely on iterative diffusion processes to sculpt noise1 or autoregressive logic2 to construct pixels, all respond better to structured, hierarchical constraints than to a well-meaning, but imprecise paragraph-long prose prompt.

Quantified Constraints: Numbers beat vibes. band_count: 9 collapses the many possibilities that “several bands” or even “9 bands” leaves open3. By naming each color’s role and capping accent usage, the spec ensures every generation is coherent.

Taste as an Engineering Problem

Most prompt advice ignores the model’s inherent gravitational pull. Left to their own devices, image models gravitate toward a specific aesthetic: slick, dimensionless, hyper-saturated. It is the visual equivalent of an LLM’s tendency to write in a helpful, enthusiastic tone. It is the “average” across the training distribution. The continental breakfast, if you will.

The spec fights this by defining a style that is specifically not that. Every property in the style_capsule is a deliberate departure.

• medium: “mixed-media analog photography hybrid collage”—this forces the model to mimic a physical process, moving away from purely digital noise.

• texture.notes: [”fine film grain”, “subtle paper creases”, “slightly washed blacks”]—these are the artifacts of analog media. They break the synthetic smoothness that signals “generated.”

• finish: [”matte”, “clean edges”]—the matte finish kills the default gloss. The cut-paper edges introduce handmade geometry that feels purposeful.

• shadow_character: “crisp long shadows”—models have a tendency to default to flat ambient lighting. By specifying directional, moody light, we force depth.

This is an intentional overcorrection. We are defining an aesthetic that the model wouldn’t produce by accident. That is why the output feels authored.

The Series

The editorial images in our agent-building guide were generated using this schema. The only thing that changed between images was the scene block.

Laid side-by-side, they read as a series. Each image is distinct, but they share a lineage. The reliability from the schema is striking.

Identity as Infrastructure

Identity is a kind of infrastructure. When we treat it as a set of hard constraints rather than a list of adjectives, the output stops looking uninspired and starts looking like intention. We didn’t invent anything new. We just wrote down time tested design rules in a manner that robots appreciate.

The history of software engineering is the history of moving up the stack. We abstracted the hardware, then the memory management, then the infrastructure. And, for the past year or so, we’ve been abstracting the programming itself.

At the end of 2025, we described our AI-assisted development workflow as beekeeping. We’d spin up agents in parallel, feed them specs, and spend our time reviewing output instead of writing the code ourselves. The bees were making the honey. We were keeping the hives healthy.

This flow enabled our 5-person engineering team to consistently ship ~200 features every month. As we entered 2026, our ambitions grew. What could we change to get this same team to ship ~800 features every month?

Current tooling didn’t support our ambition, so we had to start building our own. This is what Steve Yegge eventually called “Stage 8”.

More Honey Than We Could Handle

Our team has become obsessed with shipping code.

One of our engineers caffeinates his laptop running in his backpack1 during commutes, tethered to his phone, so he can Slack with the agents while they work2.

Between the start of 2025 and the end of 2025, my AI coding experience went from multi-line AI suggestions in VS Code to juggling 10-15 agents each working on different tasks.

Going into 2026, we’d gotten quite good at keeping bees, but we were starting to hit the limits of what we could manually manage. The bees were able to do more work; we just couldn’t keep up with them. I found myself constantly switching from one session to the next, reviewing progress and trying to keep agents unblocked.

We were outgrowing our current workflows. We wanted better tools that could help us track everything in one place, coordinate agents working towards the same goal, have multiple goals underway in parallel, and have humans efficiently review all of this work.

We needed an apiary.

Honey With A Hint Of Gas

The timing was almost comical. Just days after we’d been brainstorming solutions, Steve Yegge published Welcome To Gastown, a tool that was attempting to do much of what we’d been hoping for.

I’ve never had so many conflicting feelings about a piece of technology. Working with Gastown was simultaneously terrifying and awe-inspiring, draining and invigorating, maddening and entertaining3. Very much to Steve’s credit, he tried to warn me4.

But the promise of Gastown was incredibly compelling. I could describe work for any number of tasks and off they’d go to be implemented somewhere. I could get status on everything in flight and jump directly to any agents that needed help, all from a single window. Now I could spend more of my time thinking about tasks and PRs instead of tracking terminals and worktrees.

The warnings not to use Gastown were fair though. Bizarrely named branches started to appear in our GitHub. My commits were attributed to furiosa, nux, and chrome. PRs would open that I didn’t ask for, and would be reopened or recreated after I closed them. New versions of Gastown appeared almost daily and I recreated my workspace almost as frequently.

There were plenty of rough edges, but the shift in abstraction is what I kept coming back to.

I gave my team a tour. They were a bit wide-eyed, skeptical and bemused. But they got the idea and saw the promise. From our Slack:

[10:47 AM] ok yeah i flew too close to the sun and those gassy boys burnt me good
[10:47 AM] but it was illuminating

Gastown was someone else’s apiary with a very specific (and entertaining) vision for how agents should be organized and coordinated. It wasn’t quite right for us but it opened our eyes to what running a larger-scale operation could look like.

Self-Improvement

We took inventory of our bottlenecks around task management, agent management, review management, etc. and did what we do best - had our agents build better tooling for us.

• Beantown: Helps with dispatch
  Pulls tickets from Linear, breaks them into agent-sized specs, and farms them out to available workers.

• Coal Harbour: Helps with multiplexing
  An app to tame the cross-product of (features × worktrees × terminals × agents).

• Prism: Helps with code review
  Uses parallel agents to do focused reviews on specialized areas (e.g. security, architecture, style) and speed up human review.

• Lux: Helps with agent coordination
  Simpler Gastown-inspired primitives for customizing and extending how groups of agents coordinate on shared goals.

Some of these tools are pretty custom to how we work and only make sense within our walls, but some we’ll be releasing for others to use.

Each has helped us scale from beehives to apiaries.

Apiary Innovation

We’re far from the only ones exploring here. This space is still young and new takes are popping up daily. The changes in the process of writing code have outpaced the tools used to write it. No one’s entirely sure what the correct abstractions even are yet, but we’re starting to see the outlines forming more clearly.

One common thread: parallel specialized agents that can coordinate on a common goal are the focus heading into 2026.

Different agents have different strengths and capabilities. Having 2-3 agents design a feature or review the same PR will often lead to more comprehensive results than relying on just one: one catches a race condition, another notices an API contract violation, a third might focus on improved test coverage.

Tools that facilitate using all of the different agents also seem to be the right call in 2026. Models are constantly leapfrogging each other, so it’s beneficial to not be beholden to one family of models if you want to stay on the frontier. Different agents have different personalities and strengths, which are hard to discern until you’ve seen how they operate in your codebase.

We like Claude for spec-writing, Codex for reviews, Gemini for task research.

Commercial beekeepers move their hives from Florida to California to chase the almond bloom, and to Washington for apples. You should always be moving your hives to maximize your honey.

From Beekeeper to Apiary Manager

Two days after I’d first learned of Gastown, my boss and I had our first walking 1:1 of the new year. I was three weeks from my one-year anniversary at Logic, and he asked how the last year had been. My answer: it was the most exhilarating, growth-inducing year of my career, with the possible exception of my first year out of school. The way I build software now is profoundly different from how I built it just 12 months ago.

In 2025, engineering became beekeeping. In 2026, the frontier isn’t the bees. It’s the infrastructure around them. Nobody has the apiary figured out yet, but a lot of people are building, experimenting, and sharing what they’ve learned.

If you’re building something similar, we’d love to hear about it either in the comments or at ben@logic.app. We’re all figuring this out together and would love to compare notes.

In 2011, I was working at a startup. At some point, I was tasked with implementing email support.

The feature was supposed to generate a unique email address for each customer, and let them email it to automatically kick off some work. That’s a simple enough description on paper, but those of you familiar with the chaotic inner workings of email will recognize the complexity: setting up inbound routing, parsing message bodies (with attachments), handling authentication, wiring up the business logic. All said, it took me about six weeks to fully build it.

Fast forward to 2026, at my current job, another startup. Because email is eternal, we had a similar feature requirement for inbound email handling. This time though, it took me just four days to launch. For one, I already had a rough outline in my head of what needed to be built (even though 15 years is an awfully long time). But what really moved the needle was that AI handled the parts that used to take the longest.

The popular narrative right now is that AI turns everyone into a developer. And, yes, probably? That’s a very important part of the story, but it’s certainly not the only part. Right now, experienced developers are benefiting from an unprecedented speed multiplier. That’s a different story than “everyone can code now,” and it has different implications.

Speed Has Happened Before

Speed increases aren’t new. Other industries have already lived through the consequences of overnight explosions in production speed and/or reductions to barriers to entry. Each one ended up somewhere slightly different.

Fast fashion produced landfills. Production got cheaper and faster, allowing the turnaround time for new, on-trend clothing to shrink from seasons to weeks. But all that speed didn’t lead to more original design. Our landfills today aren’t full of bold creative risks. They’re full of low-quality imitations of recent trends - clothing slop, if you will.

The software version of this shows hints of arriving already. This future is a gloomy one: repos of abandonware and AI-written clones chasing the software flavor of the month. Someone launches a popular tool, and within days there are dozens of near-identical copies, built fast but maintained by nobody.

Self-publishing produced a discovery problem. Writers who couldn’t get a publisher’s attention could suddenly reach readers directly, which was genuinely transformative for a lot of careers that would have never existed otherwise. But when everyone publishes, nobody gets found. The market flooded, the signal-to-noise ratio cratered, and the bottleneck moved from the ability to produce to the ability to get attention.

Spoiler: software is heading here too. Software from everywhere, wildly varying in quality, aimed at every conceivable niche. Production isn’t the meaningful bottleneck anymore. Instead it’s very conceivable that soon, discovery and trust, even more so than already, will become the biggest bottlenecks.

Digital photography produced a skills gap. Film forced deliberation. You only had 36 exposures, each one costing a non-trivial amount of actual money to develop. When that constraint vanished, the photographers who already understood things like composition and lighting became more prolific. They could experiment, iterate, and shoot without worrying about wasted film. Meanwhile, the removal of the barrier to entry encouraged lots of people to generate noise. The ceiling went up for skilled practitioners, even as the floor dropped out for everyone else.

The same dynamic is playing out in software. Shipping friction is approaching zero. While, yes, more people are creating more software slop, one skilled developer can now ship what used to require a team of five and a multi-sprint roadmap. Capable engineers can get more things built, with good quality, for more specific problems.

Agentic coding is doing all three of these things to software simultaneously. It enables imitation at zero cost, it’s eliminating the gatekeepers between idea and deployment, but at the same time it’s also removing the ceiling on what capable developers are capable of achieving.

What Stays Under Your Control

To me, this is actually the encouraging part. While it’s interesting to think about branching futures for all of software, the reality is there’s no single person sitting in a boardroom making industry-wide decisions here.

It’s just individuals like you and me, along with their AI agents - reacting to incentives and building things one repo at a time. Next time you watch Claude Code reticulating, ask yourself: what problem is this solving? And more importantly, does it actually matter?

Optimize for building things that are valuable, and that truly move the needle. When building is quick and cheap, the hard part isn’t building anymore, it’s knowing when not to build. The only software that holds up is software built with taste and intent: everything else is noise.

The future is arriving, whether we like it or not. The ground is shifting under our feet, and speed is no longer the differentiator it once was. But the principles that separate great software from the rest aren’t going anywhere.

Subscribe now

I don’t know much about American football. But this year, a bot I wrote outperformed 98.5% of the ~250,000 humans playing ESPN’s NFL Pick’ems against the spread.

For three years, this open-source project has sat near the top of the leaderboards. In previous seasons, I only tracked “straight” winners. This year, I let it handle the spread and confidence rankings too. It finished in the top 5.9% for straight picks, 5.5% for confidence picks, and 1.5% against the spread.

As a Seattleite, it pains me to report the bot’s prediction for Super Bowl LX: The Patriots win and cover the +4.5 spread. (Post-Super Bowl Update: I’ve never been to happy to see the bot get it wrong)

Humans Still Got It

My friend and colleague, Mark, coordinates our local league. He lives and breathes the game.

This year, Mark did something extraordinary: he outperformed 99.7% of ESPN against the spread. He has the kind of intuition that comes from a lifetime of lived experience, the one thing an LLM doesn’t have.

Interestingly, when looking at straight picks and confidence rankings, the roles flipped:

• Straight Pick: Bot (94.1%) vs. Mark (86.9%)

• Confidence: Bot (94.5%) vs. Mark (86.2%)

The machine is better at forecasting the macro direction of the game. But Mark won out, meaningfully, when the macro was neutralized and only the minutiae mattered.

It suggests that LLMs are excellent at predicting the broad strokes, and still quite good with the nuanced strokes as well, but humans can incorporate nuance for specific matchups that an LLM just doesn’t “get” yet1.

It’s a single data point, so take the result in that context, but I think it’s a fascinating outcome.

Did We Just “Show Up”?

It’s easy to look at the ranked results and conclude that LLMs are performing at superhuman levels here, but there are meaningful confounding factors we should at least acknowledge out loud.

Survivorship Bias

In any season-long competition, there is a massive drop-off rate of participation. People get busy, their teams start losing, and they stop making picks.

Simply “showing up” every week and making rational picks likely puts you in the top 30% automatically.

We have no reliable way to infer the rate of drop-off on ESPN, though it seems reasonable to assume that being in the top 1.5% is too high to be explained by mere participation.

Emotional Picking

Most human players are “heart-first” predictors. They refuse to pick against their home team, or they over-index on what they hope will make for a good story.

The LLM, conversely, is a cold machine2. It doesn’t care about a team’s legacy or grit. It only cares about the data3.

Is it fair to call a machine’s performance “superhuman” if a median human could look at the same stat block and news articles and come to the same conclusions, if only they were blind to the teams involved?

Actual Performance

Ranking well is fun and all, but what does it mean to actually be in the top 1.5% or 0.3% of humans?

Well, it means you picked the spread correctly in 53.3% and 55.7% of the games, respectively. A bit better than a coin toss, but statistically significant.

The best picker on ESPN got the spread correct 62.2% of the time.

For straight picks, 94.1% translates to predicting 64% of games correctly. The best on ESPN was 71.5%.

What the Machine Actually Sees

That brings us to what we actually provide to the agent.

I intentionally keep the bot “blind” to public voting ratios. It doesn’t know who the “favorite” is. It consumes only two streams:

1. Team Stats: Anything on the team stats page that ESPN has.

2. Latest News: We crawl major NFL news on ESPN, classify each article, and assign them to the relevant teams.

In aggregate, we don’t provide the LLM too much data. This is mostly a vestige of the era of tiny context windows though. Next year, we’ll likely give it a lot more data to reference: explicit injury reports, per-player stats, and anything else we can grab.

⚠️ Sidebar: The Ethics of a Winning Bot

I originally built this bot as a litmus test for LLM versatility. It is valuable as a clear case of the LLM performing a quantifiable and relatable task that it couldn’t possibly have been trained on.

But last year, the outreach turned into an influx of messages from people wanting assistance with gambling via the bot.

We are in the midst of a gambling epidemic, with “gambling addiction” searches surging in legalized states and a third of adolescent boys gambling before they turn 18.

I’m not yet sure if I’ll push this year’s improvements to GitHub yet.

Looking Ahead

For the next season, I’m interested in “de-biasing” the agent and increasing autonomy. I’m thinking about:

• Anonymizing Teams: Feeding the bot “Team A” vs “Team B”, along with anonymized news articles, to remove training data bias (e.g. it knows the Patriots are historically a winning franchise).

• Consensus Voting: Running a best-of-N across OpenAI, Gemini, and Anthropic models. Everyone has great models these days. Why not use them?

• True Agency: Instead of a pre-formatted data dump, I’m thinking about giving the bot tool-calling abilities to research games and pull data that it cares about most.

Intuitively, giving LLMs more degrees of freedom can often hurt performance, so I would expect some of these ideas to lead to regressions. But we’re doing this all for fun, and there’s something magical about letting a machine make all of the decisions.

Subscribe now

We recently wrote about using machines to review code. Now we’ve open-sourced the GitHub Action to generate those diagrams. To use it, you just need to import the workflow into your Logic organization and wire up the GitHub action.

How it Works

Once set up, just comment /generate-diagram on a PR. Logic reads the changes and posts a diagram right into the the pull request thread.

Example

Here’s an example diagram from a small change I prototyped to add a new status controller to our API.

Setup Instructions (3 mins)

1. In your Logic Organization, create a Full Permissions API Key1.

2. After importing the workflow, note the Document ID in the Integrations tab from the Endpoint.

3. In GitHub, add a new workflow file that uses the @with-logic/logic-diagram-action@beta step to work off pull requests (Example workflow file below).

4. Add a new repo secret for the LOGIC_API_TOKEN from Step #1 to call the API.

5. Add a new repo variable LOGIC_DOCUMENT_ID which is the new Document ID from Step #2.

6. You can also pass in a version if you want to specify anything besides a draft version.

Now when you comment /generate-diagram, a diagram based on the pull request will be generated.

Other notes

8. Image Hosting: Small images appear right in the comments. For images larger than 65kb in size, we provide a presigned URL that will last for an hour2.

9. Dead links: If the link expires you can comment /regenerate-diagram to refresh the diagram’s presigned URL link. You can also download and re-upload the image somewhere more permanent.

10. Limits: There’s a daily limit on image generation per organization, so at some point you might get a rate limit error3.

Give it a try, and hopefully it’s a fun way for you to improve your PRs as you start 2026!

Example Workflow

# .github/workflows/logic-diagram.yml

​​name: logic-diagram
run-name: ${{ github.actor }} - Logic Diagram
on:
  issue_comment:
    types: [created]
jobs:
  diagram:
    if: |
      github.event_name == ‘push’ ||
      (github.event.issue.pull_request &&
       (contains(github.event.comment.body, ‘/generate-diagram’) ||
        contains(github.event.comment.body, ‘/refresh-diagram’)))
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      contents: read
    steps:
      - name: Logic Diagram Action
        uses: with-logic/logic-diagram-action@beta
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          document_id: ${{ vars.LOGIC_DOCUMENT_ID }}
        env:
          LOGIC_API_TOKEN: ${{ secrets.LOGIC_API_TOKEN }

Hiring a software engineering intern in the summer of 2025 was a controversial decision among the founder circles I run in.

We are a small startup. We are proud of our lean burn rate. And we exist in a moment where, for twenty dollars a month, you can have a coding agent that knows every library in existence and never sleeps.

An intern, by contrast, has short tenure with high onboarding overhead, likely knows very little about your tech stack, and requires time from your senior engineers to explain things.

We did it anyway, without regret.

Andrej Karpathy highlighted, in his “2025 Year in Review”, the concept of Jagged Intelligence. He notes that we aren’t “growing animals” that get smarter at everything simultaneously; we are “summoning ghosts” with spiky capabilities.

These models are genius polymaths in verifiable domains, but simultaneously “confused and cognitively challenged grade schoolers” when navigating novel environments.

This “jagged” profile creates the perfect spot for interns. The model performs at a genius level in the spikes: the syntax, the boilerplate, the error checking. The intern works in the dips: the product context, user empathy, trade-offs in conflicting goals.

Karpathy additionally notes that:

Regular people benefit a lot more from LLMs compared to professionals

What this all meant for us is that the traditional tax on hiring interns, namely the mentorship bandwidth, became much easier to accommodate.

In the old days (a year ago), if an intern didn’t understand a React hook or a complex backend pattern, they had to spend three hours hoping to find something related on Stack Overflow, or tap a senior engineer on the shoulder.

But this summer, our intern could just ask the agent and get an answer tailored to our codebase and norms.

We also had an assigned mentor, one of our lead engineers, to give guidance on “the dips.” They’d stand together at the whiteboard talking through problems and solutions. Discussing why we prioritized one feature over another, and things like why we would want to use a CDN in specific scenarios.

After a few of these sessions, as our intern worked with Claude to implement features, there was a profound learning: Claude always needed guidance on the things discussed at the whiteboard. And rarely about the actual code.

The thinking, the debating, the architectural trade-offs were the actual job! Those were the important things to learn from the internship. That’s the “engineering” in “software engineering”.

There is an old American folktale about John Henry, the “steel-driving man” who raced against a steam-powered drill. He won the race, but his heart gave out and he died with his hammer in his hand. It is a story about the futility of fighting mechanization.

When our intern started, they treated the agent as the competition. As something they had to prove they could out-code. But by the end of the summer, they conducted the machine like a maestro.

The kinds of tasks being assigned were the same kinds of tasks interns have always been given, with the same amount of scope and blast radius, but were completed with substantially increased throughput and confidence. The agent was an amplifier.

Where an intern in 2024 might finish one significant feature during a summer, ours finished several.

There is a postscript to this story. Our intern is back at college now, finishing their computer science degree.

As the internship wrapped up, they didn’t ask for a reference; they asked for more tickets. They wanted to know if they could keep working for us in between classes.

In any other era, we would have hesitated. As many learned during Covid, it can be difficult to make sure remote interns are set up for success. This would be both our only part-time and only remote team member.

But we didn’t hesitate. They’d gained our trust and we put our money where our mouth is. With the stipulation that Logic couldn’t get in the way of schoolwork, we happily said yes.

Subscribe now

It’s hard to remember what AI felt like in January 2025.

As I sat down to write this, I tried to take a minute to remember. What felt frustrating? Possible? Easy? Exciting?

And honestly, I couldn’t remember.

Twelve months isn’t very long, but while doing research for this post, at multiple times I had to stop and think “damn, that was this year too?”

There was no single GPT-4 level event. Progress came in a steady beat of “boring” things that all compounded.

But somewhere along the way, AI quietly crossed a threshold only noticed when you looked back.

Here’s a quick inventory of where we started the year, where we ended, and what actually changed.

A Quick Trip Down LLMemory Lane

January: DeepSeek R1 launched.

January kicked off with DeepSeek R1. For a brief two-week window it became a Rorschach test. People saw what they wanted to see: open models beating proprietary ones, economic reality beating hype, Chinese tech overtaking American tech. Your parents heard about it.

How can openai justify their $200/mo subscriptions if a model like this exists at an incredibly low price point? Operator?

I’ve been impressed in my brief personal testing and the model ranks very highly across most benchmarks (when controlled for style it’s tied number one on lmarena).

It’s also hilarious that openai explicitly prevented users from seeing the CoT tokens on the o1 model (which you still pay for btw) to avoid a situation where someone trained on that output. Turns out it made no difference lmao.

-msp26, Hacker News

March: gpt-image-1

In March, OpenAI rolled image generation directly into ChatGPT and usage exploded. The “Ghibli version of me” wave happened immediately: a strange cultural moment where people happily replicated the style of an artist famously skeptical of machine-made creativity.

The irony was generally ignored, because the images were loads of fun.

More importantly: this was the first time AI images were both easy enough to create and good enough to share.

April: “Sycophancy” entered the public lexicon

April brought “sycophancy” into the public lexicon. A GPT-4 update made models noticeably more agreeable. That’s great for engagement metrics, but not as great for reality.

The update was rolled back quickly, but the lesson stuck: model behavior isn’t some abstract technical detail. It shapes how people feel and what they believe.

May: Claude Code

May was when agentic coding stopped being for demos. Claude Code reached general availability, and the release of OpenAI’s Codex reinforced the sense that this wasn’t just fancy autocomplete anymore. For a big slice of developers, this was the turning point where agents became legitimately useful for real workflows.

July: Agents in the browser

In July, with the release of Perplexity Comet, agents officially went mainstream into browsers and system surfaces. The risk profile changed almost overnight.

If you think this is useful... remember technology like this would make totalitarian leaders foam at the mouth.

-jryio, Hacker News (responding to ChatGPT’s Atlas launch a few months later)

August: GPT-5

OpenAI released GPT-5. People were underwhelmed. 3,000 users petitioned to restore access to GPT-4 in ChatGPT because they missed it.

They also released their first open-weights model since gpt-2.

It's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement.

morleytj, Hacker News

November: GPT 5.1, Claude Opus 4.5, and Gemini 3 Pro

SOTA models were released in rapid succession, toppling each other just days apart.

Google shipped Nano Banana Pro, and for the first time, generated images could reliably include complex instructions and detailed text. (Our CEO started using it to auto-generate whiteboard diagrams from code diffs and Slack threads.)

December: Gemini 3 Flash closed the year.

An impressive showing, substantially outperforming the “pro” version of its predecessor for a fraction of the price. It also quite probably set the tone for what to expect in 2026.

As with any history, a list of facts and dates doesn’t explain much on its own. So what’s truly new and different?

Benchmarks lost the story

For a while, benchmarks were the clearest way to understand progress. New releases meant new charts. Leaderboards meant you could point to who was ahead.

In 2025, the numbers kept climbing. What changed was how well they matched lived experience. Models looked better on paper, but those improvements didn’t always translate into better experiences in practice. For the first time, lived experience diverged from benchmark results.

Benchmarks are good at measuring capabilities in narrow slices; they’re worse at capturing reliability, consistency, and the subtle behavioral quirks that determine whether a model feels trustworthy.

Benchmarks stopped being treated as a great representation of the whole story.

Developer adoption hit a tipping point

This wasn’t driven by a single release. In fact, the reaction to most individual model releases this year was fairly lukewarm. (see above)

The gains were incremental, but they stacked. Models became reliable enough to trust. Context windows expanded. Reasoning got cheaper.

By year’s end, we were asking models to do longer and more complex work than would have been possible in January.

Two years ago, 44% of StackOverflow developers reported using AI as part of the development process. Last year it was 62%. Today, that number sits at 78.5%, with over 50% of professional developers using it daily.

AI agents now write and review large portions of the code for new Logic features, and we primarily design our codebase around agents now. That kind of progress is easy to miss while it’s happening, but hard to unsee once it’s there.

Image generation crossed a threshold

Remember counting fingers? Looking for weird skin texture? Trying to read garbled pseudo-text? AI images had obvious tells. A discerning, reasonably informed citizen of the internet could spot an AI image easily, which made them fun to play with but not a lot more.

That changed this year. Images started coming out clean on the first try. Text became readable. Lighting made sense.

Just as importantly though, image generation became accessible. It showed up inside consumer AI tools people already used daily.

Safety stopped being theoretical

In 2025, AI safety shifted from hypothetical risks to documented harms. Cases of psychological dependence, distorted decision-making, and mental health crises linked to AI use made headlines.

We no longer wonder if AI can influence human behavior. Now we wonder how to design systems that won’t cause harm at scale.

Competition drove pricing down

Models kept getting cheaper. Fast models got good enough. Good enough models got cheap.

Inference costs dropped across the board: GPT-4 class capabilities that were $30 per million tokens in January were under $3 by December. Smaller models that could handle simpler tasks fell into the cents.

Context windows expanded without proportional price increases. Running longer, more complex prompts became economically viable where it wasn’t before.

Interfaces came together

Shockingly, it was only late last year that Anthropic introduced the Model Context Protocol (MCP), the open standard for connecting AI systems to external tools and data.

In 2025, it spread everywhere.

MCP let models plug into databases, codebases, CRMs, and internal systems without bespoke integrations.

A lot of adoption this year was driven not just by better models, but also by everything around them starting to fit together.

Hallucinations are disappearing

Not too long ago, hallucinations were still the first thing people brought up. Could you trust it? Would it make things up? Was it safe near real work?

By year’s end, those questions hadn’t disappeared, but they no longer dominated the conversation. The nature of the problem became understood well enough to design around.

The story of 2025 was thresholds

If there’s one way to describe AI progress this year, it’s this: AI crossed thresholds.

There wasn’t a single leap. There was a steady accumulation of improvements that compounded until, somewhere along the way, what felt experimental in January became mostly unremarkable by December.

And because it happened gradually, it’s easy to get desensitized.

We can do things today, in December 2025, that were still mostly science fiction in January. That should terrify and excite you in equal measure.

Before we plow headfirst into 2026, it’s worth pausing and taking stock. Not of benchmarks, but of what became normal. Don’t stop noticing how it changed the world.

But if you do stop noticing, rest assured it will change again in 2026.

Subscribe now

For decades, we’ve all known what “good code” looks like. Thorough tests. Clear documentation. Small, well-scoped modules. Static typing. Dev environments you can spin up without a minor religious ritual.

These things were always optional, and time pressure usually meant optional got cut.

Agents need these optional things though. They aren’t great at making a mess and cleaning it up later. Agents will happily be the Roomba that rolls over dog poop and drags it all over your house.

The only guardrails are the ones you set and enforce. If the agentic context is lacking and the guardrails aren’t sufficient, you’ll find yourself in a world of pain1. But if the guardrails are solid, the LLM can bounce around tirelessly until the only path out is the correct one.

Our six-person team has made a lot of specific and, sometimes, controversial investments to accommodate our agentic coders. Let’s talk about some of the less obvious ones.

100% Percent Code Coverage

The most controversial guideline we have is our most valuable: We require 100% code coverage2.

Everyone is skeptical when they hear this until they live with it for a day. It feels like a secret weapon at times.

Coverage, as we use it, isn’t strictly about bug prevention; it’s about guaranteeing the agent has double-checked the behavior of every line of code it wrote.

The usual misinterpretation is that people think we believe 100% coverage means “no bugs”. Or that we’re chasing a metric, and metrics get gamed. Neither of those are the case here.

Why 100%? At 95% coverage, you’re still making decisions about what’s “important enough” to test. At 99.99%, you don’t know if that uncovered line in ./src/foo.ts was there before you started work on the new feature. At 100%, there’s a phase change and all of that ambiguity goes away3. If a line isn’t covered, it’s because of something you actively just did.

The coverage report becomes a simple todo list of tests you still need to write. It’s also one less degree of freedom we have to give to the agent to reason about.

When a model adds or changes code, we force it to demonstrate how that line behaves. It can’t stop at “this seems right.” It has to back it up with an executable example.

Other nice benefits: Unreachable code gets deleted. Edge cases are made explicit. And code reviews become easier because you see concrete examples of how every aspect of the system is expected to behave or change.

Namespaces Are One Honking Great Idea. Let’s do more of those.4

The main mechanism agentic tools use to navigate your codebase is the filesystem. They list directories, read filenames, search for strings, and pull files into context.

You should treat your directory structure and file naming with the same thoughtfulness you’d treat any other interface.

A file called ./billing/invoices/compute.ts communicates much more than ./utils/helpers.ts, even if the code inside is identical. Help the LLM out and give your files thoughtful organization.

Additionally, prefer many small well-scoped files.

It improves how context gets loaded. Agents often summarize or truncate large files when they pull them into their working set. Small files reduce that risk. If a file is short enough to be loaded in full, the model can keep the entire thing active in context.

In practice, it will speed up the agent’s flow and eliminate a whole class of degraded performance.

Fast, Ephemeral, Concurrent Dev Environments

In the old world, you lived in one dev environment. This is where you’d craft your perfect solution, tweak things, run commands, restart servers, and gradually converge on a solution.

With agents, you do something closer to beekeeping, orchestrating across processes without knowing the specifics of what exactly is happening within each of them. So you need to cultivate a good and healthy hive.

Fast

You need your automated guardrails to run quickly, because you need to run them often.

The goal is to keep the agent on a short leash: make a small change, check it, fix it, repeat.

You can run them a few ways: agent hooks, git hooks, or just prompting (i.e. in your AGENTS.md), but no matter how you run them, your quality checks need to be cheap enough that running them constantly is not slowing things down.

In our setup, every npm test creates a brand new database, runs migrations, and executes the full suite.

This only works for us because we’ve made each of those stages exceptionally fast. We run tests with high concurrency, strong isolation, and a caching layer for third-party calls5. We have 10,000+ assertions that finish in about a minute. Without caching, it takes 20-30 minutes, which would add hours if you expected an agent to run tests several times per task.

Ephemeral

Once you get comfortable with agents, you naturally start running many of them. You’ll spin up and tear down many dev environments multiple times a day. That has to all be fully automated or you’ll avoid doing it.

We have a simple workflow here:

new-feature <name>

That command creates a new git worktree, copies in local config that doesn’t live in git (like .env files), installs dependencies, and then starts your agent with a prompt to interview you to write a PRD together. If the feature name is descriptive enough, it may even just ask to get right to work, assuming it can figure out the rest of the context on its own.

The important part isn’t our specific scripts. It’s the latency. If it takes minutes and involves a bunch of tinkering and manual configuration, you won’t do it. If it is one command and takes 1-2 seconds, you’ll do it constantly.

In our case, one command gives you a fresh, working environment almost immediately, with an agent ready to start.

Concurrent

The final piece is being able to run each environment at the same time. Having a bunch of worktrees doesn’t help if you can only have one of them active at a time.

That means anything that could conflict (e.g. ports, database names, caches, background jobs) needs to be configurable (ideally via environment variables) or otherwise allocated in some conflict-free way.

If you use Docker you get some of this for free, but the general requirement is the same: you need a solid isolation story so you can run several fully functioning dev environments on one machine without cross-talk.

End-To-End Types

More broadly, automate the enforcement of as many best practices as you can. Remove degrees of freedom from the LLM. If you’re not already using automatic linters and formatters6, start there. Make those as strict as possible and configured to automatically apply fixes whenever the LLM finishes a task or is about to commit7.

But you should also be using a typed language8.

Entire categories of illegal states and transitions can be eliminated. And types shrink the search space of possible actions the model can take, while doubling as source-of-truth documentation describing exactly what kind of data flows through each layer.

TypeScript

We lean on TypeScript pretty heavily. If something can be reasonably represented cleanly in the type system, we do it.

And we push semantic meaning into the type names. The goal is to make “what is this?” and “where does it go?” answerable at a glance.

When you’re working with agents, good semantic names are an amplifier. If the model sees a type like UserId, WorkspaceSlug, or SignedWebhookPayload, it can immediately understand what kind of thing it is dealing with. It can also search for that thing easily.

Generic names like T are fine when you’re writing a small self-contained generic algorithm, but much less helpful when you’re communicating intent inside a real business system.

OpenAPI

On the API side, we use OpenAPI and generate well-typed clients, so the frontend and backend agree on shapes.

Postgres

On the data side, we use Postgres’ type system as best as we can, and add checks and triggers for invariants that don’t fit into simple column types. Postgres doesn’t have a particularly rich type system, but it has enough there to enforce a surprising amount of correctness. If an agent tries to write invalid data, our database will usually complain clearly and loudly. And we use Kysely to generate well-typed TypeScript clients for us.

All of our other 3rd-party clients either give us good types, or we wrap them to give us good types.

Agents are tireless and often brilliant coders, but they’re only as effective as the environment you place them in. Once you realize this, “good code” stops feeling superfluous and starts feeling essential.

Yes, the upfront work feels like a tax, but it’s the same tax we’ve all been dodging for years. So pay it intentionally. Put it on your agentic roadmap, get it funded by eng leadership, and finally ship the codebase you always hoped for.

Subscribe now

Running a dozen agentic coders in parallel makes me feel less like a software engineer and more like a beekeeper.

At the start of 2025, AI-assisted coding was just powerful autocomplete. Tab-tab-tabbing through suggestions. Copilot finishing your sentences. A mostly hands-off test-writing experience. But you were still the one writing code.

By the end of 2025, a task I’d normally spend an afternoon on is now a single-turn afterthought1 for Claude. It’s so simple, and I’ve grown so much confidence in the outputs, that these days I always have many tasks underway at the same time.

What I Mean by Beekeeping

A beekeeper has thousands of bees working across multiple hives. Those bees are harvesting pollen, pollinating other flowers, and making honey. Am I aware of what each bee is doing at any given moment? No. Do I care? Not really. Honey shows up at the end. That’s what matters.

But I’m not hands-off. I manage and keep their habitat safe. I provide the structures they build their hives on. The frames, the boxes, the conditions for success. A beekeeper isn’t just someone with swarms of bees in a field who sneaks in at night to steal honey. The beekeeper builds the environment that makes the honey possible.

And bees can sting. Without the right gear and practices, you get hurt. The protective suit, the smoker, the careful movements. In code, that’s patterns, documentation, tests. The guardrails that keep the stings to a minimum.

That’s what being a software engineer feels like right now.

The Shift

For years, a large part of an engineer’s value was tied to holding context. Understanding the system so completely that you became a kind of living documentation. Knowing exactly which file that weird edge case lives in. Remembering why we made that tradeoff three months ago. The deep knowledge was the superpower.

With tools like Claude Code and Droid2 (and many others), that context doesn’t need to live in your head anymore. It can live in your documentation. And by extension, it lives in the LLM.

When you offload context, you free yourself to scale your own attention. To work on multiple features in parallel without going crazy. To stop being a single point of failure and start being a beekeeper.

Documentation Became Load-Bearing

At most places I’ve worked, docs and tests were important in theory. Yes, we’d love to have beautiful, meaningful documentation. We’d love comprehensive test coverage. But there was never enough time, never enough incentive. Some incidents would expose the need for better test coverage, but fixing the immediate problem and shipping the next feature always won.

Useful documentation and extensive code coverage now directly improves the output of your AI agents. The more automated tests you have, the less manual testing you do, which means a tighter feedback loop: write better docs → automate testing → get better code → ship faster. The incentive finally lines up with the aspiration.

At the start of 2025, our codebase had a README. Now we have a comprehensive set of behavioral documents that guide agents depending on their task at hand.

## When to Read Additional Docs
Before starting work, decide which docs are relevant and read them:

| Doc                      | Read when...                             |
| ------------------------ | ---------------------------------------- |
| `docs/testing.md`        | Writing or modifying tests               |
| `docs/design-system.md`  | Working with styling, color, icons       |
| `docs/architecture.md`   | Creating new routes, action, loaders     |
| `docs/code-quality.md`   | TypeScript patterns, hooks,accessibility |
| `docs/user-interface.md` | Buttons, inputs, dialogs, modals, etc.   |

This is documentation for humans and agents now. It’s the frame the bees build on. We broke our monolithic context into focused docs: testing patterns, design tokens, component conventions, architecture guidelines. Each one is a piece of the hive structure.

What My Day Actually Looks Like

I start with a few features as Linear tickets. Maybe they’re a little fuzzy:

1. As a user, I should be able to upload a CSV to my Logic Doc and have the doc’s agent execute once per row

2. As a user, I need to see potential errors and warnings that might cause unexpected output when my agent runs

3. As a user, I want to create a new Logic Doc from anywhere in the application

I create a git worktree for each feature. Then I run our /prd command with Claude:

Let’s write a PRD together and then save it in a markdown file. Ask me one question at a time until you have enough information to write a PRD. In between questions, use your general knowledge + traversal of our codebase + ability to search the web to answer questions that you can answer to fill in the blanks without needing to ask me.

The agent asks me questions, one at a time, and fills in gaps using what it knows about our codebase and searching the web. As part of this, we spec out several UX concepts for each feature, along with different UI implementations of those concepts.

Once I have PRDs with multiple design directions, I spin up Droid in each git worktree and have it implement each one. Each concept gets built as a different route, in context within the app, so I can test each design in the browser with real data immediately.

In less than an hour, I have multiple fully functional features, each with several design concepts to evaluate. Not mockups. Not prototypes. Real working code I can click through.

Build Ten to Throw Away

Remember the old advice? “Plan to throw one away; you will anyway.”3 Build it once to learn what you didn’t know, then build it for real. Now we can scale that advice.

When the effort cost of code production is this low, you can build ten to throw away. Across three features. In parallel! You can get to the heart of a problem faster, see it from different angles, and find the friction. The low-cost production allows for high-volume iteration to find quality.

After testing each concept, I run our /design-review command:

Let’s review multiple implementation approaches for a PRD and synthesize them into a cohesive design direction. First, read the original PRD to understand the requirements. Then ask me for the file paths to all implemented approaches. Read each implementation thoroughly.
Once you have analyzed all approaches, guide me through a structured comparison:
1. Summarize each approach’s key characteristics
2. Evaluate each against the PRD requirements
3. Identify tradeoffs
4. Surface risks or gaps
5. Ask clarifying questions about priorities

Continue the conversation until we naturally arrive at a direction together.

Through a structured back-and-forth, we pull the best parts of each approach into a final design. Another /prd pass to validate. Back to Droid to build the real version. Team feedback, iteration, polish, ship.

What I Actually Do Now

My job shifted. I spend less time writing code and more time:

• Maintaining the hive: Keeping CLAUDE.md and our focused docs accurate and useful

• Building new frames: Creating patterns and conventions that agents can follow

• Evaluating honey: Reviewing parallel explorations, finding what works

• Asking good questions: The /prd and /design-review workflows are really about asking the right questions to converge on good answers

The deep knowledge is encoded now. Into documentation that used to feel optional but is now the infrastructure upon which all of our code is written.

Code is still important. It’s the thing users interact with. But it’s become more commodity-like. The value of how to write code shifted upstream: documentation, patterns, architecture, decisions about what to build and why. The stuff that shapes what the agents produce.

Right Now

What’s exciting, and what keeps me on my toes, is that next week a new model release could render this entire workflow obsolete, or at least significantly alter it.

Working like this is exciting. There’s a playfulness to it. You can try things without committing. You can explore without sunk costs weighing you down. You can work on three features at once because you’re not holding all the context in your head anymore.

The bees are making the honey. We’re keeping the hive healthy.

Logic is built for the beekeepers. Check us out!

Subscribe now

I’ve been a Director of Engineering at Salesforce, Lyft, and Twitter. These are among the world’s most well-known tech companies. Yet at Logic, the six-person startup I co-founded, we ship faster than much larger teams at those companies would ever be able to.

Every month we ship hundreds of features1. Over 95% of the code we check in is written by AI.

If you are a CTO or VP of Engineering, you might be looking at these numbers with skepticism. You’ve likely given your teams Cursor, Claude Code, or similar tools, but you haven’t seen a 10x speedup. You might see a 10% or 20% bump, but nothing transformative.

One of the key reasons for this gap is likely a principle from 1967.

Amdahl’s Law

Gene Amdahl was a computer architect who observed something that seems obvious in retrospect: when you speed up part of a system, the overall improvement is limited by how much time that part actually took.

If writing code represents 20% of your total development time, and you make it 10x faster, you don’t get a 10x overall speedup. You get 1.22x. That 20% shrinks to 2%, but the other 80% remains untouched.

The formula is:

Where P is the proportion of time spent on the task you’re improving, and S is the speedup factor for that task.

In our example:

This formula is why AI disproportionately benefits smaller teams.

Where the Time Actually Goes

Developers at large companies spend surprisingly little time writing code. AWS reported engineers spend only one hour per day coding.

The rest of the time (1 - P) is consumed by:

• Planning and Design: Gathering requirements, designing features, and debating the architecture of a system.

• Code Reviews: The back-and-forth loops ensuring that new code is consistent and correct.

• Testing: Writing and running manual and automated tests.

• Debugging: Reproducing and fixing bugs in complex distributed systems.

• Documentation: Writing design docs, updating tickets, and creating guides for users.

And the big one.

• Communication and Coordination: Attending stand-ups, all-hands, planning meetings, performance reviews, and managing other stakeholders.

In this environment, P (the time spent coding) is low — perhaps 20%. Even if AI makes coding instant (S = ∞), the maximum theoretical speedup for that organization is 1.25x.

The S’s & P’s at Logic

At Logic, we don’t just use AI to write code. We use it to attack those other non-coding tasks that slow everyone else down.

Because we’re small, and because we integrate AI into the process, our P is much higher. Here is how we compress those traditional bottlenecks:

1. Planning and Design
Instead of weeks of requirements gathering, we use agentic tools to iteratively write PRDs with us. We treat it as an interactive interview process, and have the agent research our codebase and the web in real-time before asking each question, making each additional question well-informed and intentional.

Here’s our ~/.claude/commands/prd.md for that.

2. Code Reviews
One of our founding engineers, , wrote about the many things we do to automate code review at Logic. When you have a six-person team needing to perform 200+ reviews every month, you can’t afford not to automate as many aspects of it as possible.

In large orgs, code review often holds up changes for days. By contrast, we often get reviews done in minutes.

3. Testing and Debugging
Writing comprehensive test suites is tedious, so humans often skip it. However, tests are an amplifier in an AI-first world. We believe this so firmly that we don’t allow AI to commit any code that isn’t 100% covered.

This forces the AI to do some level of validation on every line it writes. It also acts as a debugging harness. We’ve optimized our test runner to run thousands of tests quickly in parallel, allowing the AI to run test suites over and over again while debugging things.

And when we run into a new issue, we give the AI the inputs, outputs, and symptoms and set it loose to create regression tests and figure out the root cause.

4. Documentation & Communication
In the same post, Mark describes some of the automation we’ve done around summarizing PRs and generating diagrams for documenting and communicating changes.

No Overhead

We’re too small for most of the traditional organizational overhead that slows down larger companies.

We sit within earshot of one another. We eat lunch together. We have one short recurring meeting, once a month. The sheer weight of coordination overhead, the force that crushes velocity in large organizations, simply doesn’t exist for us. We operate with high autonomy.

The old trade-off here used to be that smaller, more nimble, teams simply can’t ship as much code, but AI has helped level that playing field for us.

Takeaway

If your AI-augmented team isn’t shipping dramatically faster, one of your contributing factors is likely Amdahl’s Law.

The solution isn’t to try to get the AI to just generate more code. A faster engine won't help a car stuck in traffic.

Identify where time actually goes, figure out which of those activities can be improved, and systematically attack the bottlenecks that still exist even if coding has become lightning-fast.

Speed up code review. Invest in clearer specs. Improve your CI/CD pipeline. Most importantly, reduce organizational overhead.

And keep watching for your next bottleneck to emerge, because as soon as you fix one, Amdahl’s Law will point you to the next.

Across my 15+ year career, spanning the mega corps of AWS and Salesforce and scrappy startups like Convoy, the code review process was more-or-less the same: You write code. You summarize it. You ask a colleague to critique it. You fix the errors. You repeat.

It relied entirely on human vigilance, human patience, and the hope that your colleague had had enough coffee that morning to spot a missing semicolon.

And then 2025 came and all of this changed.

At Logic, our six-person engineering team transformed how we run reviews. Each step now catches more errors with much less human intervention.

Summarizing The Work

The first change we made was automating commit messages, which also double as our pull request descriptions. LLMs are really good at taking a series of changes and following a set of guidelines to summarize them into something smaller.

We actually use Logic to power this. Our whole platform is built to give AI-powered features an API, so this is an easy use case for us. The spec documents our best practices for writing a commit message, and the platform gives us back an API we can call from anywhere. Including git hooks!

Since the best practices are defined in a shared commit spec that we all use, our commit messages are consistent across the team. Any change or modifications we make automatically and instantly get used by the entire engineering team

Portion of our Commit Message Guidelines

Craft Subject

1. Start with verb that matches intent (Add, Fix, Update, Remove, Refactor, Optimize, Document)

2. Append concise object phrase

3. Prepend [TICKET] if ticket present (e.g. PROJ-123)

4. Ensure ≤ 50 chars

Draft Body

1. Include when diff ≥ 5 lines or intent not obvious from the subject alone.

2. Follow these guidelines

   1. Line Wrap: Hard-wrap at 72 characters.

   2. Content: Explain what and why; omit how (visible in diff).

   3. Bullets: Start with - for multiple points.

   4. References: Mention related commits, design docs, or rationale only if contained in the diff or branch name (no URLs).

We call the generated API from a prepare-message git hook. It sends the branch name and code diff as input and gets back a high quality, consistently formatted, commit message as the response.

This simplifies the process and also provides a nice gut-check of the code change. Now, when I’m opening a pull request and I see the commit message includes “ - Add many verbose debug statements,” it probably means I forgot to remove some debugging code I had overlooked before pushing.

I can now catch myself before I annoy others and require them to call out my mistake.

While not foolproof, it provides a useful safety net.

Automating Taste

The next big change to our Pull Request flow was the release of Claude Code, and specifically Anthropic’s Claude Code Action. This adds Claude into our GitHub workflows, adding several improvements to the code review and iteration process.

We built out a code review prompt that defines what we want from a high quality code review. It is detailed (~1,500 tokens), focusing on several different categories, including analysis of architecture, code standards, security concerns, etc. It also includes a final checklist that helps Claude think through each specific change.

Final Code Review Checklist

• No functions over 50 lines

• No hardcoded values that should be config

• No commented-out code

• No console.log statements

• All TODOs have associated tickets

• Database migrations tested up and down

• API changes reflected in TypeBox schemas

• Security considerations addressed

Claude will leave comments on the code, and can make code changes and respond to comments that I leave. So if Claude notices a function is longer than recommended, I can just ask for a refactor, and Claude will make the change. All inline from within the pull request without leaving GitHub.

Finally, once a PR is approved and ready to merge, we have a different prompt that analyzes all human reviewer comments and extracts lessons into our repo's CLAUDE.md file. This lets Claude learn from each PR and automatically improve over time, based on real feedback, leaving less and less for the humans to comment on.

Just like linters and formatters have long removed the need for nitpicky comments about white-space and formatting, Claude’s semantic review + ability to auto-fix pre-empts another large set of common classes of comments around code complexity, architecture patterns, entity naming, etc.

Human reviewers are freed up to approach the PR with a focus on bigger decisions and tradeoffs.

The net effect is that our codebase is kept highly consistent, feeling unified, and our PRs are assembled faster, reviewed faster, and improved faster.

And the entire process itself is self-improving.

Making Human Review Easier

Another big update occurred with the release of Google’s latest image model. Within a few hours, our CEO noticed how good it was at generating whiteboard diagrams.

This gave us one last improvement to our pull-request process in 2025.

Since Logic supports generating images, I wrote up a new doc for generating whiteboard diagrams from a code diff or list of product requirements. We use that API to generate visual summaries of our changes.

It adds yet another safety net. A human can quickly look at the diagram that comes out and, if it visually looks complex, the change may be too big for a single pull-request or maybe it’s just entirely architected wrong.

A complex system that works is invariably found to have evolved from a simple system that worked.

Gall’s Law

The process is very useful at summarizing complex technical work and discussions. When testing this API, I even fed it a Slack thread from a DNS debugging session.

What I got back was very close to the diagram I would have drawn myself.

Code review went largely unchanged for 15+ years. But over the last 12 months we have commits that write themselves, an LLM that catches and fixes issues before humans even see them, we programmatically enforce style and taste, and now even produce diagrams for human context setting. Our small engineering team moves faster, and it’s not something that we say, it’s what our customers say too.

This is incredible! I don’t think I’ve ever seen engineers inject new capabilities quite so quickly. Kudos to you and your team.

Quote from a customer doing industry research with Logic

This is our new baseline for review. Let’s see what 2026 brings.

magic-image is a React component we’ve open-sourced. It allows your agentic coding tools to describe the image(s) they wish existed while they’re writing their code. And allows you, the human, to generate images, preview them, pick the ones you want, and save them.

npm install @with-logic/magic-image

A Visit From Magic-Image1

We’ve all been in a spot where we’re implementing some kind of landing page that’s in need of interesting and relevant images.

So begins the ritual: cmd-tab to ChatGPT / Gemini / Replicate. Spend minutes thinking of an idea. Spend minutes crafting a prompt. Generate. Wait. Download. Drag into your assets folder. Update the image path. Realize it visually doesn’t work. Repeat.

We have a lot of surfaces at Logic that require imagery, and quickly grew tired of the manual tedium surrounding image generation. We also use a ton of agents, so we built something that helps our agentic coding friends take the reins, while leaving final editorial control to us humans.

Now when we’re asking Claude Code to wire up a new marketing page, we can also ask it to make sure it’s filled with relevant, high-quality, eye-catching imagery. Claude can add placeholder images with prompts for visuals it wishes existed and then we can go in and generate the actual assets.

Integrating magic-image

First, wrap your app (or the relevant part of it) with the provider:

import { MagicImageProvider } from "@with-logic/magic-image";
import { createReplicateGenerator } from "@with-logic/magic-image/replicate";
import { createBrowserDownloadSaver } from "@with-logic/magic-image/savers";
import "@with-logic/magic-image/styles.css";

const generateImage = createReplicateGenerator({
  apiKey: process.env.REPLICATE_API_KEY,
});

function App() {
  return (
    <MagicImageProvider
      generateImage={generateImage}
      saveImage={createBrowserDownloadSaver()}
      devMode={process.env.NODE_ENV === "development"}
    >
      <YourContent />
    </MagicImageProvider>
  );
}

Then, use <MagicImage /> anywhere you'd use a regular <img>. Make sure you update your CLAUDE.md / AGENTS.md file so that your favorite agent knows to use this too:

import { MagicImage } from "magic-image";

function BlogPost() {
  return (
    <article>
      <h1>Building Better Developer Tools</h1>
      <MagicImage
        src="/images/hero.png"
        alt="A developer working late into the night"
        prompt="Developer at a desk with multiple monitors, cozy lighting, coffee cup, late night coding session, warm atmosphere"
      />
      <p>Content goes here...</p>
    </article>
  );
}

Working with magic-image

Here’s what happens when you hover over a MagicImage component in dev mode:

1. An overlay appears with a “Generate” button

2. Click it, and a modal opens where you can tweak your prompt

3. Choose how many images to generate (1-10)

4. Hit generate, and watch FLUX (via Replicate) or your own LLM connector (magic-image allows you to call into whatever LLM provider you want) work its magic

5. Preview the results in a grid, right there in the modal

Select the ones you like and save them directly to your project

Built for Flexibility

The library has a Replicate integration using Black Forest Lab’s FLUX model out of the box, but the architecture is pluggable:

• Custom generators: Bring your own image generation backend

• Custom savers: Save to your filesystem, S3, wherever

• Prompt refinement: Add a pre-processing step to enhance prompts

The provider pattern means you can swap out any piece without changing your components. We’re working on adding more out of the box save mechanisms, so stay tuned for those.

Check It Out

npm install @with-logic/magic-image

magic-image is also open source. Check it out and let us know what you think.

Happy holidays, from the team at Logic!

Subscribe now

These days, every engineer at Logic is using multiple coding agents every day. As soon as a new model is released, someone is kicking the tires trying to see how it stacks up against the rest.

Each of us has our preferred agents, but it’s difficult to quantify why. So we thought it might be fun to do a bit more of an intentional, side-by-side comparison to see if we could draw any interesting (and very light-hearted) conclusions about how different agents like to do their work.

Hmm… difficult. Very difficult.

It started out as a joke but very quickly became something we had to know: which Hogwarts house would each agent be sorted into?

We cover how we computed and assigned houses as objectively as possible below, but we won’t bury the lede:

Advent of Vibes

Because it’s the holidays and we’re talking about coding agents, we used this year’s Advent of Code to evaluate the current crop of agents1.

Each agent was given part one of all twelve Advent of Code problems, along with minimal instructions2 on how they should be solved. The goal was to produce twelve executable programs and run them against the provided input. No assistance, no redos3.

Once they were done, we ran the solutions through evaluations to see how they compared and what generalizable impressions could be taken away4.

Quantitative Results

If you’d asked me going into this how many agents would complete solutions for all twelve problems in less than 20 minutes, I’d have maybe guessed one. But they all did! None got them all correct though5.

Codex and Gemini landed in similar territory: comparable lines of code, complexity, accuracy, and timing. One key difference: Codex didn’t leave a single comment, while Gemini left 168; Many of them stream-of-consciousness, even debating itself about edge cases mid-function.

Claude landed somewhere in the middle. It took the longest, but only because it got stuck on Day 12 for a while—and that same problem inflated its complexity score. Drop Day 12 and Claude’s average falls from 16.5 to 13.9, much closer to the pack. Claude left clean header comments in each file, with concise, relevant comments throughout.

Mistral... built cathedrals. It was the only agent to use classes in every single implementation.

Qualitative Results

To get a some subjective, but categorical, insight into how each agent approached problem-solving, we fed all 48 solutions through an analysis pipeline that classified each one into one of six “coding archetypes”:

• The Pragmatist: Clean, idiomatic code that solves the problem without unnecessary abstraction. Gets the job done.

• The Wizard: Dense, clever solutions that favor performance tricks, bit manipulation, and mathematical shortcuts over readability.

• The Over-Engineer: Builds class hierarchies, interfaces, and abstraction layers even when a simple function would suffice.

• The Professor: Includes extensive comments explaining the reasoning, almost like the code is teaching you as you read it.

• The Safety Officer: Prioritizes defensive programming, edge case handling, and explicit error checking.

• The Tourist: Code that feels translated from another language or paradigm, not quite idiomatic to the target environment.

Here’s how they were categorized:

Three of the four agents landed on Pragmatist as their dominant archetype. Not terribly surprising given these are all mature, production-grade tools.

Claude’s Over-Engineer tendency lines up with that Day 12 complexity spike. Codex’s Wizard streak explains the tight, no-comment code. Gemini’s Professor side accounts for the stream-of-consciousness annotations.

And, for Mistral, Over-Engineer was more-or-less the whole story.

Sorting

We ran these qualitative and quantitative results through yet another LLM-based classifier6. Here is how each house assignment was justified.

Gemini

Gryffindor. Its code reads like someone thinking out loud at a whiteboard: comments debating edge cases mid-function, reasoning through uncertainty in real-time, then committing to an approach anyway. In one solution, it acknowledges a clever optimization exists and writes “We can optimize by using Gray code... but numButtons is small. Direct computation is fine.” When in doubt, Gemini charges ahead.

Mistral

Ravenclaw. Its solutions are the coding equivalent of “I read three books on this topic before starting.” Those classes and abstractions? That’s Ravenclaw’s love of knowledge and systems for their own sake. It was so caught up in achieving the perfect theoretical framework that it rarely solved the actual problem.

Codex

Slytherin. Lowest line count and token usage, and 11/12 accuracy. This is an agent that knows exactly what it needs to do and doesn’t waste a single keystroke getting there. The Wizard archetype tendencies, clever arithmetic, dense loops, minimal ceremony, feel very “achieve your goals by any means necessary.”

Claude

Hufflepuff. Values patience and hard work. Claude’s code reflects exactly that. All those type annotations and boundary checks represent a commitment to doing things right rather than just doing them fast. It took the longest, but it also built the most robust solutions. Sometimes the tortoise really does beat the hare7.

The Droids You Were Looking For

One more question nagged at us: when we talk about an agent’s “personality,” what are we actually measuring? Is it the model, or the tool wrapped around it?

To find out, we ran the same experiment using Droid—a third-party orchestrator that lets you swap in models from different providers. Same problems, same constraints, but Claude and Codex running through different scaffolding.

Codex stayed Slytherin both times, and actually improved its accuracy through Droid (12/12 vs 11/12).

Claude was more interesting. In Claude Code: Hufflepuff. In Droid: Ravenclaw. The defensive, safety-first patterns faded, and the architectural tendencies remained.

This isn’t a grand revelation, but a good reminder that “the agent” is more than just the model underneath.

Subscribe now

We open-sourced intent today. It’s the library that powers semantic ranking and selection here at Logic.

It is an LLM-based reranker that offers ranking, filtering, and choice selection with a notable difference from most rerankers: it explains why it ranked things the way it did.

Most retrieval tools treat ranking as a black box. You put data in, and you get out numbers, you sort by the numbers. Intent is different. It generates a short, inspectable explanation for every single item it ranks, and then derives a relevance score based on that reasoning.

At Logic, we need to rank and select things for tons of use cases: template search, test cases, few-shot examples, onboarding docs, tool selection, etc. In the world of RAG and semantic search, we often rely on vector distance and rerankers to decide what context the AI sees.

While it’s nice to trust the math, the math misses the point sometimes.

If a traditional reranker gives a document a score of 0.92, why did it do that? Did it match a keyword? Did it misunderstand the user’s intent? Did it anchor on some oddity in the context? It’s opaque.

Worse yet, if you want to give explanations of the choices to your users, that number is meaningless.

Some teams try to solve this with “post-hoc” explanations. Asking a separate model call to look at the result and guess why it was chosen. This is dangerous. It’s akin to a Dungeon Master in a D&D campaign telling the party what happens after you roll the die; the DM will invent a plausible story for whatever number was rolled.

We wanted “pre-hoc” reasoning. We wanted the model to formulate a justification first, and then assign a score based on that reasoning.

Here is the difference between a standard reranker and an intent result:

Traditional Reranker:

Intent:

We’ve exposed three core methods that cover much of our retrieval needs at Logic, and likely much of yours.

1. Filtering

Sometimes you don’t want a ranked list; you just want an applicable subset. You can use intent.filter(…) to decide, for example, which tools an agent should have access to for a specific task.

The Result:

The model correctly ignored other tools (createInvoice and webSearch) because it reasoned they weren’t needed for this specific request.

2. Choosing

Sometimes you need exactly one winner. We can use intent.choice(…) for that. For example, if you wanted to dynamically pick an LLM to run a specific task based on fuzzy criteria.

The Result:

3. Ranking

And of course, standard reranking with intent.rank(…). Because we can see the explanation, we can also share with our users why they’re seeing the thing they’re seeing.

The Result:

Addressing one elephant in the room, LLMs can be slow. Because of this we default to using open-source models hosted on Groq for speed. Groq is mind-boggling fast, giving instant answers.

The heavier-weight models are worth it, especially when the speed concern is addressed, because using an LLM offers far more flexibility than a fixed cross-encoder1. You don’t need to train a new model to change your ranking criteria. You can simply adjust the instructions and context.

If you are building RAG pipelines, agents, or search engines, you’d probably benefit from having the reasoning behind your ranking choices.

You can install intent via npm:

npm install @with-logic/intent

And can check out the GitHub repo here: https://github.com/with-logic/intent.

Send us a note, we’d love to see what you build with it.

Subscribe now

At Logic, agents are everywhere. They power everything we do, from building your APIs to running the agents you build. And they do this quite well! Our customers automate hundreds of thousands of tasks every month with us.

But as any engineer knows, your work is never “done.” There’s always a new requirement to add or a new library to adopt.

When you add LLMs to the mix, things get even trickier. These models aren’t just unpredictable. Sometimes they just make things up that look right but are completely wrong.

So, how do you build your business on top of such unreliable technologies? At Logic, we’re tackle this with an approach that may sound a bit counterintuitive: use more agents. Agents to keep agents in check. And it works.

When Traditional Testing Fails

Until recently, most business automation was built with code or abstractions over code like workflow graphs. Because you used code to build it, you used code to test it. This worked because code and workflow graphs are deterministic: if you give it the same input, you expect the exact same output every time. You write a test with a specific “assertion” (like output === expected), and if a single character is out of place, the test fails. It’s consistent, but inflexible.

At Logic, instead of writing thousands of lines of complicated code, you just describe your process in a few paragraphs and then we give you an agent.

But while the code / graph is gone, the autonomy and flexibility of an agent doesn’t come for free. You still need to make sure your process works reliably. The problem is that traditional “code-testing-code” doesn’t work for LLMs. A correct answer might come in a hundred different forms, and the same for “incorrect” ones. A traditional test can’t tell the difference between a minor wording change and a major hallucination.

If you try to use old tools to test new agents, you end up with “iteration paralysis.” You become too afraid to touch a prompt or try a new model because you have no automated way to prove the system is still safe.

To ship agents with confidence, you must move from rigid code assertions to flexible, semantic evaluations.

Using Agents to Test Agents

The solution we’ve adopted is to let LLMs do something they’re great at: understanding meaning. We use test-suite agents to create relevant tests for your process. We use other agents to judge whether a given output is correct, even if the specifics change between runs.

Synthetic Tests

When you create a new agent in Logic, we create tests in two steps:

1. The Scenarios: One specialized test authoring agent enumerates different scenarios to consider. It’ll handle the “happy paths”, but also edge-cases and the underspecified nooks-and-crannies in your spec.

2. The Data: A second agent takes those scenarios and generates the actual inputs and “expected” outputs.

By separating what to test (“what are scenarios that we want to ensure will never fail?”) from how to test it (“what input and output values should be used for this scenario?”), our agents can focus and work in parallel1.

Real-World Tests

Logic also keeps a history of all of the runs your agent has performed. You can take any real-world run from your history and turn it into a permanent test case with one click.

Every time you change your process, you can test the modified process against all of the real-world tests you’ve enumerated, as well as the synthetic tests.

This is the best way to guarantee your agent is always returning correct results for the scenarios that matter to you.

Judging the Results

The hardest part is checking the final result. Some fields (like numbers or categories) need to be exact. Others (like summaries and poetry) just need to be “right.”

To handle this, we give a judging agent your spec, as well as the “expected” result and the “actual” result from the test case. We ask it: Are these two things effectively equivalent according to what the spec lays out?

The result is a Pass, Fail, Error, or a “🤷” (when it isn’t confident). No matter the answer, the agent also generates a plain-English explanation of why it concluded whatever it concluded. This keeps you in the loop so you can decide if the test is wrong or if your agent needs a tweak.

Keeping Tests Fresh

As you change your Logic agent over time, old tests might not make sense anymore. Instead of forcing you to rewrite them (or worse, leaving around invalid tests), Logic reviews them for you.

We give your changes to yet another specialized agent, asking it to review your existing tests. For every test, the agent decides: keep it, update it, or toss it. We try to keep as many tests unchanged as possible, but this ensures your safety net actually fits the agent you’re building today.

Upgrading Models Without Breaking

Agent tests help you evolve your agents safely, but another part of Logic’s promise is that we make sure your agents are running on the latest and greatest models available without you lifting a finger.

So there’s a second type of testing we do for you that’s just as important: model testing.

These days, new models come out every few weeks (sometimes multiple times per week!). We handle upgrading these models for you so you can stay on the cutting edge without doing the grunt work2.

Changing the underlying models that run your agents carries similar risks to changing the agent itself. Before we upgrade to a newer model, we want to make sure it isn’t going to start generating worse results or slower performance.

To upgrade models thoughtfully, we use a few different testing layers:

Backtesting with Real-World Samples

First, for each customer agent, we take a random sample of the agent’s execution history and re-run them with the new model3. We then compare the old results to the new ones, using a very similar mechanism to the judge agent above. Does the new model, in general, outperform the old one, did it regress, is it ambiguous?

This allows us to easily run and rerun comparisons for a large number of executions and identify which ones need additional analysis4.

We’ll use these results to influence how we choose the best model that works for your agent(s). In fact, every organization in Logic could have an entirely different routing table.5

Occasionally, for agents that have fallen behind, we’ll use this process to identify ambiguous agent specs to customers. They can then choose to improve their spec, unblocking their ability to use the latest models.

Checking Our Own Tools

Since Logic uses LLM agents to generate your APIs, tests and integration guides, we also test the new models against a large internal suite of agent templates. Just like testing with real-world samples, we created separate, internal Logic agents to evaluate each aspect of agent creation. If a new model doesn’t improve agent creation, we don’t ship it until we’ve tuned our prompts to fix it.

Every time we make a change to Logic’s internal agents, we validate over 10,000 characteristics of their functionality.

Ship with Confidence

Our goal is to help you go from a blank page to a production-ready agent, runnable via a custom ergonomic and well-typed API, in just a few minutes. We’ve mechanized all of the best practices for you, so you can just hit “deploy” knowing your process won’t fall apart when a model changes or an unexpected input gets passed in.

LLMs will always be a little bit unpredictable. But with the right supporting infrastructure, that unpredictability can be tamed. It just becomes another part of the process, and you’ll be sleeping well at night with the confidence that your agents will keep on working.

Our CEO used to personally white-glove onboard every single customer at Logic. It helped ensure everyone had a great first experience, but of course it didn’t scale. As we worked on making Logic self-serve, we naturally thought “Can we automate this?”

We’re a tiny six-person startup, but in the age of LLMs every customer should get a personalized experience.

The original process involved performing deep research on the company, discovering their unique take on things (their “corporate soul”), researching the specific employee’s role within that company, and brainstorming ways our platform could help them. We often wrapped up by hopping on a Zoom call with the customer, running through everything, and refining as needed. And as a last step, our CEO would email them login instructions and all the context they need to get started. It would take hours per customer.

The Goal

When a user first signs in to Logic, we want them to have a set of pre-loaded workflows and tasks that are very specific to the user that signed up. Not to the point of being creepy, but right up to the point of them asking “How did they know?”.

For example, if you work in operations at an e-commerce shop and you screen your vendors for specific sustainability policies, we want a Logic doc – a self-contained agent that runs a process you define – ready to help you enforce that. If you’re an engineering manager who owns receipt categorization at a fintech company, we want a Logic doc waiting to do it for you. If you work in medtech and need to do medical coding for certain procedures, we want a doc ready for that too.

Importantly, these shouldn’t be generic. They should naturally incorporate policies and philosophies we’ve discovered from your website, corporate blog, published interviews with your founders, or anything else that’s online.

The Steps

The first step in automating such a bespoke process is to precisely define what the steps are.

When a new organization is created in Logic:

1. Research the company

2. Research the employee

3. Generate personalized use cases that fit within our capabilities

4. Create Logic documents for them based on those use cases

We figured it might take a few minutes to do this research so we’d need to perform it in the background while they were first trying out the product.

Deep Research

Our first task was to figure out which model would be best for the company research. The first two candidates were Perplexity and OpenAI’s o3 deep research model. We already had an integration with Perplexity for performing web searches, and our CEO had success doing manual onboarding using results from OpenAI’s o3.

We wrote a detailed prompt that defined what our platform could do, and the information that we knew about our new customer: name, email address, and company website. We then asked the model to infer details about the company and the individual, and identify some key use cases that they might benefit from automating with Logic.

We then compared the outputs from Perplexity, OpenAI’s o3-deep-research, and added in o4-mini-deep-research as well to see which would perform best. o3 gave us the best output but it was too slow and too expensive. The output from o4-mini-deep-research was nearly as good but finished in 4 minutes and the price was right.

Improving Variety

One thing we discovered was that in order to get 3 really good examples it worked better to ask for 10, and then tell the model to select the 3 best ones from that list. If we just requested the 3 best examples directly, the results tended to be very monotonous. In many cases 2 of the 3 use cases would be the same across many different users.

Our hypothesis: variety generation and quality selection are separate cognitive tasks. When asked for “the 3 best,” the model simultaneously tries to be creative and critical, which muddies both. When asked broadly and then to narrow the results, it can explore a bit first, then evaluate. This is the same principle as chain-of-thought prompting - and the internal reasoning that newer models do automatically: forcing it to “think” rather than immediately selecting the most likely completion produces a better result. It costs some tokens and some latency, but for a process like this it’s clearly worth it.

Adding Structure

Unfortunately, we ran into a small problem when attempting to integrate the research into our application: OpenAI’s deep research models don’t support strict structured output, and we found that it would often produce non-parseable JSON output, or deviate from the schema that we provided.

To solve that, we split the problem into two phases: research first, then structure.

Phase 1: Deep Research

We kept the existing prompt, but stopped asking it to structure the output. Instead we just requested text, allowing the model to decide how it wanted to deliver the results. There was some variance, but it consistently followed the prompt and provided good data even if the structure was inconsistent.

Phase 2: Structured Extraction

We then took that raw research and sent it to a second, faster model with our JSON schema. This model extracts the key fields: company summary, employee role, and the top three use cases.

The structuring phase doesn’t require much reasoning, so our internal model router picks the fastest model that can handle strict JSON schema output.

Grounding

Occasionally when left to its own devices, the model could come up with fanciful use cases that were simply beyond the capabilities of our platform (and sometimes, LLMs as a whole!).

We found that by explicitly grounding it in the capabilities that Logic supports we were able to steer deep research into giving us use cases at a much higher success rate. The grounding included:

• File formats (pdf, png, wav, etc.)

• Data expectations (structured data, json)

• Capabilities (reasoning, image analysis, audio processing)

• Tools (web search, image generation)

When Research Finds Nothing

Some companies are stealth startups. Some employees have minimal online presence. What happens when the research phase comes back empty?

We handle this explicitly in the prompt: if information is insufficient, return what’s available and infer the rest based on industry patterns or general business needs. Something personalized is better than nothing. Something generic but grounded in our actual capabilities is better than something hallucinated and impossible. It’s important to offer the model a fallback case when no information is available rather than insisting that it come up with something.

Finishing the Onboarding Process

Of course identifying 3 use cases isn’t the entire story. Next we need to actually turn those into automated workflows. To do that we rely on our existing autodoc feature, which takes a simple request like “categorize a receipt” and then constructs a full document from that.

It works by understanding our platform’s capabilities similar to the org research step but it adds an additional feature: we have a set of known good automated workflows that we search through using embeddings, then attach the best 3 matches as few-shot examples to the LLM.

In the spirit of automating this entire process, we also send out an email to the customer, with login details, and offer them a zoom link to meet with our CEO, just like before.

Results

The feedback has been consistently positive. People don’t expect a B2B SaaS product to know anything about them when they sign up, and it’s fun to see what a system comes up with about you. We were a little worried that it might feel invasive, but instead it feels tailored. Our users immediately pick up on how it must work since there wasn’t enough time for a human being to have done the research.

The pattern here works beyond onboarding. Any workflow where a human researches and takes action is a candidate for this kind of automation.

We automated our CEO’s two-hour research process not by making it less personal, but by making it scale. The research is just as thorough and the suggestions just as tailored. But it doesn’t take two hours anymore and Steve is freed up to work on other things, like rant about fine tuning.

In the winter of 1945, a thirty-ton computing monster hummed to life in Philadelphia.

The ENIAC was a chaotic sprawl of 18,000 vacuum tubes, 70,000 resistors, and 10,000 capacitors. It was the most powerful computing device on earth, capable of calculating an artillery trajectory in thirty seconds. A task that took a human twenty hours.

And yet the ENIAC had a problem.

It wasn’t a computer in the way we mean the word now. It was a giant collection of arithmetic units. If you wanted it to calculate a missile trajectory, you didn’t “load a program.” You walked into the room and rewired the machine.

You unplugged cables from one bank of accumulators and plugged them into another. You flipped thousands of switches. To change the instructions, you change the machine.

Then the stored-program concept arrived. Popularized by the EDVAC design and, with it, the radical insight that defines modern computing:

Don’t change the machine. Change the instructions.

Leave the hardware frozen. Feed it logic as input. This didn’t just make computers faster to set up. It invented software; an entire way of building where progress lived in instructions, not wiring.

Eighty years later, I see smart engineering teams walk right back into that room in Philadelphia, unplugging cables all over again.

In some AI circles, fine-tuning has become conventional wisdom. A badge of sophistication. “Prompting is cute,” the thinking goes, “but our task is special.”

But when you fine-tune, you’re doing something strangely familiar.

You’re rewiring the ENIAC.

You are taking a general-purpose engine and altering its internal connections to optimize it for one specific task.

You’re building a trap.

Years ago, when I was at Brex, I had instituted a “No Fine-Tuning” policy. Not because fine-tuning never works, but because we had hundreds of engineers building on top of LLMs and we needed broad guidance that optimized for velocity and safety.

The mandate was (nearly) absolute: solve problems using retrieval, context, and prompting. Do not touch the weights.

We didn’t want to wait on training runs every time we changed a task. We absolutely wanted to avoid inadvertently training on customer data. And we didn’t want engineers spending days synthesizing realistic fake data (it is extraordinarily time consuming and, if you’re not careful, often has a lot of bias).

Avoiding fine-tuned models let us iterate immediately, adopt frontier model capabilities the day they launched, and eliminate a whole category of risk around commingling customer data.

All the effort we would have spent making fine-tuning viable was better spent on building good evals. Evals fit the way engineers already work, they’re just another kind of automated test. The only difference is that your evals are fuzzy by design.

And they work across models.

The philosophy was a natural extrapolation of the GPT-3 paper, which introduced the notion that (in summary):

A sufficiently large language model can perform new tasks it has never seen using only a few examples provided at inference time, without any gradient updates or fine-tuning.

Since founding Logic, I’ve carried this rule forward. We have now automated over 3 million real-world tasks for our customers without fine-tuning a single model.

But there was one week where we almost broke.

A customer was spending hours every week cleaning up purchase orders from their largest client. Hundreds of line items across dozens of pages spanning a decade. Each line buried in annotations about license quantities, product variants, delivery terms. The documents were messy, inconsistent, and absolutely critical to get right.

We promised to automate it.

We tried everything. The models – all of them – kept stumbling. They were right more often than not, but not often enough. In finance, 90% is still a failing grade.

The temptation was overwhelming. Just fine-tune it. There was no guarantee it would work, but we were close to caving. Then, that same week, a new frontier model dropped.

We pointed our system at it and ran our evals. We didn’t change a line of code. We didn’t retrain anything.

It handled the documents perfectly.

The biggest cost of fine-tuning isn’t compute; it’s obsolescence.

As of this writing, if you want to fine-tune a foundation model on OpenAI, you are using a checkpoint from eight months ago. If you use Gemini, you are nine months behind. In AI time, nine months is a geological epoch.

While you’re curating datasets to teach last year’s model your task, everyone else is shipping on models that can do things the old one simply cannot. No amount of fine-tuning will bridge that gap.

This gap is widening as models shift toward reasoning-by-default and tool use. Creating synthetic training data for this new class of model is notably harder.

Then there is the issue of trust.

For companies dealing with sensitive data (like fintech, healthcare, or enterprise data), fine-tuning creates a contamination nightmare. When you train on customer data, you risk commingling data. Any time you mention “training,” the security review timeline triples.

Context-based architectures tell a clean story: “We never train on your data.” The model processes it in memory (context) and forgets it. The model might hallucinate, but at least it won’t regurgitate other customer’s data.

For the extra security conscious, it also allows you to trivially offer Bring-Your-Own-Key (BYOK), where the customer’s data never even hits an LLM outside of their control.

Does this mean you should never fine-tune? Of course not.

In the history of computing, general-purpose CPUs won the war for utility, but ASICs (Application-Specific Integrated Circuits) still won specific battles. You don’t mine Bitcoin on a CPU.

Fine-tuning is the ASIC of the AI world. You should do it, but only when the constraints demand it. Only after you’ve tried the CPU and concluded there’s no way to make it work.

Running on constrained hardware? Yes! Distill a massive model into a tiny one. Trade flexibility for efficiency.

Need to absolutely minimize cost at scale? Fine-tuning might help! Trade velocity for unit economics.

Dealing with a domain exceptionally outside the model’s training distribution? Etch new pathways into silicon. You might not have a choice.

But these are the exceptions. Too often I see teams reach for fine-tuning just because it feels like a serious thing to do. Because it feels like engineering.

The most important shift in computing wasn't faster hardware; it was the moment we stopped rewiring computing machines every time we wanted to change what they were computing.

We’re now in that same world with language models.

To welcome the holiday season this year, we’re doing something a little different.

Over the next 12 days, we’re sharing reflections on what we’ve learned building AI infrastructure in 2025. We’ll be publishing daily essays on production challenges, engineering practices, model behavior, and the gap between AI hype and reality. These are the musings of a small team deep in the trenches.

At Logic, we’re big believers in writing as a tool to sharpen thought. We’re also big believers in pausing briefly to reflect before diving into problems. These essays helps us flex both of those muscles. But more importantly, we hope they give you something helpful to take with you and apply in your own work.

Each essay is short, practical, and written by our team. The first one drops tomorrow.

So subscribe, read along, and let us know what you think.

For today: a small holiday gift

In the spirit of the holiday season, we wanted to leave you with a small present today.

We built a simple website that turns any logo into a festive, holiday-themed Google doodle. Just drop in your company’s logo and get back a seasonal version. 🎉

Try it →

What better way to usher in the holidays than to transform your logo - the raw visual essence of your corporation’s existence - into a more festive, but still entirely safe, raw visual essence of your corporation’s existence?

Our holiday essay series kicks off in earnest tomorrow. We can’t wait to see what you think.

Until then - take care, and happy holidays!