building for a moving target

One of our clients runs a subscription marketing platform for real estate agents — 5,000+ of them log in for the content that keeps a pipeline warm. We'd been building with them for a while when the ask landed: three new AI features. Not one, scoped and staged, with the other two penciled in for next quarter. Three, and soon.

That request is a pretty good snapshot of where software is right now. Capability that would have been a research project not long ago is an afternoon's integration today, and clients can feel that shift — often before anyone's written a ticket for it. The expectation curve has moved about as fast as the models have.

the goal

1. Ship three AI content features — reel scripts, email campaigns, and case study summaries — each generating in the individual agent's own voice.
2. Keep the prompts in the client's hands rather than in the codebase, so their content team could tune output without waiting on a deploy.
3. Build all three in a way that made the fourth cheap.

Which is a good problem. But it does mean the thing you build has to survive the next ask.

what we built

The studio sits on top of a structured picture of each agent and a prompt layer the client controls.

• A market profile questionnaire. Before an agent gets into the studio, they answer a short set of questions — their market, their focus, the tone they want to sound like. Their answers then go through a generation step of their own, which turns free text into the structured profile every other feature reads from. It's a gate, deliberately: personalized generation needs structured input, and asking for it up front is more honest than inferring it badly.
• Admin-configured prompts. The instructions sent to the LLM live in the admin, not the repo. The client's content team writes and tunes them directly, which means improving output quality stopped being an engineering ticket.
• Reel scripts, email campaigns, and case study summaries — three output surfaces, all reading from the same agent profile and the same tone-of-voice setting.

fig. a — one shared path, three outputs

the part that actually mattered

We went back into the schema rather than shipping three parallel one-offs. The AI feature model and the admin layout both got reworked so that a feature became a configuration of a shared shape — inputs, prompt, output — instead of a bespoke build with its own admin screen and its own conventions. It was the less exciting work and it was the work that mattered.

the results

Three AI features, shipped in two weeks.

The pace came directly out of that schema decision. By the third feature, most of what remained was configuration — a new output shape hanging off a path that already existed. And the reusable version arrived at roughly the same time the bespoke version would have, which is the argument for doing it that way while you still can.

But shipping fast is only worth something if you can tell quickly whether what you shipped is any good. That's the other half of keeping up, and it's the less glamorous half: we capture every AI task at full trace sampling, so when a new model lands or a prompt changes, we can evaluate it against exact baselines in an afternoon instead of arguing about it for a week. Speed of delivery without speed of measurement is just guessing faster.

what the numbers actually say

We capture every AI task in Sentry at full trace sampling, which means the percentiles below aren't estimates from a sampled slice — they're the exact distribution of every generation that ran over thirty days. That's a rare thing to be able to say in a performance write-up, so I'll say it.

fig. b — latency by feature, full population

Two things jump out.

Generation is slow, and that's a design constraint, not a bug to fix. A reel script takes about eighteen seconds at the median; an email campaign takes twenty-one and a half, and its slowest runs stretch past thirty-five. No amount of clever engineering on our side makes that meaningfully faster, which brings us to the second thing.

Under a tenth of a second of it is ours. Of a 17.9-second reel script generation, our own infrastructure accounts for roughly 50 milliseconds. Database queries are under 10ms across the board. The model provider is over 99% of the wall clock in every single feature.

That reframes what optimization even means here. There is no profiling win waiting in our code — the entire latency budget is spent somewhere we don't control. Which cuts both ways, and mostly in our favour: it means most of our future performance gains will arrive as somebody else's release notes. That's only a good deal if the part we do own is the part a better model won't hand us for free — the market profile, the tone-of-voice setting, the prompt layer. Thin at the model boundary, thick everywhere else. So the scalability question stops being "how do we make this faster" and becomes "how do we make a twenty-second wait not matter." Which is why every one of these runs as a background task rather than something an agent sits and watches. Async wasn't a performance optimization we reached for later; it's the only shape this product could have had.

It also changes how you choose models. The market profile step — the one that turns questionnaire answers into structured data — runs on a different model than the three content features, and finishes in 2.6 seconds instead of eighteen. That's not a better model, it's a smaller job: roughly 1,800 tokens in, 93 out. Structured extraction with a tiny output doesn't need the model you'd pick for writing something a person will publish under their own name. Matching the model to the shape of the task is most of the cost and latency story.

On reliability, the thirty-day numbers are boring in the way you want: 983 provider calls across both models, zero timeouts, zero rate-limit responses, everything HTTP 200.

what's next

The client's response to three content generators was to ask for something considerably larger: an AI assistant across the whole site — effectively a real-estate-specific brain that knows the platform, the market, and the agent using it.

It's a plausible ask because the shared groundwork exists — when the model layer is reusable, "what if it could do this too" stops being a rewrite and starts being a conversation. Worth being precise about what that readiness is, though: it's architectural, not load-tested. Thirty days of clean reliability at roughly thirty generations a day proves the integration is correct, not that it scales to an assistant fielding questions across a whole site. That's the next problem. It's a much better problem than a rewrite.

the takeaway

You can't scope for how fast this space is moving. What you can do is build so that moving is cheap — put the prompts where the domain experts are, make the model layer reusable before you need it to be, instrument the thing properly so you're arguing from data instead of vibes, and accept that you're learning the shape of these products at the same time as everyone else.

We're still figuring parts of it out. That's the honest state of AI product work right now, and it's the fun part: the ceiling keeps moving up, and every feature you ship makes the next, bigger one look reachable.