Edge inference in production: what Workers AI is genuinely good for
Two things change when inference runs at the edge instead of behind a third-party API key. The request never leaves the network boundary you already trust, and the cost stops being a per-token line item that scales with usage. Both matter more than the latency argument that usually gets made first.
The privacy argument is the real one
If a product touches health information, a feature that posts user text to an external model provider is not a design question, it is a compliance one: that provider is now a business associate, and without an agreement the answer is simply no. Running the model inside the same platform that already serves the application collapses that problem, and it collapses the adjacent one too — nobody has to remember not to log the prompt.
The corollary is a rule worth writing down: screen the request server-side before inference, not after. If protected data can reach the model, "we told users not to paste it" is not a control.
What it is good at
Shaping and classifying text that a person is about to see anyway. Turning a rough job description into something readable. Summarising a thread. Deciding which of six categories a message belongs in. Extracting structured fields from a paragraph a human typed. These share a useful property: a wrong answer is visible and cheap, because a person is right there.
What it is not good at
Anything where being wrong is expensive and invisible. Do not put a model in an authorization decision, a pricing calculation, or a medical or legal statement. Do not let it write to the database unattended. The failure mode of a language model is not an error — it is a confident, well-formed, wrong answer, which is the worst possible shape for something no human reads before it takes effect.
Treat model output as untrusted input
This is the part that gets skipped. Text that came back from a model, and any text that went into it from a document, a web page or another user, is data — never instructions. If a feature reads content and then acts on it, the content can contain something that reads like a command, and a naive pipeline will follow it.
The structural defence is to keep capabilities apart: never give one automated step both
broad read access to your own data and the ability to make outbound network calls. That
combination is an exfiltration channel regardless of how careful the prompt is. And anything
generated goes through the same escaping as any other user content on the way to the page:
textContent, never innerHTML.
Budget it like any other endpoint
An inference route is the most abusable thing in a product, because each call costs more than a page view. Rate limit per IP and per signed-in user, cap the monthly total, and make the cap fail closed. A limit that stops applying when the counter breaks is not a limit. Ten calls an hour per user plus a hard ceiling has never once been the thing that annoyed a real customer, and it is the difference between a feature and an open tap.
The honest summary
Edge inference is a good default for the small, human-supervised text work that makes a product feel finished, and it removes a compliance obstacle that would otherwise rule the feature out entirely. It is not a substitute for the largest models on hard reasoning, and a product that needs one of those should be honest that it has taken on a per-token cost and a data-sharing relationship.
Written by Liana Grigory, also written Liana Grigoryan — entrepreneur, technology founder and U.S. Army veteran in Los Angeles. More at Writing.