How to Tell Whether Your Product Actually Needs an LLM
A five-step test for deciding when to reach for a model and when a plain deterministic feature wins.
Most teams start with "we should add AI" and work backwards to a feature. That is the wrong direction, and it is expensive. Start with the job. Then decide what kind of machine does it.
AI-native builds are our specialty. We build LLM-powered products, agents, and content systems, and we run two of our own in production. That is exactly why we say no to models more often than you would expect. Here is the test we run before we scope a build.
Write the job down as one sentence with no product words in it
Not "an AI assistant for onboarding." Try "turn a customer's free-text description of their problem into one of nine support categories." Or "draft a first-pass reply to a support email in the customer's tone."
The sentence tells you almost everything. If the output is one of nine things, you have a classification problem. If the output is prose that a human will read and judge, you have language generation. Those two need different machines, and the second one is the only one that starts out needing a model.
Watch for the sentence that hides a database query. "Let users ask questions about their account" is usually four report screens and a search box wearing a chatbot costume.
Check whether the task is open-ended or well-defined
This is the fork in the road. Well-defined means the inputs are structured or enumerable, the output space is finite, and there is a correct answer someone could grade. Open-ended means the input is free-form human language and the acceptable outputs are many.
The evidence on the well-defined side is blunt. A Vanderbilt and Beth Israel Deaconess study put GPT-3.5 and GPT-4 up against conventional models on two clinical prediction tasks, and the traditional models were significantly better on both discrimination and calibration. The LLMs stayed worse even when researchers fed them in-context examples. Structured inputs, a defined prediction, and the boring model won outright.
If a rule, a lookup, or a small trained model could get the right answer, that is not a job for a language model. That is a job for a feature.
Price the accuracy you are buying
Sometimes the model really is more accurate, and it is still the wrong call. A 22-dataset benchmark compared twelve text-classification approaches and found LLMs did win on effectiveness, by about 7.1% on average over traditional methods like SVM and logistic regression, and by up to 26% at the top end. The cost of that win was roughly 590x the computation, and the authors' recommendation was to reach for LLMs only when you need the best possible accuracy and can afford it, and to use classic methods for resource-limited applications.
So do the arithmetic before the build. How much is 7 points of accuracy worth on this specific screen? If a wrong classification costs a support agent thirty seconds, not much. If it silently declines a payment, a lot. The number decides, not the enthusiasm.
Run the comparison honestly
Put the two candidate builds side by side on the dimensions that will actually matter after launch. Most of the time the picture is this clear.
| What you are judging | Deterministic feature | LLM feature |
|---|---|---|
| Well-defined prediction on structured data | Significantly better discrimination and calibration in the clinical study | Worse, even with in-context examples |
| Best-possible accuracy on text classification | Baseline | Up to 26% better, about 7.1% on average |
| Compute cost for that accuracy | Baseline | Roughly 590x |
| Speed | Faster | On average 590x slower than traditional methods, 8.5x slower than small language models |
| Open-ended language work with no enumerable output | Cannot do it | The only option |
| Verdict | Reach for the model where the job is genuinely open-ended language work, or where you need the ceiling on accuracy and the budget supports it. Everywhere else the boring feature wins. | |
Ship the smallest version that proves the answer
You do not need a full build to settle this. Take the 10 to 50 real examples from your requirements list, run them through both approaches, and grade the outputs by hand. That is a few days of work and it kills bad ideas before they become roadmap items.
This is why Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027, on escalating costs, unclear business value, or inadequate risk controls. Its analyst's point is worth pinning to the wall: many use cases positioned as agentic today do not require agentic implementations. Those projects were not killed by bad engineering. They were killed by never being tested against a simpler alternative.
We scope work into fixed milestones with pricing agreed before the build starts, so the first milestone on an AI project is often exactly this test. If the answer comes back "you need a rules engine and a good search index," we build that instead. We are not done until it is live and it works.
When the LLM is the right call
There is a real yes here, and it looks like this: the input is free-form human language, the output is language, and nobody can write down the full set of acceptable answers in advance. Drafting, summarizing, rewriting, extracting structure from messy documents, holding a multi-turn conversation, generating content at volume.
We build those. Goblyn is an autonomous AI platform that plans, writes, publishes, and optimizes a full brand blog, with a feedback loop tuned for search rankings and for citations inside AI assistants. MagicTrips is an AI travel planner, live in production and generating affiliate revenue, using Google Places integration and programmatic SEO content produced at scale. Both are genuinely open-ended language jobs. Neither would work as a rules engine.
If you are trying to work out which side of the line your feature falls on, send it over. We read every inbound message and reply within two business days.
Great software starts at zero. We get you to one, with or without a model.
- Write the job as one plain sentence first. If the output is one of a fixed set of things, you have a classification problem, not a language problem.
- On well-defined tasks with structured data, traditional models beat GPT-3.5 and GPT-4 on both discrimination and calibration, even with in-context examples.
- LLMs can win on text-classification accuracy, about 7.1% on average, but at roughly 590x the compute. Decide whether those points are worth the bill on your specific screen.
- Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, and says many use cases framed as agentic do not need an agent at all.
- Say yes to the model when the input is free-form language, the output is language, and the acceptable answers cannot be enumerated in advance.
- Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027 due to escalating costs, unclear business value, or inadequate risk controls, and many use cases positioned as agentic do not require agentic implementations.
- A Vanderbilt/BIDMC study on two clinical prediction tasks found traditional ML models significantly outperformed GPT-3.5 and GPT-4 on discrimination and calibration, including with in-context learning.
- A 22-dataset benchmark of twelve text-classification approaches found LLMs won on effectiveness by up to 26% and about 7.1% on average over methods like SVM and logistic regression, at roughly 590x the computational cost and 590x slower than traditional methods and 8.5x slower than SLMs, with a recommendation to use LLMs only where best-possible accuracy is needed and affordable.
Is an LLM ever worth the extra cost on a classification task?
Yes, when you need the ceiling on accuracy and can afford it. The 22-dataset benchmark found LLMs beat traditional methods by up to 26% on effectiveness, and its authors recommend LLMs specifically for cases where the best possible accuracy matters, with classic methods for resource-limited applications. The trade is roughly 590x the compute, so the question is what those accuracy points are worth on your screen.
How do I test this without committing to a full build?
Collect 10 to 50 real input and output examples, run both the deterministic approach and the model approach against them, and grade the results by hand. It takes days, not months. We often make this the first fixed milestone on an AI project, with pricing agreed before the build starts, so the decision is made on evidence rather than enthusiasm.
You specialize in AI builds. Why would you talk anyone out of one?
Because AI-native builds are our specialty and we run our own in production, which means we know what the models are good at and what they are not. Gartner's analysts note that many use cases positioned as agentic today do not require agentic implementations. Shipping the wrong machine is how projects get canceled, and we are not done until it is live and working.