Rules engine, search index, or LLM: which one should actually power your feature?
An LLM is the only one of the three that can give you a different answer to the same question. Choose with that in mind.
You have a feature to build. Route the ticket. Answer the question. Pick the right plan for this user. Three technologies can plausibly do it: a rules engine, a search index, or an LLM. Teams usually compare them on accuracy, and that is the wrong first question. The first question is whether the same input gives you the same output twice, because that single property decides how you test the feature, how you debug it, and what you can tell a customer when it gets something wrong.
| Criterion | Rules engine | Search index | LLM |
|---|---|---|---|
| Same input, same output | Yes, by construction | Yes, until the index changes | Not guaranteed, even at temperature 0 |
| Behavior shifts with server load | No | No | Yes, output can depend on batch size |
| Reproducibility control | The code is the control | The index config is the control | Seed is best effort only |
| Explaining a wrong answer | Read the rule that fired | Read the ranking inputs | Reproduce it first, if you can |
| Ranking a large body of existing content | Brittle | What it is built for | Overkill |
| Inputs nobody anticipated | Falls through | Returns nothing useful | Where it earns its keep |
| Test suite you can trust | Assertions hold | Assertions hold | 1,000 identical prompts, 80 distinct completions |
| Verdict | Default to the rules engine and the search index. Reach for an LLM only when the feature genuinely has to generalize to inputs you cannot list in advance. | ||
Reproducibility is the criterion that decides most features
A rules engine is a decision you wrote down. Run it twice, get the same result twice. A search index is the same story against a fixed index: the ranking moves when you change the index or the query, not on its own.
An LLM does not work that way, and temperature 0 does not make it work that way. In a controlled experiment, sampling one open-weights model 1,000 times on the same prompt at temperature 0 produced 80 unique completions, and the most common one showed up only 78 times. Greedy decoding did not deliver identical output.
The cause is mechanical. Inference kernels are not batch invariant, so a request's output depends on how many other requests were batched alongside it. Read that again in product terms: your feature's behavior can change with server load, not just with your prompt.
You cannot configure your way out of it either. OpenAI's own guidance is explicit that the seed parameter is a best-effort reproducibility control, not a guarantee, and that outputs can change when the backend configuration changes. None of this is new or fringe. Engineers documented hosted GPT-4 and GPT-3.5-turbo returning different answers at temperature 0 back in 2023, long before the mechanism was pinned down.
The variance is a permanent tax, not a launch bug
Non-reproducible infrastructure changes what your team does every week.
- Testing. You cannot assert on an exact string. You end up writing looser checks, which catch fewer regressions.
- Debugging. A user reports a bad answer. You paste the same input and get a different, fine answer. The bug is real and your reproduction is gone.
- Support. "It works differently sometimes" is a hard sentence to put in a help article.
It gets worse when the team chases the wrong fix. The widely repeated folk explanation, floating-point non-associativity plus GPU concurrency, is wrong, so weeks get spent hardening the wrong layer while the feature stays flaky.
A rules engine and a search index give you an answer. An LLM gives you an answer this time.
When a rules engine wins
If you can write the logic down, write it down. Eligibility, pricing tiers, routing by account type, compliance checks, anything where a human can state the policy in a sentence. A rule is auditable, testable, and explainable to the customer who got the answer they did not want. When our evaluation points to a rules engine rather than a model, that is what we build.
When a search index wins
If the answer already exists inside your content, the job is retrieval and ranking, not generation. Help centers, catalogs, documentation, internal knowledge. A good index is fast, cheap to reason about, and wrong in ways you can see and correct by changing the index. Reaching for a model here usually means paying generation costs to restate something you already had, and giving up reproducibility to do it.
When an LLM earns its place
An LLM is the right call when the input space is genuinely open. Free-form user text you cannot enumerate. Judgment across messy, unstructured material. Producing something that did not exist before.
AI-native builds are our specialty, and we run our own in production. Goblyn plans, writes, publishes, and optimizes a full brand blog end to end, with multi-tenant architecture, custom domains, subscriptions, and a feedback loop tuned for both search rankings and citations inside AI assistants. MagicTrips is an AI travel planner that is live and generating affiliate revenue, using Google Places integration and programmatic SEO content produced at scale. Neither of those is a rules engine wearing a costume. Both need open-ended generalization, so both pay the variance tax on purpose.
How the choice gets made before a build is committed
We work scope and feasibility through with an LLM before committing to a build, so the decision rests on evidence rather than enthusiasm. On an AI project, the first fixed milestone is usually a feasibility check. If it points to a rules engine and a good search index rather than a model, we build that instead.
Projects are mapped into fixed milestones with clear pricing before work starts. Milestones stay small, each has a named deliverable and its own demo, and closed means shipped, reviewed, and finished, never "mostly working, we will circle back."
Before an AI feature meets users, we work through a five-point checklist we developed while building Goblyn:
- User input that flows straight into a prompt
- Token spend with no ceiling
- Latency
- Output shapes the code did not expect
- No fallback when a model call fails
If you are weighing this call right now, whether you have just an idea, designs ready, or an existing product, bring us the feature and we will tell you which of the three it needs. You work directly with Alec from start to finish, with no account managers and no hand-offs. We read every inbound message and reply within two business days.
- Temperature 0 is not determinism. In one controlled experiment, 1,000 samples of the same prompt produced 80 distinct completions, with the most common appearing only 78 times.
- The variance tracks server batching, so an LLM feature's output can depend on how many other requests were batched with it, not only on your prompt.
- The popular floating-point-plus-concurrency explanation for LLM randomness is wrong, which is why teams debugging flaky AI features often harden the wrong layer.
- Rules engine for logic you can write down, search index for content you already have, LLM for inputs you cannot enumerate.
- On an AI project our first fixed milestone is usually a feasibility check, and if it points to a rules engine and a search index, that is what we build.
- Sampling the same prompt 1,000 times at temperature 0 from one open-weights model produced 80 unique completions, the most common occurring 78 times.
- OpenAI's guidance states the seed parameter is best-effort reproducibility, not a guarantee, and outputs can change when the backend configuration changes.
- Engineers documented hosted GPT-4 and GPT-3.5-turbo returning different answers at temperature 0 in 2023.
- The folk explanation that floating-point non-associativity plus GPU concurrency causes LLM randomness is wrong, and teams debugging flaky features often chase the wrong fix.
Doesn't setting temperature to 0 make an LLM deterministic?
No. Sampling one open-weights model 1,000 times on the same prompt at temperature 0 produced 80 unique completions in a controlled experiment, and OpenAI's guidance states that the seed parameter is best-effort reproducibility rather than a guarantee, with outputs able to change when the backend configuration changes.
How do you decide between a rules engine, a search index, and an LLM before committing to a build?
Scope and feasibility get worked through with an LLM before a build is committed, so the decision rests on evidence rather than enthusiasm. On an AI project the first fixed milestone is usually a feasibility check, and when that evaluation points to a rules engine and a good search index rather than a model, we build that instead.
We already shipped an LLM feature and it is flaky. What now?
Start by not chasing the folk explanation, since the widely repeated floating-point-plus-concurrency account of LLM randomness is wrong. Leveling up an existing product with new features, performance work, AI capabilities, or rescue work on a stalled codebase is one of the engagement types we take on. We read every inbound message and reply within two business days.