Is “Hello” Worth an LLM Call?
A user opens your app and types:
Hello
Meanwhile, the backend gets busy.
Load history. Search the vector DB. Rerank some docs. Build the prompt. Call the expensive model.
A moment later:
Hello! How can I help you today?
Great.
We just woke up half the AI stack to do something an if statement could handle.
This is the less exciting side of AI security: guardrails aren't only there to stop jailbreaks and prompt injection. They can also stop pointless requests from reaching the expensive parts of your system.
Not every request needs to meet the model
Say you're building an internal HR assistant.
Three requests arrive:
Hello
Who won the 2018 World Cup?
How many vacation days do I have left?
You can send all three through the same pipeline.
Or:
Hello
→ greeting
→ quick reply
World Cup
→ off-topic
→ fallback
Vacation days
→ valid
→ retrieval → LLM
Only the last one really needs the full stack.
The OpenAI Agents SDK explicitly supports this pattern. A cheap input guardrail can run before an expensive agent model. In blocking mode, if the guardrail trips, the expensive model never starts and tools don't run.
Amazon Bedrock makes the cost part even clearer: if a Guardrail blocks the input prompt, you pay for the guardrail evaluation, but not for the foundation-model inference.
At that point, your security layer is basically the guy at the door saying:
“He only wants directions to the bathroom. Don't open the VIP room.”
Even valid requests don't all deserve the biggest model
These are both valid:
Where do I reset my password?
and:
Compare parental-leave policies across three countries,
analyze the legal risks, and cite the evidence.
Sending both to your best model makes the code simple.
It also makes the bill simple.
RouteLLM looked at exactly this problem: put a router in front of a stronger and weaker model and decide which query needs which one.
In some of its experimental settings, routing cut costs by more than 2× without sacrificing response quality.
[RouteLLM — arXiv:2406.18665]
The idea isn't particularly magical:
easy → cheap model
hard → strong model
And "Hello" may not need either.
If the answer is already broken, stop generating it
Input filtering is only half the story.
Imagine the model is streaming a long answer.
By sentence two, your guardrail already knows the response has crossed a policy boundary.
You could do this:
keep generating...
another 2000 tokens...
finish
→ moderate
→ throw it away
Very polite.
We let the model finish its essay before putting the essay in the bin.
NeMo Guardrails can instead check streamed output in chunks. If a rail blocks a chunk, the stream can terminate immediately.
There's research around the same idea.
SentGuard checks safety at sentence boundaries while generation is still happening. In its experiments, it detected 90.5% of unsafe cases within two sentences.
[SentGuard — arXiv:2606.02041]
StreamGuard pushes the idea further: instead of waiting until a response is clearly unsafe, it uses the partial generation to forecast whether it's heading there.
[StreamGuard — arXiv:2604.03962]
There is an important caveat:
Stopping the client stream does not automatically mean every provider stops billing every remaining token immediately.
Cancellation and billing semantics vary.
Still, the engineering rule is obvious enough:
Once you know the result is unusable, stop doing unnecessary work on it.
Guardrails can also become the thing burning money
Of course, there's an easy way to ruin this idea:
request
↓
LLM safety judge
↓
LLM topic classifier
↓
LLM intent classifier
↓
LLM router
↓
main LLM
↓
LLM output judge
User:
Hi
System:
“Please wait while the AI committee reviews your greeting.”
Guardrails have a cost too.
A greeting may only need a rule.
Some intent checks can use a small classifier.
An obviously off-topic question probably doesn't need a reasoning model.
Save expensive LLM judges for the cases that are actually ambiguous.
Good AI security isn't about putting another model in front of every model.
It's about knowing when to stop.
Prompt injection? Block it.
Off-topic? Send it home.
Easy task? Route it cheaply.
Sketchy tool call? Don't execute it.
Output already violated the rule? Stop letting it write another 2,000 tokens.
Frontier models are worth paying for.
Just not for everything.
Sometimes good AI security protects your data.
Sometimes it protects your infrastructure.
And sometimes it protects the thing everyone eventually checks:
the API bill.
References
- RouteLLM: Learning to Route LLMs with Preference Data — arXiv:2406.18665
- SentGuard: Sentence-Level Streaming Guardrails for Large Language Models — arXiv:2606.02041
- Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming — arXiv:2604.03962
- OpenAI Agents SDK — Guardrails
- Amazon Bedrock — How Guardrails works
- NVIDIA NeMo Guardrails — Output Rail Streaming