Watching a System You Cannot See: Setting an OpenTelemetry SLO

I shipped an AI feature. People are already using it. From the outside, the logs are clean: no crash, nothing red. I still can't tell whether it's working, or wrong for the one user in twenty whose question doesn't look like anyone else's.

You watch a system you cannot see by giving every AI call a name a database can hold. OpenTelemetry's GenAI semantic conventions turn a model call, a retrieval step, and a tool call into one structured, queryable record. With a success SLO on top of that, you find out before anyone has to email you.

None of this means anything without real traffic hitting the feature first. If you're still working out what a serving layer has to guarantee once real people show up, that's the piece to read before this one.

How do you log an AI call that isn't one request?

A normal web request is one line in a log:

• a path
• a status code
• a duration

A single AI answer isn't one thing. It can involve a model call, a retrieval step reading your own documents, and a tool call to another system, and none of that shows up as one traceable event by default.

OpenTelemetry's GenAI semantic conventions fix that with a fixed vocabulary of field names, so any of those three steps becomes a structured span instead of free text buried in a log line. The spec is still marked Development, not yet Stable, so the exact field names are worth checking against the live page before you build on them.

The real attribute names, confirmed against the current spec:

gen_ai.operation.name = "chat"
gen_ai.provider.name = "anthropic"
gen_ai.request.model = "claude-sonnet-5"
gen_ai.usage.input_tokens = 812
gen_ai.usage.output_tokens = 96

gen_ai.operation.name is the field that lets a model call, a retrieval step, and a tool call share one vocabulary: chat for the model call, retrieval and execute_tool for the other two. A query filtered on gen_ai.operation.name = "retrieval" returns every retrieval call your feature made this week, the same way a status code filters a web log.

How did Honeycomb set an SLO for its query assistant?

Honeycomb sells observability tooling, and it pointed that tooling at its own AI feature: a query assistant that turns a typed question into a Honeycomb query. Once the calls were instrumented, the team set a number they'd hold the feature to.

What Honeycomb measured	The number
Success SLO, at launch	75% over a rolling seven-day window
Success SLO, raised to	80%
Error rate	down close to 15% since launch
Error budget, self-reported	"over half exhausted"

A success SLO is a target percentage you commit to in writing and check on a fixed schedule.

An error budget is what's left of your allowed failure rate before you're supposed to stop shipping new features and go fix what's broken. Honeycomb's own words for where theirs stood: "Could be better, could be worse. Nothing to worry about right now." [1]

Why did complex-schema users use it the most?

Before they measured anything, Honeycomb's team assumed one thing: users with large, complicated schemas would get less value from the assistant, because they'd already know their own data well enough to write the query themselves.

They measured it, and found the opposite. Teams with the biggest schemas used the assistant the most. Honeycomb's own line: "As it turns out, they use Query Assistant the most! Oopsie." [1]

Nobody would have caught that from a demo or a handful of support tickets. A demo shows the feature working on a schema someone picked to look good. The reversal only shows up once real traffic is broken down by who's using the feature most. That breakdown is exactly what an SLO's own instrumentation already records.

How do you set an SLO for your own AI feature?

Start with the calls, not the target. You can't set a number you can't measure yet.

1. Attach the fields. Add gen_ai.operation.name, gen_ai.request.model, and gen_ai.usage.input_tokens / gen_ai.usage.output_tokens to every model call your feature makes, and give a retrieval step or a tool call its own span with the same operation.name field.
2. Pick the metric that matters for this feature. Whether it returned a response isn't the test; a wrong answer returns fine too. Use whatever "success" means for this feature: a correct format, a passed grading check, a tool call that came back with data instead of an error.
3. Set a percentage you're willing to hold yourself to, over a fixed window like Honeycomb's rolling seven days. Write it down somewhere you'll look at again.
4. Decide what happens when the error budget runs out, before it runs out. Pick one, now, while nothing is on fire:
   • Stop shipping new features to that surface until it's fixed.
   • Page someone.
   • Roll back the last change.

Once you have a percentage and a window, the error budget is just arithmetic: 100% minus your SLO target, spent down by every failure inside that window. If you're at an 80% SLO and you're failing 15% of requests, you've spent 75% of the 20% you were allowed.

What does an SLO change for any business?

Say you shipped an AI feature three months ago. Nobody's complained. It looks fine. You still have no way to tell whether it's working for the users who never write in, because a support ticket only tells you about the users who noticed, got annoyed enough to say something, and knew who to tell.

An SLO is how you see the failure before anyone writes in. Without one, you hear about it only when someone emails, and most people who hit the problem never bother.

With how a retrieval cutoff sets its own threshold, you pick a number and act on whatever crosses it, one query at a time. An SLO holds a whole feature to a percentage instead, counting every query in the window together.

Go check one real field name in your own AI feature's traces this week. If you can't find gen_ai.operation.name, or anything like it, on a single call, start there before you set any SLO.

Quick recap

• A single AI answer can involve a model call, a retrieval step, and a tool call. None of that is one traceable event unless you name the fields.
• OpenTelemetry's GenAI semantic conventions give you that vocabulary: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, and the input/output token counts. The spec is Development status, not yet Stable, so check it before you build on it.
• Honeycomb set a 75% success SLO on its query assistant, raised it to 80%, cut errors close to 15%, and reported its error budget as over half exhausted.
• Honeycomb assumed complex-schema users would get less value from the feature. Measured reality was the opposite, and only measuring caught it.
• Pick the metric that matters for your feature.
• Set a percentage you'll hold yourself to over a fixed window.
• Decide what happens when the error budget runs out, before it runs out.

Start Here

The intake at daisyguti.ai/work-with-me is about nine questions and takes a few minutes, held as a live conversation with an AI. It turns your answers into a brief with the specifics that matter: what you're trying to solve, what you've already tried, what a working version looks like to you.

I read that and reply with whether the gap is an evaluation problem you can grade before launch, or a watching-it-live problem like the one in this guide.

Sources

1. Honeycomb, "Improving LLMs in Production With Observability," Phillip Carter, updated September 26, 2023 - https://www.honeycomb.io/blog/improving-llms-production-observability - the 75%-to-80% success SLO, the close-to-15% error reduction, "our error budget is over half exhausted," and the complex-schema reversal, "they use Query Assistant the most! Oopsie."
2. OpenTelemetry, Generative AI semantic conventions - inference spans, Development status as of this writing - https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md - the real gen_ai.* attribute names and the documented chat, retrieval, and execute_tool values of gen_ai.operation.name.