Logs 2 Tracing

This is just my perspective after using logs for years and finally moving to traces rather unexpectedly (and suddenly). I should also add I’m by no means an expert on tracing but have used logs in some way for years.

So logs, right? Love them or hate them, we need them.

when something goes wrong in prod with no logs/traces, image source here

We place(d) them lovingly within our code, sometimes with little puns on DEBUG log level to make another dev’s day. Then came full-text search tools that allow you QoL on your log searches instead of suffering. Log analysers followed close behind that told you how your application is doing, cool features all over in each one. Fancy log parsing with complexities rivaling interstellar jump calculations were becoming common. Why? All to understand whatever is happening when certain things hit the fan.

Why Tracing Though?

Just take a look at this:

looking complex and crowded, but look closer

Now that looks like a blob of stuff not so different than normal logs but NO. It lists all activity within the code, how long it took and…

making more sense now

…this is all related with each other in a parent-child relationship even within the same service. I can already see an overhead view of some methods taking too long and some problematic designs are more visible this way. All connected under a trace with multiple spans per endpoint I want instrumented.

well

Traces can go as deep in to the code as you decide.

and they are searchable

If you really want, you can throw events to capture specific info about a span…

events: logs who know where they are

…but in most cases, attributes will be more than enough. You can turn these attributes to facets in Datadog to be able to search traces using them.

simple blob reduction

I almost forgot, you can also see the actual problem when things turn sour. All the way from the deepest level you setup tracing for.

clear sight to errors

I still use logs within my code for local tests and haven’t deleted them. Instead, moved them to a method(func) that also sends events to relevant traces. If I also need a specific info, I use it in the log and also add it as a span attribute.

By the way, setting up tracing in code is surprisingly easy as well. For Go, the steps are here

Thanks for reading and see you next time.