91视频

鈥淢onitoring tells you when something is wrong. Observability helps you understand why.鈥�

The 鈥淲hy鈥�: From Monitoring to Observability

You might wonder, 鈥淒on鈥檛 we already monitor our services?鈥�
While monitoring is a standard practice, it鈥檚 reactive. It answers basic questions like 鈥淚s my API returning 200s?鈥� or 鈥淚s CPU usage high?鈥�, but when issues arise, you鈥檙e left sifting through logs and metrics. 

Observability, on the other hand, is about understanding your system deeply enough to answer new, unanticipated questions quickly. It provides insights before problems arise, rather than just tracking them after. That鈥檚 the journey we started at ClickPost.

The Challenge: High Cost and Low Control

We started with New Relic, but over time, it became clear it wasn鈥檛 right for us. The costs were high especially after estimating usage through the and we lacked control over critical aspects like trace retention and data residency. Additionally, its lack of flexibility made it difficult to customize trace ingestion. We needed a solution that was affordable, customizable, and scalable. Hence, we decided to build our own.

We needed answers quickly, not just more dashboards. So, we built our own solution.

The Stack: Open-Source Observability at Scale

To create a solution that met our needs, we turned to open-source tools. The foundation of our system is OpenTelemetry, combined with tools like Grafana for visualization and Prometheus for metrics. We deconstructed a $100k+ observability SaaS offering and rebuilt it in-house; for a fraction of the cost, and full control.

The How: Instrument Everything

Instrumentation was our first step. We used OpenTelemetry鈥檚 auto-instrumentation to track data from Java and Python services. For our highly-used Kafka service, we manually instrumented it to get an end-to-end trace of our transactions. This setup ensures we can track and visualize every critical aspect of our system.

We used OTLP GRPC exporters to forward trace data to our custom local OpenTelemetry Collector. The Collector was configured with a memory limiter and batch processor to control resource usage and avoid overwhelming future components like Prometheus and Tempo.

The Real Magic: Collector + Trace-to-Metrics

Once we started sending the traces to the Collector, it worked its magic by converting spans into Prometheus metrics. This gave us the ability to easily track metrics like request counts, latencies, and error rates without needing to write custom metric code.

At the same time, we used the default Language for simple transformations on the received OTel data in the collector. This enabled custom views without adding complexity. Now, we can create customized dashboards that fit our needs.

Going Deeper: Tailoring Observability to Our Needs

While OpenTelemetry provided a strong foundation, we needed more. Two pain points stood out:

1. Breaking Down Parent Traces 

Large parent traces often collapsed important details. We couldn鈥檛 easily segment them by consumer name, trace parent name, or other dimensions. To solve this, we built a custom Go-based processor plugin inside our own distribution of the custom OpenTelemetry Collector. This plugin injected extra metadata into spans, allowing us to slice traces more intelligently. Suddenly, instead of staring at a flat parent trace, we could break it down into meaningful segments and analyse them with query languages and beautiful dashboards.

2. Fixing Noisy Span Names

Out-of-the-box instrumentation often gives you generic span names like HTTP GET or SELECT. That works for basic tracing, but when you鈥檙e viewing them at a glance, it becomes noisy and hard to interpret. We wanted spans that were human-readable and directly tied to business logic. In Python, we forked , modified the span naming logic, and deployed our fork across services. In Java, we achieved the same using OpenTelemetry extensions, overriding default span names for HTTP operations. The result: our traces told a clear story instead of drowning us in noise.

These two changes 鈥� segmentation and human-readable span names 鈥� gave us clarity beyond most commercial APMs. Developers could debug faster, understand flows at a glance, and trust what the traces were saying.

Dashboards That Tell Stories

Metrics and traces are only as good as the insights you can derive from them. For us, Grafana wasn鈥檛 just a place to dump numbers 鈥� it became the canvas where system behavior turned into stories that engineers could act on. Here鈥檚 how we structured our dashboards:

Traces in Grafana

We started with the foundation: traces.
With Tempo鈥檚 out-of-the-box integration in Grafana, we could drill into a single request 鈥� from the incoming API call all the way to the database query.

Instead of chasing logs in the dark, traces lit up the system鈥檚 flow like a map.

APM Dashboards

On top of traces, we layered APM-style dashboards to get a high-level view of application health. These boards became our quick pulse check:

• Request rates, error distributions, latency percentiles

• Heatmaps catching tail latencies that averages usually hide

• Request distribution charts showing which routes were really driving system load

This way, we could quickly tell if a spike in latency was because of DB queries鈥� or just uneven traffic.

Segment Breakdown Dashboards

But sometimes, a parent trace was just too broad to be useful.
So, we built segment breakdown dashboards from our custom collector.

This gave us the power to slice traces by:

• API route

• Consumer

• Or other attributes we cared about

From there, we had span-level insights of a particular trace along with pie charts of load share and latency distributions (P5/P50/P95).

• Which DB operations were the real bottlenecks

• Which consumer was causing latency spikes right now

In short 鈥� it gave us the 鈥渨hy鈥� behind the trace.

JVM Metrics Dashboards: 

Finally, we complement traces with JVM-level dashboards for our Java/Kotlin services. These cover CPU utilization, heap memory trends, thread states, and garbage collection behavior. Instead of waiting for 鈥淥ut of Memory鈥� errors, we can proactively watch heap usage patterns. Thread breakdowns help us identify bottlenecks like blocked threads or runaway pools before they escalate.

Together, these dashboards create a multi-layered view 鈥� from a single slow query in a trace, to its impact on latency distributions, to JVM resource pressure on the host. They don鈥檛 just surface metrics; they let us connect dots, explain behavior, and make performance actionable.

Why We Invested

As Andrew Mallaband puts it in his excellent piece:

We saw this first-hand. Outages don鈥檛 just drain revenue 鈥� they pull engineers away from building new features, frustrate customers, and create cascading risks.

That鈥檚 why we invested the time to build observability right, in-house.

Cost vs Power: Why This Was Worth It

Yes 鈥� building our system took time and effort. But the benefits far outweighed the challenges. Beyond saving costs, we gained flexibility, transparency, and long-term control.

This isn鈥檛 just an observability platform. It鈥檚 an investment in resilience and cost predictability.

What鈥檚 Next

We鈥檙e not stopping here. As our system evolves, we鈥檒l continue to add new features. Here鈥檚 what鈥檚 on the roadmap:

• Frontend Telemetry: Introduce browser-side tracing with OpenTelemetry JS to capture page loads, API calls, and user interactions鈥攇iving us end-to-end visibility from click to backend.

• EDA Instrumentation: Extending instrumentation to other event-driven architectures flows like RabbitMQ.

• Logging Pipelines: Integrate Loki for advanced logging and correlation with traces and metrics.

• Custom Flows: Perfecting flows to comprehensively visualize entire transactional journeys across services.

• AI/ML-assisted Detection: Experiment with Grafana Machine Learning plugins for anomaly detection and forecasting in metrics.

We鈥檒l keep improving our observability system to stay ahead of our needs.

Why You Should Care

If you鈥檙e at a startup or scaling business, you know how critical observability is. But observability doesn鈥檛 have to break the bank; it can be tailored to your business needs

• Don鈥檛 overspend on external tools with limited control.

• Build a system that fits your specific needs.

• Own your data, own your insights.

Take it from us: observability is not optional.

And when you build it yourself, you don鈥檛 just see your system 鈥� you own your visibility.