Observability - Keeping Business-Critical Applications Healthy

In todays distributed systems and especially in microservice architectures, simple logging is not sufficient anymore. The process has to be traceable through various services or methods, since it is often the interaction between microservices that results in problems or performance bottlenecks. This requires that, aside from the end-user calls, additional information on service calls is provided and stored in special tracing log events as well. Moreover, these tracing logs have also to be stored centrally for all services involved in order to be able to display the call hierarchy. When using external libraries or services, they too will have to meet those additional requirements.

Metrics are numerical representations of statuses (e. g. number of open connections) or throughputs (e. g. writing volume on a hard drive since a specific point in time, calling of a specific functionality). This way they differ from logs and tracing data relating to individual events.

Metrics can be queried of standard applications (e. g. NGINX, DBs or objects in Kubernetes) via so-called exporters or metric endpoints. Customer-specific applications should be instrumented in a way that allows to measure SLAs as well as to obtain further information on the usage (e. g. number and response time of critical calls) for detailed performance observations.

The currently widely used metrics format was introduced by Prometheus and standardized via OpenMetrics. Metric points here are composed as follows:

• Metrics name: describes what is represented, e. g. server_open_connection_count
• Labels: on the basis of labels, differently measured instances can be distinguished, e. g. instance=127.0.0.1:8080.
• Timestamp: at what time was this value valid?
• Value: numeric value

This way, the performance and where appropriate also the number of errors or specific statuses can be compactly represented and graphically visualized in Grafana.

Based on these numeric values, rules can be defined that provide a message if the system has exceeded limit values – e. g. if more than 90 % of the available connections were occupied for more than 10 minutes or if during 5 minutes an average of more than 2 % of the queries resulted in errors. A monitoring tool for metrics like Prometheus stores the metrics, checks these conditions, and can subsequently notify the ones in charge.

A relatively new discipline within observability, error management focuses on the structured tracking of error messages occurring in the monitored software. Detected errors, such as those found in logs, are recorded and correlated with similar errors (so that a recurring error appearing 10,000 times generates only a single entry). SREs and software engineers are automatically alerted to new errors, enabling them to investigate and resolve issues more efficiently—far more effective than manually sifting through massive application logs!

Amid the myriad individual values in application metrics, crucial indicators of potential issues can be hidden—but how do you find that needle in the haystack? This is where anomaly detection comes into play, a fascinating use case for Artificial Intelligence (AI). In essence, an anomaly detection system automatically learns what observability data looks like when a system is functioning "normally."

Based on this understanding, the system can then automatically detect deviations from this normal operation (e.g., dramatically increased traffic, filling buffers, etc.) and issue warnings if necessary, even though it was never explicitly programmed to monitor specific metrics. This approach is a highly valuable complement to manually configured alerts, helping to identify "blind spots" in alert configurations and detect unforeseen problems in general.