SRE: Site Reliability Engineering

Ending points for liveness and readiness probes are often implemented very simply: They respond with 200 OK as soon as the application is started. In a number of projects, we have experienced that this may not be enough. Therefore, we have started to test the accessibility of all adjacent systems and message queues with the support of health checks. This enables us to detect problems during deployment and, ideally, resolve them automatically.

During one project, a problem with EJB Timer Services caused the transaction to roll back after each run. If one of the next runs would prove successful, this process would be unproblematic in itself. To find out if we deal with expected rollbacks or with real bugs, we implemented a metric measuring time passed since the last successful run. This allowed us to distinguish between temporary and permanent errors.

With Java applications, it is worth drawing regularly thread dumps regularly. Thread dumps help with post-mortem analysis and profiling. The development of thread dumps e.g. quickly brings to light when an external system is blocked and new threads with blocked calls are constantly being started. In particular, it is recommended to draw two to three thread dumps in the stop script in order to analyze the status after a restart of the application.

With Mapped Diagnostic Context (MDC), logging frameworks offer the possibility to log information like user names by default, allowing to trace which log lines belong together during log analysis. However, MDC data are not always available if, for example, the user has not yet been determined. Therefore, it is worthwhile to additionally include the thread in the log format. The thread provides a secure and easy way to track which log lines belong to the same request.

During a project in the telecommunications industry, we were facing the challenge of many microservices calling the same endpoint, yet the total number of calls could not exceed a certain threshold value per second. We solved this by using Zookeeper to coordinate the calls. The advantage: We were able to avoid a central coordination system as a single point of failure.

Manual steps during build and deployment are a common source of errors. It's worth to automate everything. Modern CI/CD pipelines do not only reduce the risk of errors, but also relieve the SREs of annoying recurring tasks.

The biggest challenge in load testing is to generate realistic test data. This does not only include the content of the data. In a large migration project, we found out, that the way data are fragmented in a database can have a significant impact on performance. The fact that we already determined this in the load test phase was decisive for the project’s success.

The more measuring points an application has, the better. This does not only help in operation. Load testing, for example, is much more valuable when it shows not only whether a service is adhering to SLOs, but also where potential bottlenecks are.

Software engineers like to use modern design patterns like circuit breakers. To avoid overload, however, you should not lose sight of classic configuration options. Pool sizes in Java application servers, for example, should be designed in a way that, in the event of an unexpected peak load, the stop occurs as soon as possible so that downstream components are not getting overloaded.

Site Reliability Engineers must familiarize themselves with the normal behavior of their services and regularly check their logs. Otherwise, in case of an error, a lot of time is lost in investigating oddities that have nothing to do with the acute fault.