Modern applications use logging a lot for debugging and diagnosing a running production service. Usually logs are harvested, indexed and stored in a separate service for further investigation.
The userver framework addresses modern logging requirements and provides multiple facilities for efficient work with logs, including Logging at runtime.
Below are the introductions to main developer logging facilities.
For information on controlling log level in unit tests see Unit Tests and Benchmarks
Macros are used for logging:
The right part of the expression is calculated only if the logging level is less or equal to the macro log level. I.e., the right part of the expression LOG_DEBUG() << ... is calculated only in the case of the DEBUG or TRACE logging level.
Sometimes it is useful to set the logging level of a given log entry dynamically:
Not all logs get into the log file, but only those that are not lower than the logger's log level. The logger log level is set in static config (see components::Logging).
The log level can be changed in the static config for a particular handle. In this case, the log level of the logger is changed only for the request handling task of the handle and for all the subtasks:
Logging per line and file can be overriden at runtime using USERVER_LOG_DYNAMIC_DEBUG dynamic config.
The logging level that was set in the static config of the components::Logging component for the entire service can be changed on the fly.
See Logging at runtime for more info.
For per-handle limiting of the request body or response data logging you can use the request_body_size_log_limit and request_headers_size_log_limit and response_data_size_log_limit static options of the handler (see server::handlers::HandlerBase). Or you could override the server::handlers::HttpHandlerBase::GetRequestBodyForLogging and server::handlers::HttpHandlerBase::GetResponseDataForLogging functions.
If some line of code generates too many logs and a small number of them is enough, then LOG_* should be replaced with LOG_LIMITED_*. For example, LOG(lvl) should be replaced with LOG_LIMITED(lvl); LOG_DEBUG() should be replaced with LOG_LIMITED_DEBUG(), etc.
In this case, the log message is written only if the message index is a power of two. The counter is reset every second.
Typical recommended places to use limited logging:
resource unavailability
Examples: insufficient memory; another service is not responding.
Explanation: in case of a problem such errors are usually reported in thousands per second; their messages do not differ.
incorrect use of a component or service
Examples: negative numbers are passed to your library in a method that accepts only positive ones; the requested functionality is not supported by a third-party microservice.
Explanation: Such errors should be caught at the testing stage. However, if a miracle suddenly happened and someone did not test the service, then the same errors are be reported in thousands per second
Nuances:
template function logs via LOG_LIMITED_X, then each specialization of the function template has a separate counterLOG_LIMITED_X is called in different places, then all its calls use the same counterIf you want to add tags to as single log record, then you can create an object of type logging::LogExtra, add the necessary tags to it and output the LogExtra object to the log:
If the same fields must be added to each log in a code block, it is recommended to use tracing::Span, which implicitly adds tags to the log.
Sometimes it is useful to write a full stacktrace to the log. Typical use case is for logging a "never should happen happened" situation. Use logging::LogExtra::Stacktrace() for such cases:
Important: getting a text representation of a stacktrace is an expensive operation. In addition, the stacktrace itself increases the log record size several times. Therefore, you do not need to use a stack trace for all errors. Use it only when it is 100% useful for diagnostics and other diagnostic tools are ineffective.
You can create any number of additional loggers. All you need is to add them to your logging.loggers section of the static config (see components::Logging for an example)
After that, you can get a logger in the code like this:
To write to such a logger, it is convenient to use LOG_XXX_TO, for example:
Note: do not forget to configure the logrotate for your new log file!
The userver implements a request tracing mechanism that is compatible with the opentelemetry standard.
It allows you to save dependencies between tasks, between requests through several services, thereby building a trace of requests and interactions. It can be used to identify slow query stages, bottlenecks, sequential queries, etc.
See tracing::DefaultTracingManagerLocator for more info.
When processing a request, you can create a tracing::Span object that measures the execution time of the current code block (technically, the time between its constructor and destructor) and stores the resulting time in the log:
Example log for tracing::Span span{"cache_invalidate"}:
Log record example for some POST /v1/upload handle:
tracing::Span can only be created on stack. Currently, the ability to create tracing::Span as a member of a class whose objects can be passed between tasks is not supported.
In addition to trace_id, span_id, parent_id and other tags specific to opentracing, the tracing::Span class can store arbitrary custom tags. To do this, Span implicitly contains LogExtra. You can add tags like this:
Unlike simple LogExtra, tags from Span are automatically logged when using LOG_XXX(). If you create a Span, and you already have a Span, then LogExtra is copied from the old one to the new one (except for the tags added via AddNonInheritableTag).
TraceId propagates both to sub-spans within a single service, and from client to serverLink propagates within a single service, but not from client to server. A new link is generated for the "root" request handling taskSpanId identifies a specific span. It does not propagateParentSpanId, which is the inner-most SpanId of the clientParentLink, which is the Link of the clientAll logs in the current task are implicitly linked to the current Span for the user, and tags of this Span are added to them (trace_id, span_id, etc.). All Span in the current task are implicitly stacked, and if there are several similar Span, the last created One (i.e., located at the top of the Span stack of the current task) will be used for logging.
If you want to get the current Span (for example, you want to write something to LogExtra, but do not want to create an additional Span), then you can use the following approach:
A Span is automatically created for each request, handled by the server::handlers::HttpHandlerBase inherited handles. trace_id, parent_id and link are automatically populated for the request headers (if any).
Cache handlers do not have a parent Span, so a Span without a parent is automatically created for them, with a new trace_id for each update. The situation is similar for all utils::PeriodicTask.
When creating a new task via utils::Async, a new Span is created and linked with the current Span of the current task.
DB drivers and the components::HttpClient automatically create a Span for each request to trace them.
The HTTP client sends the current link/span_id/trace_id values in each request to the server, they do not need to be specified.
When the HTTP server handles the request, it extracts data from the request headers and puts them in the Span.
Names of the headers varry depending on tracing::DefaultTracingManagerLocator static configuration and on the chosen tracing::Format value. For example, with tracing::Format::kYandexTaxi the following headers would be used:
Using the server dynamic config USERVER_NO_LOG_SPANS, you can set names and prefixes of Span names that do not need to be logged. If the span is not logged, then the tracing::ScopeTime of this span and any custom tags attached to the span via the methods of the Add*Tag*() are not put into the logs.
For example, this is how you can disable logging of all Span for MongoDB (that is, all Span with stopwatch_name starting with mongo) and Span with stopwatch_name=test:
It is possible to use OpenTelemetry protocol for logs and tracing exporting. To use it, register all prerequisites for gRPC client and register otlp::LoggerComponent in component list in your main function. The static config file should contain a component section with OTLP logger.
Also remove default logger from logging.loggers as the default logger is handled by otlp component from now on.
The related part of static config:
endpoint contains OTLP collector host and port, service-name contains your service name, and log-level contains service log level.
After the service start you'll see the following in stderr:
It means the logger is successfully initialized and is ready to process the logs.
If somethings goes wrong (e.g. OTLP collector agent is not available), you'll see errors in stderr. The service buffers not-yet-sent logs and traces in memory, but drops them on overflow.
In certain environments, such as Kubernetes, applications typically write logs to stdout/stderr, while traces are sent efficiently through the 'push' model (via OTLP transport). Kubernetes stores the container's stdout/stderr in files on nodes, making logs available for log collectors using the pull model. This approach ensures that logs remain accessible even if the application fails, capturing the most critical information.
To configure separate sinks for logs and traces, use the optional sinks block in the otlp-logger configuration:
If the 'sinks' block is not present in the 'otlp-logger' configuration, both logs and traces use the OTLP transport and are delivered to an OpenTelemetry-compatible collector by a background task in userver.
In the sinks block, you can set a value for each stream. See: otlp::LoggerComponent.
If you choose otlp for both streams, ensure that logging.loggers is empty:
Otherwise, add the default logger in the logging component's loggers field: