🐙 userver has built-in support for functional service tests using Yandex.Taxi Testsuite. Testsuite is based on pytest and allows developers to test their services in isolated environment. It starts service binary with minimal database and all external services mocked then allows developer to call service handlers and test their result.
Supported features:
utils::datetime::Now()userver_testsuite_add()With userver_testsuite_add() function you can easily add testsuite support to your project. Its main purpose is:
venv environment or use an existing one.PYTHONPATH and passes extra arguments to pytest.ctest target.start-* target that starts the service and databases with testsuite configs and waits for keyboard interruption to stop the service.cmake/UserverTestsuite.cmake library is automatically added to CMake path after userver environment setup. Add the following line to use it:
Then create testsuite target:
testsuite- and start- CMake target names.pytest.PYTHONPATH.venv.OFF to disable pretty printing.Some of the most useful arguments for PYTEST_ARGS:
| Argument | Description |
|---|---|
-v | Increase verbosity |
-s | Do not intercept stdout and stderr |
-x | Stop on first error |
-k EXPRESSION | Filter tests by expression |
--service-logs-pretty | Enable logs coloring |
--service-logs-pretty-disable | Disable logs coloring |
--service-log-level=LEVEL | Set the log level for the service. Possible values: trace, debug, info, warning, error, critical |
--service-wait | With this argument the testsuite will wait for the service start by user. For example under gdb. Testsuite outputs a hint on starting the service |
-rf | Show a summary of failed tests |
userver_testsuite_add_simple()userver_testsuite_add_simple() is a version of userver_testsuite_add() that makes some assumptions of the project structure. It should be invoked from the service's CMakeLists.txt as follows:
It supports the following file structure (and a few others):
configs/config.yamlconfigs/config_vars.[testsuite|tests].yaml [optional]configs/dynamic_config_fallback.json [optional]configs/[secdist|secure_data].json [optional][testsuite|tests]/conftest.pyYou may want to create new virtual environment with its own set of packages. Or reuse existing one. That could be done this way:
PYTHON_BINARY is specified then it is used.REQUIREMENTS, if any, are installed. Requirements of userver itself (based on selected USERVER_FEATURE_* flags) are installed as well.Basic requirements.txt file may look like this:
Creating per-testsuite virtual environment is a recommended way to go. It creates Python venv in the current binary directory:
${CMAKE_CURRENT_BINARY_DIR}/venv-testsuite-${SERVICE_TARGET}
userver_testsuite_add() registers a ctest target with name testsuite-${SERVICE_TARGET}. Run all project tests with ctest command or use filters to run specific tests:
To run pytest directly userver_testsuite_add() creates a testsuite runner script that could be found in corresponding binary directory. This may be useful to run a single testcase, to start the testsuite with gdb or to start the testsuite with extra pytest arguments:
${CMAKE_CURRENT_BINARY_DIR}/runtests-testsuite-${SERVICE_TARGET}
You can use it to manually start testsuite with extra pytest arguments, e.g.:
Please refer to testuite and pytest documentation for available options. Run it with --help argument to see the short options description.
To debug the functional test you can start testsuite with extra pytest arguments, e.g.:
At the beginning of the execution the console will display the command to start the service, e.g.:
Now you can open a new terminal window and run this command in it or if you use an IDE you can find the corresponding CMake target and add arg --config /.../config.yaml. After that it will be possible to set breakpoints and start target with debug.
By default pytest_userver plugin is included in python path. It provides basic testsuite support for userver service. To use it add it to your pytest_plugins in root conftest.py:
It requires extra PYTEST_ARGS to be passed:
The plugins match the userver cmake targets. For example, if the service links with userver-core its tests should use the pytest_userver.plugins.core plugin.
| CMake target | Matching plugin for testsuite |
|---|---|
| userver-core | pytest_userver.plugins.core |
| userver-grpc | pytest_userver.plugins.grpc |
| userver-postgresql | pytest_userver.plugins.postgresql |
| userver-clickhouse | pytest_userver.plugins.clickhouse |
| userver-redis | pytest_userver.plugins.redis |
| userver-mongo | pytest_userver.plugins.mongo |
| userver-rabbitmq | pytest_userver.plugins.rabbitmq |
| userver-mysql | pytest_userver.plugins.mysql |
Userver has built-in support for testsuite.
In order to use it you need to register corresponding components:
Headers:
Add components to components list:
Add testsuite components to config.yaml:
testsuite-enabled must be disabled in production environment. Testsuite sets the testsuite-enabled variable into true when it runs the service. In the example above this variable controls whether tests-control component is loaded. This component must be disabled for production.The essential parts of the testsuite are pytest_userver.plugins.service_client.service_client and pytest_userver.plugins.service_client.monitor_client fixtures that give you access to the pytest_userver.client.Client and pytest_userver.client.ClientMonitor respectively. Those types allow to interact with a running service.
Testsuite functions reference could be found at Testsuite Python support.
pytest_userver modifies static configs config.yaml and config_vars.yaml passed to pytest before starting the userver based service.
To apply additional modifications to the static config files declare USERVER_CONFIG_HOOKS variable in your pytest-plugin with a list of functions or pytest-fixtures that should be run before config is written to disk. USERVER_CONFIG_HOOKS values are collected across different files and all the collected functions and fixtures are applied.
Example usage:
Fixture service_client is used to access the service being tested:
When tests-control component is enabled service_client is pytest_userver.client.Client instance. Which supports special testsuite related methods.
On first call to service_client service state is implicitly updated, e.g.: caches, mocked time, etc.
Use service_env fixture to provide extra environment variables for your service:
Use extra_client_deps fixture to provide extra fixtures that your service depends on:
Note that auto_client_deps fixture already knows about the userver supported databases and clients, so usually you do not need to manually register any dependencies.
Mockserver allows to mock external HTTP handlers. It starts its own HTTP server that receives HTTP traffic from the service being tested. And allows to install custom HTTP handlers within testsuite. In order to use it all HTTP clients must be pointed to mockserver address.
Mockserver usage example:
To connect your HTTP client to the mockserver make the HTTP client use a base URL of the form http://{mockserver}/{service_name}/.
This could be achieved by patching static config as described in config hooks and providing a mockserver address using mockserver_info.url(path):
Userver provides a way to mock internal datetime value. It only works for datetime retrieved with utils::datetime::Now(), see Mocked time section for details.
From testsuite you can control it with mocked_time plugin.
Example usage:
Example are available here:
Testpoints are used to send messages from the service to testcase and back. Typical use cases are:
First of all you should include testpoint header:
It provides TESTPOINT() and family of TESTPOINT_CALLBACK() macros that do nothing in production environment and only work when run under testsuite. Under testsuite they only make sense when corresponding testsuite handler is installed.
All testpoints has their own name that is used to call named testsuite handler. Second argument is formats::json::Value() instance that is only evaluated under testsuite.
TESTPOINT() usage sample:
Then you can use testpoint from testcase:
In order to eliminate unnecessary testpoint requests userver keeps track of testpoints that have testsuite handlers installed. Usually testpoint handlers are declared before first call to service_client which implicitly updates userver's list of testpoint. Sometimes it might be required to manually update server state. This can be achieved using service_client.update_server_state() method e.g.:
Accessing testpoint userver is not aware of will raise an exception:
Testsuite can be used to test logs written by service. To achieve this the testsuite starts a simple logs capture TCP server and tells the service to replicate logs to it on per test basis.
Example usage:
Example on logs capture usage could be found here:
Testsuite tasks facility allows to register a custom function and call it by name from testsuite. It's useful for testing components that perform periodic job not related to its own HTTP handler.
You can use testsuite::TestsuiteTasks to register your own task:
After that you can call your task from testsuite code:
Or spawn the task asynchronously using context manager:
An example on testsuite tasks could be found here:
Testsuite provides access to userver metrics written by utils::statistics::Writer and utils::statistics::MetricTag via monitor_client , see tutorial on configuration. It allows to:
None if no such metric: await monitor_client.single_metric_optional(path, labels)Example usage:
For a metric tag that is defined as:
and used like:
the metrics could be retrieved and reset as follows:
For metrics with labels, they could be retrieved in the following way:
The Metric python type is hashable and comparable:
Different monitoring systems and time series databases have different limitations. To make sure that the metrics of your service could be used on most of the popular systems, there is special action in server::handlers::TestsControl.
To use it you could just write the following test:
Note that warnings are grouped by type, so you could check only for some particular warnings or skip some of them. For example:
Testsuite provides a way to start standalone service with all mocks and database started. This can be done by adding --service-runner-mode flag to pytest, e.g.:
Please note that -s flag is required to disable console output capture.
pytest_userver provides default service runner testcase. In order to override it you have to add your own testcase with @pytest.mark.servicetest: