Redis service

Table of Contents

• Before you start
• Step by step guide
• HTTP handler component
• Initializing the database
• KeyValue::HandleRequestThrow
• KeyValue::GetValue
• KeyValue::PostValue
• KeyValue::DeleteValue
• Static config
• Dynamic config
• int main()
• Build and Run
• Functional testing
• Full sources

Before you start

Make sure that you can compile and run core tests and read a basic example Writing your first HTTP server.

Step by step guide

Microservices that have state often work with database to store their data and replicate that state across instances of the microservice. In this tutorial we will write a service that is a simple key-value storage on top of Redis database. The service would have the following Rest API:

• HTTP POST by path /v1/key-value with query parameters key and value stores the provided key and value or 409 Conflict if such key already exists
• HTTP GET by path /v1/key-value with query parameter key returns the value if it exists or 404 Not Found if it is missing
• HTTP DELETE by path /v1/key-value with query parameter key deletes the key if it exists and returns number of deleted keys (cannot be more than 1, since keys are unique in Redis database)

HTTP handler component

Like in Writing your first HTTP server we create a component for handling HTTP requests:

#include <userver/clients/dns/component.hpp>
#include <userver/components/component.hpp>
#include <userver/components/minimal_server_component_list.hpp>
#include <userver/server/handlers/http_handler_base.hpp>
#include <userver/storages/redis/client.hpp>
#include <userver/storages/redis/component.hpp>
#include <userver/storages/secdist/component.hpp>
#include <userver/storages/secdist/provider_component.hpp>
#include <userver/utils/daemon_run.hpp>
 
namespace samples::redis {
class EvalSha final : public server::handlers::HttpHandlerBase {
 public:
  static constexpr std::string_view kName = "handler-script";
 
  EvalSha(const components::ComponentConfig& config,
          const components::ComponentContext& context);
 
  std::string HandleRequestThrow(
      const server::http::HttpRequest& request,
      server::request::RequestContext&) const override;
 
 private:
  std::string EvalShaRequest(const server::http::HttpRequest& request) const;
  std::string ScriptLoad(const server::http::HttpRequest& request) const;
 
  storages::redis::ClientPtr redis_client_;
  storages::redis::CommandControl redis_cc_;
};
 
class KeyValue final : public server::handlers::HttpHandlerBase {
 public:
  static constexpr std::string_view kName = "handler-key-value";
 
  KeyValue(const components::ComponentConfig& config,
           const components::ComponentContext& context);
 
  std::string HandleRequestThrow(
      const server::http::HttpRequest& request,
      server::request::RequestContext&) const override;
 
 private:
  std::string GetValue(std::string_view key,
                       const server::http::HttpRequest& request) const;
  std::string PostValue(std::string_view key,
                        const server::http::HttpRequest& request) const;
  std::string DeleteValue(std::string_view key) const;
 
  storages::redis::ClientPtr redis_client_;
  storages::redis::CommandControl redis_cc_;
};
 
}  // namespace samples::redis

Note that the component holds a storages::redis::ClientPtr - a client to the Redis database. That client is thread safe, you can use it concurrently from different threads and tasks.

Initializing the database

To access the database from our new component we need to find the Redis component and request a client to a specific cluster by its name. After that we are ready to make requests.

KeyValue::KeyValue(const components::ComponentConfig& config,
                   const components::ComponentContext& context)
    : server::handlers::HttpHandlerBase(config, context),
      redis_client_{
          context.FindComponent<components::Redis>("key-value-database")
              .GetClient("taxi-tmp")},
      redis_cc_{std::chrono::seconds{15}, std::chrono::seconds{60}, 4} {}

KeyValue::HandleRequestThrow

In this sample we use a single handler to deal with all the HTTP methods. The KeyValue::HandleRequestThrow member function mostly dispatches the request to one of the member functions that actually implement the key-value storage logic:

std::string KeyValue::HandleRequestThrow(
    const server::http::HttpRequest& request,
    server::request::RequestContext& /*context*/) const {
  const auto& key = request.GetArg("key");
  if (key.empty()) {
    throw server::handlers::ClientError(
        server::handlers::ExternalBody{"No 'key' query argument"});
  }
 
  switch (request.GetMethod()) {
    case server::http::HttpMethod::kGet:
      return GetValue(key, request);
    case server::http::HttpMethod::kPost:
      return PostValue(key, request);
    case server::http::HttpMethod::kDelete:
      return DeleteValue(key);
    default:
      throw server::handlers::ClientError(server::handlers::ExternalBody{
          fmt::format("Unsupported method {}", request.GetMethod())});
  }
}

Warning

Handle* functions are invoked concurrently on the same instance of the handler class. In this sample the KeyValue component only uses the thread safe DB client. In more complex cases synchronization primitives should be used or data must not be mutated.

KeyValue::GetValue

Executing a query to the Redis database is as simple as calling the corresponding method of storages::redis::ClientPtr.

Note that some methods return an optional result, which must be checked. Here it can indicate a missing key value.

std::string KeyValue::GetValue(std::string_view key,
                               const server::http::HttpRequest& request) const {
  const auto result = redis_client_->Get(std::string{key}, redis_cc_).Get();
  if (!result) {
    request.SetResponseStatus(server::http::HttpStatus::kNotFound);
    return {};
  }
  return *result;
}

KeyValue::PostValue

Here we use storages::redis::Client::SetIfNotExist() to ensure not to change already existing keys.

std::string KeyValue::PostValue(
    std::string_view key, const server::http::HttpRequest& request) const {
  const auto& value = request.GetArg("value");
  const auto result =
      redis_client_->SetIfNotExist(std::string{key}, value, redis_cc_).Get();
  if (!result) {
    request.SetResponseStatus(server::http::HttpStatus::kConflict);
    return {};
  }
 
  request.SetResponseStatus(server::http::HttpStatus::kCreated);
  return std::string{value};
}

KeyValue::DeleteValue

Note that mutating queries are automatically executed on a master instance.

std::string KeyValue::DeleteValue(std::string_view key) const {
  const auto result = redis_client_->Del(std::string{key}, redis_cc_).Get();
  return std::to_string(result);
}

Static config

Static configuration of service is quite close to the configuration from Writing your first HTTP server except for the handler and DB:

# yaml
components_manager:
    components:                       # Configuring components that were registered via component_list
        handler-key-value:
            path: /v1/key-value                  # Registering handler by URL '/v1/key-value'.
            method: GET,POST,DELETE              # GET, POST and DELETE requests only.
            task_processor: main-task-processor  # Run it on CPU bound task processor
        handler-script:
            path: /v1/script                     # Registering handler by URL '/v1/key-value'.
            method: POST                         # GET, POST and DELETE requests only.
            task_processor: main-task-processor  # Run it on CPU bound task processor
 
        key-value-database:
            groups:
              - config_name: taxi-tmp  # Key to lookup in secdist configuration
                db: taxi-tmp           # Name to refer to the cluster in components::Redis::GetClient()
 
            subscribe_groups:  # Array of redis clusters to work with in subscribe mode
 
            thread_pools:
                redis_thread_pool_size: 8
                sentinel_thread_pool_size: 1
 
        secdist: {}                                      # Component that stores configuration of hosts and passwords
        default-secdist-provider:                        # Component that loads configuration of hosts and passwords
            config: /etc/redis_service/secure_data.json  # Values are supposed to be stored in this file
            missing-ok: true                             # ... but if the file is missing it is still ok
            environment-secrets-key: SECDIST_CONFIG      # ... values will be loaded from this environment value
 
        testsuite-support:
 
        server:
            # ...

Dynamic config

We are not planning to get new dynamic config values in this sample. Because of that we just write the defaults to the fallback file of the components::DynamicConfigFallbacks component: samples/redis_service/dynamic_config_fallback.json

All the values are described in a separate section Dynamic configs .

A production ready service would dynamically retrieve the above options at runtime from a configuration service. See Writing your own configs server for insights on how to change the above options on the fly, without restarting the service.

int main()

Finally, after writing down the dynamic config values into file at dynamic-config-fallbacks.fallback-path, we add our component to the components::MinimalServerComponentList(), and start the server with static config kStaticConfig.

int main(int argc, char* argv[]) {
  const auto component_list =
      components::MinimalServerComponentList()
          .Append<samples::redis::KeyValue>()
          .Append<samples::redis::EvalSha>()
          .Append<components::Secdist>()
          .Append<components::DefaultSecdistProvider>()
          .Append<components::Redis>("key-value-database")
          .Append<components::TestsuiteSupport>()
          .Append<clients::dns::Component>();
  return utils::DaemonMain(argc, argv, component_list);
}

Build and Run

To build the sample, execute the following build steps at the userver root directory:

mkdir build_release
cd build_release
cmake -DCMAKE_BUILD_TYPE=Release ..
make userver-samples-redis_service

The sample could be started by running make start-userver-samples-redis_service. The command would invoke testsuite start target that sets proper paths in the configuration files, prepares and starts the DB, and starts the service.

To start the service manually start the DB server and run ./samples/redis_service/userver-samples-redis_service -c </path/to/static_config.yaml> (do not forget to prepare the configuration files!).

Now you can send a request to your service from another terminal:

bash
$ curl -X POST 'localhost:8088/v1/key-value?key=hello&value=world' -i
HTTP/1.1 201 Created
Date: Wed, 27 Oct 2021 16:45:13 UTC
Content-Type: text/html
X-YaSpanId: 015fb0becd2926ef
X-YaRequestId: 7830671d7dd2462ba9043db532c2b82a
Server: userver/1.0.0 (20211027123413; rv:c1879aa03)
X-YaTraceId: d7422d7bcdc9493997fc687f8be24883
Connection: keep-alive
Content-Length: 5
 
world
$ curl -X DELETE 'localhost:8088/v1/key-value?key=hello&value=world' -i
HTTP/1.1 200 OK
Date: Wed, 27 Oct 2021 16:46:35 UTC
Content-Type: text/html
X-YaSpanId: e83698e2ef8cc729
X-YaRequestId: ffbaacae38e64bb588affa10b928b759
Server: userver/1.0.0 (20211027123413; rv:c1879aa03)
X-YaTraceId: cd3e6acc299742739bb22c795b6ef3a7
Connection: keep-alive
Content-Length: 1
 
1

Functional testing

Functional tests for the service could be implemented using the testsuite. To do that you have to:

• Prepare the pytest by importing the pytest_userver.plugins.redis plugin:

  import json
   
  import pytest
   
  pytest_plugins = ['pytest_userver.plugins.redis']

• Add the Redis settings info to the service environment variable:

  @pytest.fixture(scope='session')
  def service_env(redis_sentinels):
      secdist_config = {
          'redis_settings': {
              'taxi-tmp': {
                  'password': '',
                  'sentinels': redis_sentinels,
                  'shards': [{'name': 'test_master0'}],
              },
          },
      }
   
      return {'SECDIST_CONFIG': json.dumps(secdist_config)}

  The auto_client_deps fixture already knows about the redis_store fixture, so there's no need to override the extra_client_deps fixture.
• Write the test:

  async def test_redis(service_client):
      response = await service_client.delete('/v1/key-value?key=hello')
      assert response.status == 200
   
      response = await service_client.post('/v1/key-value?key=hello&value=world')
      assert response.status == 201
      assert response.content == b'world'
   
      response = await service_client.request('GET', '/v1/key-value?key=hello')
      assert response.status == 200
      assert response.content == b'world'

Full sources

See the full example:

• samples/redis_service/redis_service.cpp
• samples/redis_service/static_config.yaml
• samples/redis_service/dynamic_config_fallback.json
• samples/redis_service/CMakeLists.txt

---

⇦ MongoDB service | Custom Authorization/Authentication via PostgreSQL ⇨