HTTP Flatbuf handler and requests

Before you start

Make sure that you can compile and run core tests and read a basic example Writing your first HTTP server.

Step by step guide

JSON is a nice format, but it does not suit well for high-load applications.

This tutorial shows you how to send and receive Flatbuffers over HTTP using userver.

In this sample we use the samples/flatbuf_service/flatbuffer_schema.fbs Flatbuffers scheme and compile it via the ‘flatc --cpp --gen-object-api --filename-suffix ’.fbs' flatbuffer_schema.fbs` command.

HTTP Flatbuffer handler component

There are two ways to write a handler that deals with Flatbuffers:

• We could do that by creating a component derived from server::handlers::HttpHandlerBase as in the Writing your first HTTP server example. In that case we would need
  • to manually parse the input
  • and manually to serialize the result to a std::string that holds the bytes of a Flatbuffers.
• Or we could just take a server::handlers::HttpHandlerFlatbufBase that does all the above steps for us.

We are going to take the second approach. All the Flatbuffers related action happens in the HandleRequestFlatbufThrow method:

#include <userver/server/handlers/http_handler_flatbuf_base.hpp>
#include "flatbuffer_schema.fbs.h"
 
namespace samples::fbs_handle {
 
class FbsSumEcho final : public server::handlers::HttpHandlerFlatbufBase<fbs::SampleRequest, fbs::SampleResponse> {
public:
    // `kName` is used as the component name in static config
    static constexpr std::string_view kName = "handler-fbs-sample";
 
    // Component is valid after construction and is able to accept requests
    FbsSumEcho(const components::ComponentConfig& config, const components::ComponentContext& context)
        : HttpHandlerFlatbufBase(config, context) {}
 
    fbs::SampleResponse::NativeTableType
    HandleRequestFlatbufThrow(const server::http::HttpRequest& request, const fbs::SampleRequest::NativeTableType& fbs_request, server::request::RequestContext&)
        const override {
        request.GetHttpResponse().SetContentType(http::content_type::kApplicationOctetStream);
        fbs::SampleResponse::NativeTableType res;
        res.sum = fbs_request.arg1 + fbs_request.arg2;
        res.echo = fbs_request.data;
        return res;
    }
};
 
}  // namespace samples::fbs_handle

HTTP Flatbuffer request

A clients::http::Client is needed to make HTTP requests. It could be obtained from the components::HttpClient component.

class FbsRequest final : public components::ComponentBase {
public:
    static constexpr std::string_view kName = "fbs-request";
 
    FbsRequest(const components::ComponentConfig& config, const components::ComponentContext& context)
        : ComponentBase(config, context),
          http_client_{context.FindComponent<components::HttpClient>().GetHttpClient()},
          task_{utils::Async("requests", [this]() { KeepRequesting(); })} {}
 
    void KeepRequesting() const;
 
private:
    clients::http::Client& http_client_;
    engine::TaskWithResult<void> task_;
};

After that, we just send the data and validate the response:

void FbsRequest::KeepRequesting() const {
    // Fill the payload data
    fbs::SampleRequest::NativeTableType payload;
    payload.arg1 = 20;
    payload.arg2 = 22;
    payload.data = "Hello word";
 
    // Serialize the payload into a std::string
    flatbuffers::FlatBufferBuilder fbb;
    auto ret_fbb = fbs::SampleRequest::Pack(fbb, &payload);
    fbb.Finish(ret_fbb);
    std::string data(reinterpret_cast<const char*>(fbb.GetBufferPointer()), fbb.GetSize());
 
    // Send it
    const auto response = http_client_.CreateRequest()
                              .post("http://localhost:8084/fbs", std::move(data))
                              .timeout(std::chrono::seconds(1))
                              .retry(10)
                              .perform();
 
    // Response code should be 200 (use proper error handling in real code!)
    UASSERT_MSG(response->IsOk(), "Sample should work well in tests");
 
    // Verify and deserialize response
    const auto body = response->body_view();
    const auto* response_fb = flatbuffers::GetRoot<fbs::SampleResponse>(body.data());
    flatbuffers::Verifier verifier(reinterpret_cast<const uint8_t*>(body.data()), body.size());
    UASSERT_MSG(response_fb->Verify(verifier), "Broken flatbuf in sample");
    fbs::SampleResponse::NativeTableType result;
    response_fb->UnPackTo(&result);
 
    // Make sure that the response is the expected one for sample
    UASSERT_MSG(result.sum == 42, "Sample should work well in tests");
    UASSERT_MSG(result.echo == payload.data, "Sample should work well in tests");
}

Build and Run

To build the sample, execute the following build steps at the userver root directory:

bash
mkdir build_release
cd build_release
cmake -DCMAKE_BUILD_TYPE=Release ..
make userver-samples-flatbuf_service

The sample could be started by running make start-userver-samples-flatbuf_service. The command would invoke testsuite start target that sets proper paths in the configuration files and starts the service.

To start the service manually run ./samples/flatbuf_service/userver-samples-flatbuf_service -c </path/to/static_config.yaml>.

Now you can send a request to your server from another terminal:

bash
$ echo "100000000c00180000000800100004000c00000014000000140000000000000016000000000000000a00000048656c6c6f20776f72640000" \
      | xxd -r -p | curl --data-binary "@-" http://localhost:8084/fbs -v --output /dev/null
* TCP_NODELAY set
* Connected to localhost (::1) port 8084 (#0)
> POST /fbs HTTP/1.1
> Host: localhost:8084
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 56
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 56 out of 56 bytes
< HTTP/1.1 200 OK
< Date: Wed, 16 Jun 2021 12:52:22 UTC
< Content-Type: text/html
< X-YaRequestId: c8b2aee7ca5f4165ad25119b1850e778
< Server: userver/2.0 (20210616074040; rv:2c78282ea)
< X-YaTraceId: 8f4765a7176e41d28c3c6a677f00193e
< Connection: keep-alive
< Content-Length: 48
<
* Connection #0 to host localhost left intact

Functional testing

Naive functional tests for the service could be implemented using the testsuite in the following way:

async def test_flatbuf(service_client):
    body = bytearray.fromhex(
        '100000000c00180000000800100004000c000000140000001400000000000000'
        '16000000000000000a00000048656c6c6f20776f72640000',
    )
    response = await service_client.post('/fbs', data=body)
    assert response.status == 200
    assert 'application/octet-stream' == response.headers['Content-Type']

Do not forget to add the plugin in conftest.py:

# Adding a plugin from userver/testsuite/pytest_plugins/
pytest_plugins = ['pytest_userver.plugins.core']

Full sources

See the full example:

• samples/flatbuf_service/flatbuf_service.cpp
• samples/flatbuf_service/flatbuffer_schema.fbs
• samples/flatbuf_service/static_config.yaml
• samples/flatbuf_service/CMakeLists.txt
• samples/flatbuf_service/tests/test_flatbuf.py

---

⇦ Pre-caching data from HTTP remote | gRPC client and service ⇨