Before you start

Take a look at official S3 documentation. Have your access key and secret key ready.

Step by step guide

In this example, we will create a client and make a request to S3 endpoint.

Installation

Find and link to userver module:

find_package(
    userver
    COMPONENTS core s3api
    REQUIRED
)
 
add_library(${PROJECT_NAME}_objs OBJECT src/s3api_client.cpp)
target_link_libraries(${PROJECT_NAME}_objs PUBLIC userver::s3api)
target_include_directories(${PROJECT_NAME}_objs PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src)

Creating client in code

We recommend wrapping your clients into some component. This way you can make sure that reference to http_client outlives S3 client. It can look, for example, like this:

 
// This component is not required to create S3 api client. It is used for demonstration purposes only.
class S3ApiSampleComponent : public components::LoggableComponentBase {
public:
    // `kName` is used as the component name in static config
    static constexpr std::string_view kName = "s3-sample-component";
 
    S3ApiSampleComponent(const components::ComponentConfig& config, const components::ComponentContext& context);
 
    s3api::ClientPtr GetClient();
 
    static yaml_config::Schema GetStaticConfigSchema();
 
private:
    const std::string url_;
    // http_client MUST outlive any dependent s3 client
    clients::http::Client& http_client_;
};
 

To create client itself, you do only few simple steps:

 
s3api::ClientPtr S3ApiSampleComponent::GetClient() {
    // Create connection settings
    auto connection_cfg = s3api::ConnectionCfg(
        std::chrono::milliseconds{100}, /* timeout */
        2,                              /* retries */
        std::nullopt                    /* proxy */
    );
 
    // Create connection object
    auto s3_connection = s3api::MakeS3Connection(http_client_, s3api::S3ConnectionType::kHttps, url_, connection_cfg);
 
    // Create authorizer.
    auto auth = std::make_shared<s3api::authenticators::AccessKey>("my-access-key", s3api::Secret("my-secret-key"));
 
    // create and return client
    return s3api::GetS3Client(std::move(s3_connection), std::move(auth), "mybucket");
}
 

Using client

Here is usage example

 
void DoVeryImportantThingsInS3(s3api::ClientPtr client) {
    std::string path = "path/to/object";
    std::string data{"some string data from S3ApiSampleComponent start"};
 
    client->PutObject(path, std::move(data));
    client->GetObject(path);
}
 

Testing

We provide google mock for client, that you can access by including header

#include <userver/s3api/utest/client_gmock.hpp>

With this mock, you have full power of Google Mock at your fingertips:

UTEST(S3ClientTest, HappyPath) {
    auto mock = std::make_shared<s3api::GMockClient>();
    EXPECT_CALL(
        *mock,
        PutObject(
            std::string_view{"path/to/object"},
            ::testing::_,
            ::testing::_,
            ::testing::_,
            ::testing::_,
            ::testing::_
        )
    )
        .Times(1)
        .WillOnce(::testing::Return("OK"));
    EXPECT_CALL(*mock, GetObject(std::string_view{"path/to/object"}, ::testing::_, ::testing::_, ::testing::_))
        .Times(1)
        .WillOnce(::testing::Return(std::string{"some data"}));
 
    samples::DoVeryImportantThingsInS3(mock);
}

To add tests to your project, add appropriate lines to CMakeLists.txt, like this:

add_executable(${PROJECT_NAME}-unittest unittests/client_test.cpp)
target_link_libraries(${PROJECT_NAME}-unittest ${PROJECT_NAME}_objs userver::utest userver::s3api-utest)
add_google_tests(${PROJECT_NAME}-unittest)

Testsuite support

Testsuite module is provided as part of testsuite plugins and can be found at:

testsuite/pytest_plugins/pytest_userver/plugins/s3api.py

To mock the S3 in testsuite adjust path to the S3 mocked URL in static service config from config hook:

import pytest
 
pytest_plugins = ['pytest_userver.plugins.s3api']
 
USERVER_CONFIG_HOOKS = ['userver_s3_mock']
 
 
@pytest.fixture(scope='session')
def userver_s3_mock(mockserver_info):
    def _do_patch(config_yaml, config_vars):
        components = config_yaml['components_manager']['components']
        components['s3-sample-component']['url'] = mockserver_info.url('/mds-s3')
 
    return _do_patch

Changes from the service would be visible in s3_mock_storage fixture:

async def test_sample_component_data_write(s3_mock_storage, service_client):
    bucket = 'mybucket'
    object_path = 'path/to/object'
    data = b'some string data from S3ApiSampleComponent start'
 
    assert s3_mock_storage[bucket].get_object(object_path) is None
 
    await service_client.run_task('some-s3-fill-task')
    stored = s3_mock_storage[bucket].get_object(object_path)
    assert stored.data == data

To fill the mocked S3 with data either a @pytest.mark.s3 can be used:

@pytest.mark.s3('mybucket0', {'some_path/to/an/object': 'preset.txt'})
async def test_s3_marker_preloads(s3_mock_storage, load):
    obj = s3_mock_storage['mybucket0'].get_object('some_path/to/an/object')
    assert obj is not None
    assert obj.data == load('preset.txt').encode()

or it can be done directly via the s3_mock_storage fixture.

Full Sources

See the full example at:

⇦ File uploads and multipart/form-data testing | Non-Coroutine Console Tool ⇨