userver: File uploads and multipart/form-data testing
Loading...
Searching...
No Matches
File uploads and multipart/form-data testing

Before you start

Make sure that you can compile and run core tests as described at Configure, Build and Install.

Take a look at the Writing your first HTTP server and make sure that you do realize the basic concepts.

Step by step guide

It is a common task to upload an image or even work with multiple HTML form parameters at once.

An OpenAPI scheme for such form could look like:

# yaml
requestBody:
content:
multipart/form-data: # Media type
schema: # Request payload
type: object
properties: # Request parts
address: # Part1 (object)
type: object
properties:
street:
type: string
city:
type: string
profileImage: # Part 2 (an image)
type: string
format: binary

Here we have multipart/form-data request with a JSON object and a file.

Let's implement a /v1/multipart handler for that scheme and do something with each of the request parts!

HTTP handler component

Just like in the Writing your first HTTP server the handler itself is a component inherited from server::handlers::HttpHandlerBase:

class Multipart final : public server::handlers::HttpHandlerBase {
public:
// `kName` is used as the component name in static config
static constexpr std::string_view kName = "handler-multipart-sample";
// Component is valid after construction and is able to accept requests
using HttpHandlerBase::HttpHandlerBase;
std::string HandleRequestThrow(const server::http::HttpRequest& req, server::request::RequestContext&)
const override;
};

The primary functionality of the handler should be located in HandleRequestThrow function. To work with the multipart/form-data parameters use the appropriate server::http::HttpRequest functions:

std::string Multipart::HandleRequestThrow(const server::http::HttpRequest& req, server::request::RequestContext&)
const {
const auto content_type = http::ContentType(req.GetHeader(http::headers::kContentType));
if (content_type != "multipart/form-data") {
req.GetHttpResponse().SetStatus(server::http::HttpStatus::kBadRequest);
return "Expected 'multipart/form-data' content type";
}
const auto& image = req.GetFormDataArg("profileImage");
static constexpr std::string_view kPngMagicBytes = "\x89PNG\r\n\x1a\n";
if (!utils::text::StartsWith(image.value, kPngMagicBytes)) {
req.GetHttpResponse().SetStatus(server::http::HttpStatus::kBadRequest);
return "Expecting PNG image format";
}
req.GetHttpResponse().SetContentType(http::content_type::kApplicationJson);
const auto& address = req.GetFormDataArg("address");
auto json_addr = formats::json::FromString(address.value);
return fmt::format("city={} image_size={}", json_addr["city"].As<std::string>(), image.value.size());
}

Note the work with the image in the above snippet. The image has a binary representation that require no additional decoding. The bytes of a received image match the image bytes on hard-drive.

JSON data is received as a string. FromString function converts it to DOM representation.

Warning
Handle* functions are invoked concurrently on the same instance of the handler class. Use synchronization primitives or do not modify shared data in Handle*.

Static config

Now we have to configure the service by providing task_processors and default_task_processor options for the components::ManagerControllerComponent and configuring each component in components section:

# yaml
components_manager:
task_processors: # Task processor is an executor for coroutine tasks
main-task-processor: # Make a task processor for CPU-bound coroutine tasks.
worker_threads: 4 # Process tasks in 4 threads.
fs-task-processor: # Make a separate task processor for filesystem bound tasks.
worker_threads: 1
default_task_processor: main-task-processor # Task processor in which components start.
components: # Configuring components that were registered via component_list
server:
listener: # configuring the main listening socket...
port: 8080 # ...to listen on this port and...
task_processor: main-task-processor # ...process incoming requests on this task processor.
logging:
fs-task-processor: fs-task-processor
loggers:
default:
file_path: '@stderr'
level: debug
overflow_behavior: discard # Drop logs if the system is too busy to write them down.
handler-multipart-sample:
path: /v1/multipart
method: POST
task_processor: main-task-processor

Note that all the components and handlers have their static options additionally described in docs.

int main()

Finally, we add our component to the components::MinimalServerComponentList(), and start the server with static configuration file passed from command line.

int main(int argc, char* argv[]) {
const auto component_list = components::MinimalServerComponentList().Append<samples::Multipart>();
return utils::DaemonMain(argc, argv, component_list);
}

Build and Run

To build the sample, execute the following build steps at the userver root directory:

mkdir build_release
cd build_release
cmake -DCMAKE_BUILD_TYPE=Release ..
make userver-samples-multipart_service

The sample could be started by running make start-userver-samples-hello_service. The command would invoke testsuite start target that sets proper paths in the configuration files and starts the service.

To start the service manually run ./samples/multipart_service/userver-samples-multipart_service -c </path/to/static_config.yaml>.

Note
Without file path to static_config.yaml userver-samples-multipart_service will look for a file with name config_dev.yaml
CMake doesn't copy static_config.yaml files from samples directory into build directory.

Now you can send a request to your server from another terminal:

bash
$ curl -v -F address='{"street": "3, Garden St", "city": "Hillsbery, UT"}' \
-F "profileImage=@../scripts/docs/logo_in_circle.png" \
http://localhost:8080/v1/multipart
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> POST /v1/multipart HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Length: 10651
> Content-Type: multipart/form-data; boundary=------------------------048363632fdb9acc
>
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 24 Oct 2023 09:31:42 UTC
< Content-Type: application/octet-stream
< Server: userver/2.0 (20231024091132; rv:unknown)
< X-YaRequestId: f7cb383a987248179e5683713b141cea
< X-YaTraceId: 7976203e08074091b20b08738cb7fadc
< X-YaSpanId: 29233f54bc7e5009
< Accept-Encoding: gzip, identity
< Connection: keep-alive
< Content-Length: 76
<
* Connection #0 to host localhost left intact
city=Hillsbery, UT image_size=10173

Functional testing

Functional tests for the service could be implemented using the service_client fixture from pytest_userver.plugins.core in the following way:

import json
import aiohttp
async def test_ok(service_client, load_binary):
form_data = aiohttp.FormData()
# Uploading file
image = load_binary('logo_in_circle.png')
form_data.add_field('profileImage', image, filename='logo_in_circle.png')
# Adding JSON payload
address = {'street': '3, Garden St', 'city': 'Hillsbery, UT'}
form_data.add_field(
'address', json.dumps(address), content_type='application/json',
)
# Making a request and checking the result
response = await service_client.post('/v1/multipart', data=form_data)
assert response.status == 200
assert 'application/json' in response.headers['Content-Type']
assert response.text == f'city={address["city"]} image_size={len(image)}'

Do not forget to add the plugin in conftest.py:

pytest_plugins = ['pytest_userver.plugins.core']

Full sources

See the full example at: