Formats (JSON, YAML, BSON, ...)

Userver provides classes for reading, working with, and serializing various data formats. Classes for different formats have an almost identical interface. There are common points for customizing parsers and serializers of custom types.

formats::*::Value

Classes formats::json::Value, formats::bson::Value, yaml_config::YamlConfig and formats::yaml::Value are intended for non-modifying work with formats (in other words, for reading data).

Usage Example:

  // #include <userver/formats/json.hpp>
 
  formats::json::Value json = formats::json::FromString(R"({
    "key1": 1,
    "key2": {"key3":"val"}
  })");
 
  const auto key1 = json["key1"].As<int>();
  ASSERT_EQ(key1, 1);
 
  const auto key3 = json["key2"]["key3"].As<std::string>();
  ASSERT_EQ(key3, "val");

Customization of formats::*::Value::As<T>()

In order for formats::*::Value to be able to represent data as a C++ type, you should write a special function Parse for that C++ type. Parse should be located in the namespace of the type or may be located in the formats::common namespace if the type comes from third-party library that you have no control of:

namespace my_namespace {
 
struct MyKeyValue {
  std::string field1;
  int field2;
};
//  The function must be declared in the namespace of your type
MyKeyValue Parse(const formats::json::Value& json,
                 formats::parse::To<MyKeyValue>) {
  return MyKeyValue{
      json["field1"].As<std::string>(""),
      json["field2"].As<int>(1),  // return `1` if "field2" is missing
  };
}
 
TEST(FormatsJson, ExampleUsageMyStruct) {
  formats::json::Value json = formats::json::FromString(R"({
    "my_value": {
        "field1": "one",
        "field2": 1
    }
  })");
  auto data = json["my_value"].As<MyKeyValue>();
  EXPECT_EQ(data.field1, "one");
  EXPECT_EQ(data.field2, 1);
}
}  // namespace my_namespace

You can write a single parser for all formats, just make it a template:

#include <userver/formats/json.hpp>
#include <userver/formats/yaml.hpp>
 
USERVER_NAMESPACE_BEGIN
 
namespace my_namespace {
 
struct MyKeyValue {
  std::string field1;
  int field2;
};
 
// The function must be declared in the namespace of your type
template <class Value>
MyKeyValue Parse(const Value& data, formats::parse::To<MyKeyValue>) {
  return MyKeyValue{
      data["field1"].template As<std::string>(),
      data["field2"].template As<int>(-1),  // return `1` if "field2" is missing
  };
}
 
TEST(CommonFormats, Parse) {
  // json
  formats::json::Value json = formats::json::FromString(R"({
    "my_value": {
        "field1": "one",
        "field2": 1
    }
  })");
  auto data_json = json["my_value"].As<MyKeyValue>();
  EXPECT_EQ(data_json.field1, "one");
  EXPECT_EQ(data_json.field2, 1);
 
  // yaml
  formats::yaml::Value yaml = formats::yaml::FromString(R"(
    my_value:
        field1: "one"
        field2: 1
  )");
  auto data_yaml = yaml["my_value"].As<MyKeyValue>();
  EXPECT_EQ(data_yaml.field1, "one");
  EXPECT_EQ(data_yaml.field2, 1);
}
 
}  // namespace my_namespace

Inline helpers formats::*::Make*

To build objects of trivial types some of the formats provide inline helpers, like formats::json::MakeArray(), formats::json::MakeObject():

const auto kDoc = formats::json::MakeObject(
    "key1", 1,
    "key2", "val",
    "key3", formats::json::MakeObject("sub", -1),
    "key4", formats::json::MakeArray(1, 2, 3),
    "key5", 10.5,
    "key6", false
);

Or formats::bson::MakeDoc(), formats::bson::MakeArray():

  const auto doc = formats::bson::MakeDoc(
      "key_a", formats::bson::MakeArray(0, 1, 2),           //
      "key_d", formats::bson::MakeDoc("one", 1, "two", 2),  //
      "key_n", nullptr                                      //
  );
 
  auto umap = doc["key_d"].As<std::unordered_map<std::string, int>>();
  EXPECT_EQ(1, umap["one"]);
  EXPECT_EQ(2, umap["two"]);

Those inline helper functions usually work slightly faster than formats::*::ValueBuilder. However, if you need a std::string with JSON the fastest way would be to use the Streaming Serialization. Inline helpers could not be customized for used provided types, unlike other format building types. Inline helpers could produce broken value on bad input because they skip some of the checks, for example a key uniqueness check.

formats::*::ValueBuilder

Classes formats::json::ValueBuilder, formats::bson::ValueBuilder and formats::yaml::ValueBuilder are designed for building objects of a given format.

Usage Example:

  // #include <userver/formats/json.hpp>
  formats::json::ValueBuilder builder;
  builder["key1"] = 1;
  builder["key2"]["key3"] = "val";
  formats::json::Value json = builder.ExtractValue();
 
  ASSERT_EQ(json["key1"].As<int>(), 1);
  ASSERT_EQ(json["key2"]["key3"].As<std::string>(), "val");

Customization of formats::*::ValueBuilder

In order for formats::*::ValueBuilder to be able to represent a C++ type in the specified format, you should write a special function Serialize for that C++ type. Serialize should be located in the namespace of the type or may be located in the formats::common namespace if the type comes from third-party library that you have no control of:

namespace my_namespace {
 
struct MyKeyValue {
  std::string field1;
  int field2;
};
 
// The function must be declared in the namespace of your type
formats::json::Value Serialize(const MyKeyValue& data,
                               formats::serialize::To<formats::json::Value>) {
  formats::json::ValueBuilder builder;
  builder["field1"] = data.field1;
  builder["field2"] = data.field2;
 
  return builder.ExtractValue();
}
 
TEST(JsonValueBuilder, ExampleCustomization) {
  MyKeyValue object = {"val", 1};
  formats::json::ValueBuilder builder;
  builder["example"] = object;
  auto json = builder.ExtractValue();
  ASSERT_EQ(json["example"]["field1"].As<std::string>(), "val");
  ASSERT_EQ(json["example"]["field2"].As<int>(), 1);
}
 
TEST(JsonValueBuilder, StringViewRemove) {
  formats::json::ValueBuilder builder;
  const std::string str = "ab";
  builder["a"] = 1;
  builder[str] = 2;
  builder.Remove(std::string_view(str.data(), 1));
 
  EXPECT_EQ(1, builder.GetSize());
 
  const auto value = builder.ExtractValue();
  EXPECT_EQ(2, value[str].As<int>());
}
 
TEST(JsonValueBuilder, StringViewHasMember) {
  formats::json::ValueBuilder main_builder;
  const std::string str = "ab";
  main_builder[str] = 2;
  EXPECT_EQ(false, main_builder.HasMember("a"));
  EXPECT_EQ(false, main_builder.HasMember(std::string_view(str.data(), 1)));
  EXPECT_EQ(true, main_builder.HasMember(std::string_view(str.data(), 2)));
}
 
TEST(JsonValueBuilder, StringViewEmplaceNocheck) {
  formats::json::ValueBuilder main_builder;
  const std::string str = "ab";
  main_builder.EmplaceNocheck(std::string_view(str.data(), 1), 1);
  main_builder.EmplaceNocheck(std::string_view(str.data(), 2), 2);
  main_builder.EmplaceNocheck(std::string_view(std::string(1024, 'a') + 'b'),
                              1025);
 
  const auto value = main_builder.ExtractValue();
  EXPECT_EQ(1, value["a"].As<int>());
  EXPECT_EQ(2, value["ab"].As<int>());
  EXPECT_EQ(1025, value[std::string(1024, 'a') + 'b'].As<int>());
}
 
}  // namespace my_namespace
 

You can write a single serializer for all formats, for make it a template:

#include <userver/formats/json.hpp>
#include <userver/formats/yaml.hpp>
 
USERVER_NAMESPACE_BEGIN
 
namespace my_namespace {
 
struct MyKeyValue {
  std::string field1;
  int field2;
};
 
// The function must be declared in the namespace of your type
template <class Value>
Value Serialize(const MyKeyValue& data, formats::serialize::To<Value>) {
  typename Value::Builder builder;
  builder["field1"] = data.field1;
  builder["field2"] = data.field2;
 
  return builder.ExtractValue();
}
 
TEST(CommonFormats, Serialize) {
  MyKeyValue obj = {"val", 1};
 
  // json
  formats::json::ValueBuilder builderJson;
  builderJson["example"] = obj;
  auto json = builderJson.ExtractValue();
  EXPECT_EQ(json["example"]["field1"].As<std::string>(), "val");
  EXPECT_EQ(json["example"]["field2"].As<int>(), 1);
 
  // yaml
  formats::yaml::ValueBuilder builderYaml;
  builderYaml["example"] = obj;
  auto yaml = builderYaml.ExtractValue();
  EXPECT_EQ(yaml["example"]["field1"].As<std::string>(), "val");
  EXPECT_EQ(yaml["example"]["field2"].As<int>(), 1);
}
 
}  // namespace my_namespace

Streaming Serialization

For runtime-critical code, it is possible to use streaming serializers. They allow you to serialize several times faster than formats::json::ValueBuilder, but should be used carefully because may produce broken format.

At the moment, stream serialization is implemented only for JSON via the formats::json::StringBuilder.

In order for stream serialization to work with your data type, you need to define the WriteToStream function in the namespace of your type:

namespace my_namespace {
 
struct MyKeyValue {
  std::string field1;
  int field2;
};
 
// The function must be declared in the namespace of your type
inline void WriteToStream(const MyKeyValue& data,
                          formats::json::StringBuilder& sw) {
  // A class that adds '{' in the constructor and '}' in the destructor to JSON
  formats::json::StringBuilder::ObjectGuard guard{sw};
 
  sw.Key("field1");
  // Use the WriteToStream functions for values, don't work with StringBuilder
  // directly
  WriteToStream(data.field1, sw);
 
  sw.Key("field2");
  WriteToStream(data.field2, sw);
}
 
TEST(JsonStringBuilder, ExampleUsage) {
  StringBuilder sb;
  MyKeyValue data = {"one", 1};
  WriteToStream(data, sb);
  ASSERT_EQ(sb.GetString(), "{\"field1\":\"one\",\"field2\":1}");
}
 
}  // namespace my_namespace

Note that you may get invalid JSON, since:

• methods format::json methods::StringBuilder::Key do not check the uniqueness of keys
• StringBuilder itself does not put curly brackets, you need to use formats::json::StringBuilder::ObjectGuard
• StringBuilder itself does not put square brackets, you need to use formats::json::StringBuilder::ArrayGuard
• You can write any nonsense in JSON using StringBuilder methods
• etc.

Test your serializers!

---

⇦ Synchronization Primitives | Logging and Tracing ⇨