utils::encoding Namespace Reference

Detailed Description

Encoders/decoders: TSKV, hex, and related parsers.

Classes
class	TskvParser
	A streaming parser for the TSKV variant that userver emits. More...

Enumerations
enum class	EncodeTskvMode {   kKey ,   kValue ,   kKeyReplacePeriod }

Functions
char	ToHexChar (int num)
	Converts number to hex character; number must be within range [0,16)
constexpr size_t	FromHexUpperBound (size_t size) noexcept
	Return upper limit on number of characters required to unhex input of given size.
void	ToHex (std::string_view input, std::string &out) noexcept
	Converts input to hex and writes data to output out.
void	ToHexBuffer (std::string_view input, utils::span< char > out) noexcept
	Converts input to hex and writes data to output out.
std::string	ToHex (std::string_view data) noexcept
	Allocates std::string, converts input and writes into said string.
std::string	ToHex (const void *encoded, size_t len) noexcept
	Allocates std::string, converts input and writes into said string.
size_t	FromHex (std::string_view encoded, std::string &out) noexcept
	Converts as much of input from hex as possible and writes data into out.
std::string	FromHex (std::string_view encoded) noexcept
	This FromHex overload allocates string and calls FromHex; if data is not fully a hex string, then it will be only partially processed.
std::string_view	GetHexPart (std::string_view encoded) noexcept
bool	IsHexData (std::string_view encoded) noexcept
	Checks that given range is fully a hex string. That is, if passed to FromHex, it will be fully processed.
std::string	ToHexString (uint64_t value)
	Interprets uint64_t value as array of bytes and applies ToHex to it.
template<typename OutIter>
OutIter	EncodeTskv (OutIter destination, char ch, EncodeTskvMode mode)
	Encode according to the TSKV rules, but without escaping the quotation mark (").
template<typename Container>
void	EncodeTskv (Container &container, std::string_view str, EncodeTskvMode mode)
	Encode according to the TSKV rules, but without escaping the quotation mark (").
template<typename OutIter>
USERVER_IMPL_FORCE_INLINE OutIter	EncodeTskv (OutIter destination, char ch, EncodeTskvMode mode)
	Encode according to the TSKV rules, but without escaping the quotation mark (").
template<typename TagConsumer>
TskvParser::RecordStatus	TskvReadRecord (TskvParser &parser, TagConsumer consumer)
	Read all keys-values for 1 TSKV record.
constexpr size_t	LengthInHexForm (size_t size) noexcept
	Calculate expected length of input after being hex encoded.
constexpr size_t	LengthInHexForm (std::string_view data) noexcept

Variables
constexpr char	kTskvKeyValueSeparator = '='
constexpr char	kTskvPairsSeparator = '\t'

Enumeration Type Documentation

EncodeTskvMode

enum class utils::encoding::EncodeTskvMode

strong

Definition at line 46 of file tskv.hpp.

Function Documentation

EncodeTskv() [1/3]

template<typename Container>

void utils::encoding::EncodeTskv	(	Container &	container,
		std::string_view	str,
		EncodeTskvMode	mode )

Encode according to the TSKV rules, but without escaping the quotation mark (").

Note

New contents are appended at the end of container. Some extra memory is reserved as necessary.

Template Parameters

Container

must be continuous and support at least the following operations: 1) c.data() 2) c.size() 3) c.resize(new_size)

Definition at line 410 of file tskv.hpp.

EncodeTskv() [2/3]

template<typename OutIter>

OutIter utils::encoding::EncodeTskv	(	OutIter	destination,
		char	ch,
		EncodeTskvMode	mode )

Encode according to the TSKV rules, but without escaping the quotation mark (").

Returns

The iterator to after the inserted chars.

Definition at line 67 of file tskv.hpp.

EncodeTskv() [3/3]

template<typename OutIter>

USERVER_IMPL_FORCE_INLINE OutIter utils::encoding::EncodeTskv	(	OutIter	destination,
		char	ch,
		EncodeTskvMode	mode )

Encode according to the TSKV rules, but without escaping the quotation mark (").

Returns

The iterator to after the inserted chars.

Definition at line 67 of file tskv.hpp.

FromHex() [1/2]

std::string utils::encoding::FromHex ( std::string_view encoded )

inlinenoexcept

This FromHex overload allocates string and calls FromHex; if data is not fully a hex string, then it will be only partially processed.

Definition at line 87 of file hex.hpp.

FromHex() [2/2]

size_t utils::encoding::FromHex ( std::string_view encoded, std::string & out )

noexcept

Converts as much of input from hex as possible and writes data into out.

To convert some range from hex, range must have even number of elements and every element must be hex character. To avoid throwing, algorithms consumes as much data as possible and returns how much it was able to process

Parameters

encoded	input range to convert
out	Result will be written into out. Previous value will be cleared.

Returns

Number of characters successfully parsed.

FromHexUpperBound()

size_t utils::encoding::FromHexUpperBound ( size_t size )

constexprnoexcept

Return upper limit on number of characters required to unhex input of given size.

For example:

• FromHexUpperBound(1) = 0, because you can't really unhex one char, it is only half byte. Where is the second half?
• FromHexUpperBound(2) = 1. Two chars will be converted into one byte
• FromHexUpperBound(5) = 2. First 4 chars will be unhexed into 2 bytes, and there will be one left.

Definition at line 38 of file hex.hpp.

GetHexPart()

std::string_view utils::encoding::GetHexPart ( std::string_view encoded )

noexcept

Returns range that constitutes hex string - e.g. sub-view of encoded that could be fully interpreted as hex string. Basically, if you have string_view a and c = GetHexPart(a), then range FromHex(c) will be fully parsed.

Parameters

encoded

input array of bytes.

LengthInHexForm() [1/2]

size_t utils::encoding::LengthInHexForm ( size_t size )

constexprnoexcept

Calculate expected length of input after being hex encoded.

Definition at line 24 of file hex.hpp.

LengthInHexForm() [2/2]

size_t utils::encoding::LengthInHexForm ( std::string_view data )

constexprnoexcept

Definition at line 26 of file hex.hpp.

ToHex() [1/3]

std::string utils::encoding::ToHex ( const void * encoded, size_t len )

inlinenoexcept

Allocates std::string, converts input and writes into said string.

Parameters

encoded	start of continuous range in memory
len	size of that range

Definition at line 67 of file hex.hpp.

ToHex() [2/3]

std::string utils::encoding::ToHex ( std::string_view data )

inlinenoexcept

Allocates std::string, converts input and writes into said string.

Parameters

data	range of input bytes

Definition at line 58 of file hex.hpp.

ToHex() [3/3]

void utils::encoding::ToHex ( std::string_view input, std::string & out )

noexcept

Converts input to hex and writes data to output out.

Parameters

input	bytes to convert
out	string to write data. out will be cleared

ToHexBuffer()

void utils::encoding::ToHexBuffer ( std::string_view input, utils::span< char > out )

noexcept

Converts input to hex and writes data to output out.

Warning

out must be pre-allocated to at least utils::encoding::LengthInHexForm bytes.

Parameters

input	bytes to convert
out	buffer to write data

ToHexChar()

char utils::encoding::ToHexChar

(

int

num

)

Converts number to hex character; number must be within range [0,16)

Exceptions

out_of_bounds

exception if num is out of range

ToHexString()

std::string utils::encoding::ToHexString ( uint64_t value )

inline

Interprets uint64_t value as array of bytes and applies ToHex to it.

Definition at line 105 of file hex.hpp.

TskvReadRecord()

template<typename TagConsumer>

TskvParser::RecordStatus utils::encoding::TskvReadRecord	(	TskvParser &	parser,
		TagConsumer	consumer )

Read all keys-values for 1 TSKV record.

Parameters

parser	parser that should have already found the start of the TSKV record using TskvParser::SkipToRecordBegin
consumer	a lambda with the signature (const std::string&, const std::string&) -> bool; the strings are temporaries, references to them should not be stored; can return false to skip the record and return immediately

Usage example:

struct Record {
    std::vector<std::pair<std::string, std::string>> tags;
};
struct Junk {
    std::string data;
};
struct Incomplete {
    std::string data;
};
using SingleParsedRecord = std::variant<Record, Junk, Incomplete>;
using ParsedRecords = std::vector<SingleParsedRecord>;
 
ParsedRecords ParseTskv(std::string_view in) {
    ParsedRecords result;
    utils::encoding::TskvParser parser{in};
 
    const auto account_incomplete = [&](const char* incomplete_begin) {
        const std::string_view remaining(incomplete_begin, in.data() + in.size() - incomplete_begin);
        if (!remaining.empty()) {
            result.push_back(Incomplete{std::string(remaining)});
        }
    };
 
    while (true) {
        const char* const junk_begin = parser.GetStreamPosition();
        const char* const record_begin = parser.SkipToRecordBegin();
 
        if (!record_begin) {
            // Note: production code should stop and wait for more data here.
            account_incomplete(junk_begin);
            break;
        }
 
        // Note: accounting for junk in-between records is supported, but is
        // typically not important in production.
        if (record_begin != junk_begin) {
            result.push_back(Junk{std::string(junk_begin, record_begin)});
        }
 
        std::vector<std::pair<std::string, std::string>> tags;
 
        const auto
            status = utils::encoding::TskvReadRecord(parser, [&](const std::string& key, const std::string& value) {
                tags.emplace_back(key, value);
                return true;
            });
 
        if (status == utils::encoding::TskvParser::RecordStatus::kIncomplete) {
            // Note: production code should stop and wait for more data here.
            account_incomplete(record_begin);
            break;
        }
 
        result.push_back(Record{std::move(tags)});
    }
 
    return result;
}

Definition at line 38 of file tskv_parser_read.hpp.

Variable Documentation

kTskvKeyValueSeparator

char utils::encoding::kTskvKeyValueSeparator = '='

inlineconstexpr

Definition at line 40 of file tskv.hpp.

kTskvPairsSeparator

char utils::encoding::kTskvPairsSeparator = '\t'

inlineconstexpr

Definition at line 41 of file tskv.hpp.