userver: kafka::Producer Class Reference
Loading...
Searching...
No Matches
kafka::Producer Class Referencefinal

#include <userver/kafka/producer.hpp>

Detailed Description

Apache Kafka Producer Client.

It exposes the API to send an arbitrary message to the Kafka Broker to certain topic by given key and optional partition.

Important implementation details

Implementation does not block on any send to Kafka and asynchronously waits for each message to be delivered.

Producer periodically polls the metadata about delivered messages from Kafka Broker, blocking for some time, in separate task processor.

Producer maintains the per topic statistics including the broker connection errors.

Remarks
Destructor may block for no more than a couple of seconds to ensure all sent messages are properly delivered
See also
https://docs.confluent.io/platform/current/clients/producer.html

Definition at line 44 of file producer.hpp.

Public Member Functions

 Producer (std::unique_ptr< impl::Configuration > configuration, engine::TaskProcessor &producer_task_processor, std::chrono::milliseconds poll_timeout, std::size_t send_retries)
 Creates the Kafka Producer.
 
 ~Producer ()
 Waits until all messages are sent for a certain timeout and destroy the inner producer.
 
 Producer (const Producer &)=delete
 
 Producer (Producer &&)=delete
 
Produceroperator= (const Producer &)=delete
 
Produceroperator= (Producer &&)=delete
 
void Send (const std::string &topic_name, std::string_view key, std::string_view message, std::optional< std::uint32_t > partition=std::nullopt) const
 Sends given message to topic topic_name by given key and partition (if passed) with payload contains the message data. Asynchronously waits until the message is delivered or the delivery error occured.
 
engine::TaskWithResult< void > SendAsync (std::string topic_name, std::string key, std::string message, std::optional< std::uint32_t > partition=std::nullopt) const
 Same as Producer::Send, but returns the task which can be used to wait the message delivery.
 
void DumpMetric (utils::statistics::Writer &writer) const
 Dumps per topic messages produce statistics.
 

Static Public Attributes

static constexpr std::chrono::milliseconds kDefaultPollTimeout {10}
 Time producer waits for new delivery events.
 
static constexpr std::size_t kDefaultSendRetries = 5
 How many times Produce::Send* retries when delivery failures. Retries take place only when errors are transient.
 

Member Function Documentation

◆ DumpMetric()

void kafka::Producer::DumpMetric ( utils::statistics::Writer & writer) const

Dumps per topic messages produce statistics.

See also
impl/stats.hpp

◆ Send()

void kafka::Producer::Send ( const std::string & topic_name,
std::string_view key,
std::string_view message,
std::optional< std::uint32_t > partition = std::nullopt ) const

Sends given message to topic topic_name by given key and partition (if passed) with payload contains the message data. Asynchronously waits until the message is delivered or the delivery error occured.

No payload data is copied. Method holds the data until message is delivered.

thread-safe and can be called from any number of threads simultaneously.

Producer::Send call may take at most delivery_timeout_ms x send_retries_count milliseconds.

If partition not passed, partition is chosen by internal Kafka partitioner.

Warning
if enable_idempotence option is enabled, do not use both explicit partitions and Kafka-chosen ones
Exceptions
std::runtime_errorif message is not delivery and acked by Kafka Broker

◆ SendAsync()

engine::TaskWithResult< void > kafka::Producer::SendAsync ( std::string topic_name,
std::string key,
std::string message,
std::optional< std::uint32_t > partition = std::nullopt ) const

Same as Producer::Send, but returns the task which can be used to wait the message delivery.

Warning
If user schedules a batch of send requests with Producer::SendAsync, some send requests may be retried by the library (for instance, in case of network blink). Though, the order messages are written to partition may differ from the order messages are initially sent

Member Data Documentation

◆ kDefaultPollTimeout

constexpr std::chrono::milliseconds kafka::Producer::kDefaultPollTimeout {10}
staticconstexpr

Time producer waits for new delivery events.

Definition at line 47 of file producer.hpp.

◆ kDefaultSendRetries

constexpr std::size_t kafka::Producer::kDefaultSendRetries = 5
staticconstexpr

How many times Produce::Send* retries when delivery failures. Retries take place only when errors are transient.

Remarks
librdkafka already has a retry mechanism. Moreover, user-retried requests may lead to messages reordering or duplication. Nevertheless, the library retries a small list of delivery errors (such as message guaranteed timeouts), including errors those are not retried by librdkafka and errors that may occure when the Kafka cluster or topic have just been created (for instance, in tests)
See also
impl/producer_impl.cpp for the list of errors retryable by library

Definition at line 59 of file producer.hpp.


The documentation for this class was generated from the following file: