kafka::ConsumerComponent Class Reference

#include <userver/kafka/consumer_component.hpp>

Detailed Description

Apache Kafka Consumer client component.

Static configuration example:

        # yaml
        kafka-consumer:                  # Kafka consumer component name must start with kafka-consumer
            group_id: test-group         # Consumers group current consumer will join on start
            topics:                      # List of topics consumer starts to listen
              - test-topic-1
              - test-topic-2
            auto_offset_reset: smallest  # Strategy for reading topics without committed offsets
            max_batch_size: 5            # Number of messages consumers waits until user-callback is called
            security_protocol: PLAINTEXT # Broker connection settings

Secdist format

A Kafka alias in secdist is described as a JSON object kafka_settings, containing credentials of Kafka brokers.

@pytest.fixture(scope='session')
def kafka_components() -> list[str]:
    return ['kafka-consumer', 'kafka-producer']
 
 
@pytest.fixture(scope='session')
def service_env(kafka_secdist) -> dict:
    """
    Note: kafka_secist fixture generates the secdist config
 
    Expected secdist format is:
    "kafka_settings": {
        "<kafka-component-name>": {
            "brokers": "<brokers comma-separated endpoint list>",
            "username": "SASL2 username (may be empty if use PLAINTEXT)",
            "password": "SASL2 password (may be empty if use PLAINTEXT)"
        }
    }
    """
 
    return {'SECDIST_CONFIG': json.dumps(kafka_secdist)}

Static options of kafka::ConsumerComponent :

Name	Description	Default value
group_id	Consumer group id. Topic partition evenly distributed between consumers with the same group_id.	–
client_id	Client identifier. May be an arbitrary string. Optional, but you should set this property on each instance because it enables you to more easily correlate requests on the broker with the client instance which made it, which can be helpful in debugging and troubleshooting scenarios.	userver
topics.[]	List of topics consumer subscribes. Each of the elements: topic name.	–
max_batch_size	Maximum number of messages consumer waits for new messages before calling a callback.	1
poll_timeout	Maximum amount of time consumer waits for new messages before calling a callback.	1s
message_key_log_format	Specifies the logging format for the message key. - 'plaintext': logs the message key as-is. - 'hex': logs the message key in hexadecimal format. Possible values: plaintext, hex.	plaintext
max_callback_duration	Duration user callback must fit not to be kicked from the consumer group. The duration must fit in [1ms; 86400000ms].	5m
restart_after_failure_delay	Backoff consumer waits until restart after user-callback exception.	10s
auto_offset_reset	Action to take when there is no initial offset in offset store or the desired offset is out of range: smallest, earliest, beginning - automatically reset the offset to the smallest offset largest, latest, end - automatically reset the offset to the largest offset, error - trigger an error (ERR__AUTO_OFFSET_RESET). Note: the policy applies iff there are not committed offsets in topic. Possible values: smallest, earliest, beginning, largest, latest, end, error.	–
env_pod_name	If defined and group_id value contains {pod_name} substring, the substring is replaced with the value of the environment variable env_pod_name.	none
security_protocol	Protocol used to communicate with brokers. Possible values: PLAINTEXT, SASL_SSL, SASL_PLAINTEXT, SSL.	–
sasl_mechanisms	SASL mechanism to use for authentication. Must be set if security_protocol equals SASL_SSL. Possible values: PLAIN, SCRAM-SHA-512.	none
ssl_ca_location	File or directory path to CA certificate(s) for verifying the broker's key. Must be set if security_protocol equals SASL_SSL or SSL. If set to probe, CA certificates are probed from the default certificates paths.	none
topic_metadata_refresh_interval	Period of time at which topic and broker metadata is refreshed in order to discover any new brokers, topics, partitions or partition leader changes.	5m
metadata_max_age	Metadata cache max age. Recommended value is 3 times topic_metadata_refresh_interval.	15m
rd_kafka_custom_options.*	A map of arbitrary librdkafka library configuration options. Full list of options is available by link: https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md. Note: This options is not guaranteed to be supported in userver-kafka, use it at your own risk. Each of the elements: librdkafka option value.	{}
debug_info_log_level	Log level for everything debug information.	debug
operation_log_level	Log level for infos about ordinary actions.	debug

Options inherited from components::ComponentBase :

Name	Description	Default value
load-enabled	Set to false to disable loading of the component.	true

Examples

samples/kafka_service/main.cpp, and samples/kafka_service/src/consumer_handler.cpp.

Definition at line 41 of file consumer_component.hpp.

Inheritance diagram for kafka::ConsumerComponent:

Public Member Functions
	ConsumerComponent (const components::ComponentConfig &config, const components::ComponentContext &context)
ConsumerScope	GetConsumer ()
	Returns consumer instance.
ComponentHealth	GetComponentHealth () const override
void	OnLoadingCancelled () override
void	OnAllComponentsLoaded () override
void	OnAllComponentsAreStopping () override

Static Public Member Functions
static yaml_config::Schema	GetStaticConfigSchema ()

Static Public Attributes
static constexpr std::string_view	kName {"kafka-consumer"}
	The default name of kafka::ConsumerComponent component.

Protected Types
using	LoggableComponentBase = ComponentBase

Member Typedef Documentation

LoggableComponentBase

using components::ComponentBase::LoggableComponentBase = ComponentBase

protectedinherited

Deprecated

use components::ComponentBase instead.

Definition at line 67 of file component_base.hpp.

Member Function Documentation

GetComponentHealth()

ComponentHealth components::ComponentBase::GetComponentHealth ( ) const

inlineoverridevirtualinherited

Override this function to inform the world of the state of your component.

Warning

The function is called concurrently from multiple threads.

Reimplemented from components::RawComponentBase.

Reimplemented in server::handlers::Restart.

Definition at line 35 of file component_base.hpp.

GetConsumer()

ConsumerScope kafka::ConsumerComponent::GetConsumer

(

)

Returns consumer instance.

See also

kafka::ConsumerScope

OnAllComponentsAreStopping()

void components::ComponentBase::OnAllComponentsAreStopping ( )

inlineoverridevirtualinherited

Component may use this function to stop doing work before the stop of the components that depend on it.

Base components may override it and make final to do some work before the derived object constructor is called. Don't use it otherwise.

Reimplemented from components::RawComponentBase.

Reimplemented in components::Server, and urabbitmq::ConsumerComponentBase.

Definition at line 58 of file component_base.hpp.

OnAllComponentsLoaded()

void components::ComponentBase::OnAllComponentsLoaded ( )

inlineoverridevirtualinherited

Component may use this function to finalize registration of other components that depend on it (for example, handler components register in server component, and the latter uses OnAllComponentsLoaded() to start processing requests).

Base components may override it and make final to do some work after the derived object constructor is called. Don't use it otherwise.

Reimplemented from components::RawComponentBase.

Reimplemented in components::Server, server::handlers::Ping, and urabbitmq::ConsumerComponentBase.

Definition at line 51 of file component_base.hpp.

OnLoadingCancelled()

void components::ComponentBase::OnLoadingCancelled ( )

inlineoverridevirtualinherited

Called once if the creation of any other component failed. If the current component expects some other component to take any action with the current component, this call is a signal that such action may never happen due to components loading was cancelled. Application components might not want to override it.

Reimplemented from components::RawComponentBase.

Definition at line 42 of file component_base.hpp.

---

The documentation for this class was generated from the following file:

• userver/kafka/consumer_component.hpp