kafka::ConsumerComponent Class Reference

#include <userver/kafka/components/consumer_component.hpp>

Detailed Description

Apache Kafka Consumer client component.

Static configuration example:

        # yaml
        kafka-consumer:
            group_id: test-group
            topics:
              - test-topic-send
              - test-topic-consume-1
              - test-topic-consume-2
              - test-topic-consume-produced-1
              - test-topic-consume-produced-2
            auto_offset_reset: smallest
            max_batch_size: 10
            env_pod_name: HOSTNAME
            enable_auto_commit: false
            security_protocol: PLAINTEXT
            rd_kafka_custom_options:
                session.timeout.ms: 77000
                fetch.wait.max.ms: 1000

Secdist format

A Kafka alias in secdist is described as a JSON object kafka_settings, containing credentials of Kafka brokers.

@pytest.fixture(scope='session')
def service_env():
    single_setting = {
        'brokers': os.getenv('KAFKA_RECIPE_BROKER_LIST'),
        'username': '',
        'password': '',
    }
    logging.info(f'Brokers are: {single_setting["brokers"]}')
 
    secdist_config = {
        'kafka_settings': {
            'kafka-consumer': single_setting,
            'kafka-producer-first': single_setting,
            'kafka-producer-second': single_setting,
        },
    }
 
    return {'SECDIST_CONFIG': json.dumps(secdist_config)}

Static options:

Name	Description	Default value
group_id	consumer group id	–
topics	list of topics consumer subscribes	–
enable_auto_commit	whether to automatically and periodically commit offsets	false
auto_offset_reset	action to take when there is no initial offset in offset store	smallest
max_batch_size	maximum batch size for one callback call	–
env_pod_name	environment variable to substitute {pod_name} substring in group_id	none
security_protocol	protocol used to communicate with brokers	–
sasl_mechanisms	SASL mechanism to use for authentication	none
ssl_ca_location	File or directory path to CA certificate(s) for verifying the broker's key	5m
topic_metadata_refresh_interval	period of time at which topic and broker metadata is refreshed	15m
metadata_max_age	metadata cache max age	none
rd_kafka_custom_options	a map of librdkafka library additional options	'{}'

Definition at line 56 of file consumer_component.hpp.

Inheritance diagram for kafka::ConsumerComponent:

Public Member Functions
	ConsumerComponent (const components::ComponentConfig &config, const components::ComponentContext &context)
ConsumerScope	GetConsumer ()
ComponentHealth	GetComponentHealth () const override
void	OnLoadingCancelled () override
void	OnAllComponentsLoaded () override
void	OnAllComponentsAreStopping () override

Static Public Member Functions
static yaml_config::Schema	GetStaticConfigSchema ()

Static Public Attributes
static constexpr std::string_view	kName = "kafka-consumer"

Protected Types
using	LoggableComponentBase = ComponentBase
	Legacy alias, use ComponentBase instead.

Member Typedef Documentation

LoggableComponentBase

using components::ComponentBase::LoggableComponentBase = ComponentBase

protectedinherited

Legacy alias, use ComponentBase instead.

Definition at line 69 of file component_base.hpp.

Member Function Documentation

GetComponentHealth()

ComponentHealth components::ComponentBase::GetComponentHealth ( ) const

inlineoverridevirtualinherited

Override this function to inform the world of the state of your component.

Warning

The function is called concurrently from multiple threads.

Reimplemented from components::RawComponentBase.

Definition at line 35 of file component_base.hpp.

OnAllComponentsAreStopping()

void components::ComponentBase::OnAllComponentsAreStopping ( )

inlineoverridevirtualinherited

Component may use this function to stop doing work before the stop of the components that depend on it.

Base components may override it and make final to do some work before the derived object constructor is called. Don't use it otherwise.

Reimplemented from components::RawComponentBase.

Reimplemented in urabbitmq::ConsumerComponentBase, and components::Server.

Definition at line 60 of file component_base.hpp.

OnAllComponentsLoaded()

void components::ComponentBase::OnAllComponentsLoaded ( )

inlineoverridevirtualinherited

Component may use this function to finalize registration of other components that depend on it (for example, handler components register in server component, and the latter uses OnAllComponentsLoaded() to start processing requests).

Base components may override it and make final to do some work after the derived object constructor is called. Don't use it otherwise.

Reimplemented from components::RawComponentBase.

Reimplemented in urabbitmq::ConsumerComponentBase, components::Server, and server::handlers::Ping.

Definition at line 53 of file component_base.hpp.

OnLoadingCancelled()

void components::ComponentBase::OnLoadingCancelled ( )

inlineoverridevirtualinherited

Called once if the creation of any other component failed. If the current component expects some other component to take any action with the current component, this call is a signal that such action may never happen due to components loading was cancelled. Application components might not want to override it.

Reimplemented from components::RawComponentBase.

Definition at line 44 of file component_base.hpp.

Member Data Documentation

kName

constexpr std::string_view kafka::ConsumerComponent::kName = "kafka-consumer"

staticconstexpr

Definition at line 58 of file consumer_component.hpp.

---

The documentation for this class was generated from the following file:

• userver/kafka/components/consumer_component.hpp