Event-Based Consumer/Worker Apps On Devtron

Event-Based Consumer/Worker

1. Separate Containerized App

We must be creating separate consumer/worker apps on Devtron for the kafka-based consumers or rabbitmq based workers

Benefits:-
a) This way we can scale the consumers independently and can increase or decrease the pods of the consumer app

b) Separation Of Concerns of Logic: This is more of following the monolith vs microservices architecture pattern. Each Consumer is a separate entity and can be managed easily.

2. Consumers in the same service

We should create the consumers of a particular microservice in the same repository itself. In this way, we can reuse that particular service's already-written code to implement that app's functionality.

3.Separate Dockerfile

We must create a separate dockerfile for the consumer/worker app and a separate requirements file to use for it. In this way, we are separating the requirements of both the containerised applications as there can be some packages which are not needed to be installed in the consumer/worker app.

E.g. Dockerfile.consumer

4. Code Structure

a) Separate Package:- We are creating a new consumers package in our application for kafka-based consumers.

Package Name:- consumers

b) start_consumer.py :- There will be an entry point to start this consumer.We are running the consumer here to poll the messages received from the Kafka topic. We are also initializing the Django apps here so that we can use the already created Django apps in our consumer applications as well.

c) consumer_config.py:- All the Kafka-related configurations are put into this file

d) base_kafka_consumer.py:- We have created an abstract base consumer class for the kafka-based consumer. We have kept the processMessage method here which can be implemented by the subclasses.

e) consumer_factory.py:-We have created a subclass of the BaseKafkaConsumer class which is processing the messages received for the consumer and it is returning the implementation class which will handle the message.

f) news_consumers.py:- We have created the factory classes here in this file which will be actual implementation classes to process the messages received from a particular topic.