Kafka Architect - Kolkata, India - Ultrabot Innovations

Description

Job Description :

As a Kafka Architect specializing in Spark and Apache Server, you will play a key role in designing, architecting, and implementing real-time data streaming solutions using Apache Kafka, Apache Spark, and related technologies.

You will work closely with our clients and internal teams to understand business requirements, design robust architectures, and implement scalable and reliable data pipelines to enable real-time data processing and analytics.

This role offers an exciting opportunity to work on cutting-edge projects and shape the future of data streaming and analytics solutions.

Key Responsibilities :

Architectural Design :

Design and architect end-to-end data streaming solutions using Apache Kafka, Apache Spark, and other relevant technologies, considering scalability, reliability, performance, and security requirements.

Platform Setup and Configuration :

Set up and configure Kafka clusters, Spark clusters, and Apache Servers, including installation, configuration, tuning, and optimization for optimal performance and resource utilization.

Data Ingestion and Processing :

Implement data ingestion pipelines to capture, collect, and ingest data from various sources into Kafka topics, and design and develop Spark streaming applications for real-time data processing, transformation, and analysis.

Integration :

Integrate Kafka and Spark with other systems and technologies in the data ecosystem, including data lakes, databases, messaging systems, and analytics platforms, to facilitate seamless data exchange and interoperability.

Monitoring and Performance Tuning :

Implement monitoring and alerting solutions to track the health and performance of Kafka and Spark clusters, and perform performance tuning and optimization to improve throughput, latency, and reliability.

Security and Compliance :

Implement security controls and best practices to ensure data privacy, confidentiality, and compliance with regulatory requirements, including encryption, authentication, authorization, and auditing.

Documentation and Best Practices :

Create and maintain comprehensive documentation, architectural diagrams, and best practices for Kafka and Spark implementations, and provide guidance and training to internal teams and clients.

Technical Leadership :

Provide technical leadership and guidance to development teams, mentor junior architects and engineers, and collaborate with cross-functional teams to drive successful project delivery.

Qualifications :

- Bachelor's degree in Computer Science, Information Technology, or related field.

- 6-10 years of experience in architecting and implementing data streaming solutions using Apache Kafka, Apache Spark, and related technologies.

- Deep understanding of Kafka architecture, components, and ecosystem, including Kafka brokers, producers, consumers, topics, partitions, and replication.

- Hands-on experience with Apache Spark, including Spark Streaming, Spark SQL, and Spark MLlib, for real-time data processing, analytics, and machine learning.

- Proficiency in setting up and configuring Kafka clusters and Spark clusters, including cluster sizing, deployment, configuration management, and performance tuning.

- Experience with Apache Server setup and configuration for serving web applications and APIs.

- Strong knowledge of data integration and interoperability concepts, including data formats (Avro, Parquet, JSON), serialization (Apache Avro, Apache Thrift), and messaging protocols (Kafka protocol, RESTful APIs).

- Excellent problem-solving skills, with the ability to analyze complex technical challenges and design scalable and reliable solutions.

- Strong communication and interpersonal skills, with the ability to effectively communicate technical concepts to non-technical stakeholders and collaborate with cross-functional teams.