Senior Data Engineer - Chennai, India - MARICI Solutions INC

Description

Job DescriptionThis is a remote position.

We are seeking a highly skilled Senior Spark Engineer with expertise in Python, Spark, Databricks, and AWS to join our dynamic team.

The ideal candidate will possess a deep understanding of Spark architecture and be proficient in fine-tuning Spark jobs for optimal performance.

Additionally, strong knowledge of software engineering best practices, DevOps principles including CI/CD pipelines, Kubernetes, and either Jenkins, Airflow, or SageMaker, is essential for this role.

Experience in MLOps is desirable but not mandatory.

RequirementsProficiency in Spark, Python, Databricks, and AWS services.

7-10 years of experience

Lead the design, development, and implementation of Spark-based solutions for complex data processing tasks.

Collaborate with cross-functional teams to gather requirements and translate them into technical specifications.

Implement best practices in software engineering, DevOps, and MLOps methodologies.

Design and maintain data pipelines for efficient data processing and analysis.

Perform data modeling and schema design to support business requirements.

Mentor junior team members and provide technical guidance as needed.

Stay updated with the latest advancements in Spark, Python, and related technologies, and advocate for their adoption when appropriate.

Bachelor's degree in Computer Science, Engineering, or a related field. Master's degree preferred.

Strong understanding of software engineering principles and best practices.

Hands-on experience with DevOps tools and practices including CI/CD pipelines, Kubernetes, and either Jenkins, Airflow, or SageMaker.

Experience with MLOps practices is a plus.

Solid understanding of data engineering concepts and data modeling techniques.

Excellent problem-solving skills and attention to detail.

Ability to work independently and lead tasks effectively.

Strong communication and interpersonal skills.

RequirementsProficiency in Spark, Python, Databricks, and AWS services. 7-10 years of experience Lead the design, development, and implementation of Spark-based solutions for complex data processing tasks. Collaborate with cross-functional teams to gather requirements and translate them into technical specifications. Implement best practices in software engineering, DevOps, and MLOps methodologies. Design and maintain data pipelines for efficient data processing and analysis. Perform data modeling and schema design to support business requirements. Mentor junior team members and provide technical guidance as needed. Stay updated with the latest advancements in Spark, Python, and related technologies, and advocate for their adoption when appropriate. Bachelor's degree in Computer Science, Engineering, or a related field. Master's degree preferred. Strong understanding of software engineering principles and best practices. Hands-on experience with DevOps tools and practices including CI/CD pipelines, Kubernetes, and either Jenkins, Airflow, or SageMaker. Experience with MLOps practices is a plus. Solid understanding of data engineering concepts and data modeling techniques. Excellent problem-solving skills and attention to detail. Ability to work independently and lead tasks effectively. Strong communication and interpersonal skills.