Data Engineer - Chennai Metropolitan Area, India - Mastek

Description

Data Engineer - AWS (Glue, Athena, Lambda)

We are seeking a talented and experienced Data Engineer with expertise in AWS services, specifically AWS Glue, Athena, and Lambda. The ideal candidate will play a crucial role in designing, implementing, and optimizing data pipelines, transformations, and analytics solutions on the AWS platform. As a key member of our data engineering team, you will contribute to the development of scalable and efficient data architectures that support our organization's data-driven initiatives.

Responsibilities:

• Data Pipeline Development:
• Design, develop, and implement scalable and efficient data pipelines using AWS Glue for data extraction, transformation, and loading (ETL) processes.
• Ensure data quality, integrity, and consistency throughout the pipeline.
• Data Cataloging and Metadata Management:
• Utilize AWS Glue Data Catalog to create and maintain metadata for various data sources, ensuring accurate and up-to-date information for the organization's data assets.
• Serverless Computing with Lambda:
• Design and implement serverless data processing using AWS Lambda functions, optimizing for performance, scalability, and cost-effectiveness.
• Data Lake Architecture:
• Contribute to the design and implementation of data lake architectures on AWS, leveraging services such as Amazon S3 for scalable and cost-efficient storage.
• Querying and Analysis with Athena:
• Develop and optimize queries using Amazon Athena for ad-hoc analysis and reporting on data stored in the data lake.
• Performance Tuning and Optimization:
• Monitor and optimize the performance of data pipelines, ETL processes, and queries to ensure efficient and timely data processing.
• Security and Compliance:
• Implement and adhere to best practices for data security and compliance within the AWS environment.
• Documentation and Knowledge Sharing:
• Document data engineering processes, architectures, and configurations.
• Share knowledge and best practices with the broader data engineering and analytics teams.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer with a focus on AWS services, including Glue, Athena, and Lambda.
• Strong proficiency in Python or other relevant programming languages used in AWS Glue and Lambda.
• Expertise in designing and implementing ETL processes, data pipelines, and data lake architectures.
• In-depth understanding of data cataloging, metadata management, and data governance principles.
• Experience with query languages, SQL, and optimization techniques for Athena.
• Familiarity with AWS security and compliance practices.
• Excellent communication and collaboration skills.
• AWS certifications in relevant areas (e.g., AWS Certified Data Analytics, AWS Certified Big Data) are a plus.