Data Engineer - India - MethodHub

Description

We are a company in Cloud and Infrastructure, Data Engineering, BI Analytics , AI and RPA. We are rapidly growing fast in the market and now we reached employee size of We have offices in USA, Canada, 100+ Development Center in Thailand and 100+ Development Center in Chandigarh, Chennai ,Hyderabad and Bangalore.

US HQ, 1000-strong, Technology and Business Process Company; $40Mn in revenues

2 Development Centre Chandigarh

Payroll :-MethodHub

Client: Infosys

Experience :4+

• Design, develop, and maintain end-to-end data pipelines on AWS, utilizing serverless architecture.
• Implement data ingestion, validation, transformation procedures using AWS services such as Lambda, Glue, Kinesis, SNS, SQS, and CloudFormation.
• Write orchestration tasks within Apache Airflow.
• Develop and execute data quality checks using Great Expectations to ensure data integrity and reliability.
• Collaborate with other teams to understand mission objectives and translate them into data pipeline requirements.
• Utilize PySpark for complex data processing tasks within AWS Glue jobs.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or related field.
• Strong proficiency in Python programming language.
• Hands-on experience with AWS services.
• Experience with serverless architecture and Infrastructure as Code (IaC) using AWS CDK.
• Proficiency in Apache Airflow for orchestration of data pipelines.
• Familiarity with data quality assurance techniques and tools, preferably Great Expectations.
• Experience with SQL for data manipulation and querying.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment.
• Experience with Data Lakehouse, dbt, Apache Hudi data format is a plus.