Data Engineer - Mumbai, India - Domnic Lewis International

Description

• Implement data ingestion processes to extract data from various sources, including databases, APIs, and streaming platforms.
• Develop and maintain Extract, Transform, Load (ETL) pipelines using AWS Glue, AWS Lambda, or custom scripts.
• Perform data cleansing activity such as removing spaces, upper/lower, drop duplicate data, Renaming columns, etc.
• Data Transformation with PySpark, Python and SQL.
• Configure and optimize storage systems such as Amazon S3, Amazon Redshift, or Amazon DynamoDB to meet performance, scalability, and cost requirements.
• Performance tuning of long running PySpark or SQL queries.
• Monitor data pipelines, storage systems, and processing jobs to identify performance bottlenecks, data quality issues, and potential failures.
• Optimize data processing workflows and infrastructure configurations to improve performance, reliability, and cost efficiency.

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Proven experience of 3+ years Data Engineer, preferably with hands-on experience in designing and implementing data solutions on AWS/Azure
• Knowledge of Technical Skills such as PySpark, Python, SQL, AWS data Services (Glue, S3, EMR, Lambda and Redshift)
• Good to have data visualization experience using python or any BI tools.
• Strong knowledge of data security and compliance frameworks (e.g., GDPR, HIPAA) and implementation on AWS