Specialist - Data Engineering - Bhubaneswar, India - Larsen & Toubro Infotech Ltd

Description

Specialist - Data Engineering

How you'll help us Keep Climbing (overview & key responsibilities):

Customer org IT is on a journey of transformation. We are changing the way we do business from top to bottom. As thought leaders within Customer org, we strive to create meaningful and innovative solutions and are looking for team members to help us realize our vision. Customer org IT resources are thinkers, doers, innovators. We are proactive. We are collaborative. We deliver impact to our customers. Join us on our transformation journey in becoming a world-class IT organization at the world's best airline

The Data Engineer position play a leading role on the Portfolio Data Science/Maintenance Productivity team, responsible for transforming data from disparate systems to provide insights and analytics for business stakeholders. You'll leverage cloud-based infrastructure to implement technology solutions that are scalable, resilient, and efficient. You will collaborate with other Data Engineers, Data Scientists, Data Analysts, Software Development Engineers on a cross-functional teams. You will design, implement, and operate data engineering solutions, using Agile methodology, that empower users to make informed business decisions.

Responsibilities

•Ingest data from various internal and external sources into AWS data lake S3 buckets, utilizing appropriate methods including AWS Glue, Lambda, Lake Formations, DMS, Airflow, Cross account replication and/or SFTP.

•Building infrastructure as code using, AWS CDK or Cloud Formation.

•Create and utilize CI/CD pipelines to promote code through test and production environments.

•Build glue ETL pipelines to structure and curate data.

•Conduct code reviews on all code deployments.

•Maintain AWS environment to eliminate vulnerabilities and keep the systems working smoothly.

•Coordinate and/or complete resolution of production issues.

•Drive path to production processes, including documentation and approval processes.

•Partner with business team on data governance

Qualifications / Preferred Qualifications

•Bachelor of Science degree in Computer Science or equivalent

•At least 7 years of post-degree professional experience

•4+ years of experience with AWS

•4+ years development experience building and maintaining AWS ETL pipelines.

•Hands-on experience with AWS services such as S3, Lambda Functions, Step Functions, CDK, AWS Glue and IAM role configuration

•Experience with cloud data migration tools such as AWS DMS and Cross Account Replication

•Strong knowledge and experience with Python

•Preferred to have experience with Data Lake Formations.

•Familiar with best practices for data ingestion and data design

•Track record of advancing new technologies to improve data quality and reliability.

•Continuously improve quality, efficiency, and scalability of data pipelines

•Self-motivated, work independently, and have direct experience with all aspects of the software development lifecycle, from design to deployment.

•Excellent written and speaking communication skills are required as we work in a collaborative cross-functional environment and interact with the full spectrum of business divisions.

•Expert skills working with queries/applications, including performance tuning, utilizing indexes, and materialized views to improve query performance.

•Identify necessary business rules for extracting data along with functional or technical risks related to data sources (e.g. data latency, frequency, etc.)

•Develop initial queries for profiling data, validating analysis, testing assumptions, driving data quality assessment specifications, and define a path to deployment.

Job Segment: Testing, Cloud, Computer Science, Engineer, Database, Technology, Engineering