Big Data Engineer - India - Insight Global

Description

Key Responsibilities

• Build and maintain data pipelines to enable faster, better, data-informed decision-making through customer enterprise business analytics
• Support ETL processes, fixing errors, repairing data, upgrading designs and code for resiliency
• Monitor and support data platforms and services: respond to alerts, improve process efficiency, reduce costs, manage data quality and integrity
• Ensure data integrity within reports and dashboards by reviewing data, identifying and resolving gaps and inconsistencies, and escalating as required to foster a partnered approach to data accuracy for business reporting purposes.
• Manage existing cloud-based storage solutions such as blob storage, databases, and data-warehouses
• Measure throughput, reliability and latency of data ingestion and processing for development, testing, and production environments
• Creation of analytical reports and/or dashboards
• Create and maintain documentation on data processes, policies, and configurations
• Provide on-call support for production systems as needed
• Continuously evaluate current systems and processes against industry standards and best practices

Required Skills

• Experience working with cloud-native data services and platforms such as Data Factory, Data Bricks, PerView, UnityCatalog, Calibra Data Marketplace, Synapse and Snowflake
• Strong skillsets within Azure Cloud Data Service Offerings:Kafka, ADF, ADLS Gen2, Databricks, PowerBI, Tableau, Snowflake, data pipeline orchestration
• Working proficiency building and managing ETL pipelines (data extraction, transformation, and loading),and data storage solutions
• Experience working with big data technologies such as Hadoop, Apache Hive, Redshift, Kafka, or Apache Spark
• Strong software development or scripting background with Python and/or C#
• In-depth understanding of data ingestion architecture, processes and best practices
• Strong experience working with continuous integration tools such as Jenkins, GitLab, etc.
• Understanding of data modeling, algorithms, and data transformation techniques
• Experience managing data integrity and quality for enterprise analytical reporting
• Ability to understand and adapt to communicated requirements from the client or team leads
• Experience providing production level support for labs/sandbox services
• Excellent communication and time management skills