Data Engineer - Chennai, India - KAYA Global

Description

As a Senior Azure Data Engineer, your primary role is to design, implement, and manage data solutions on the Microsoft Azure cloud platform. You will work closely with data architects, data scientists, and other stakeholders to develop and optimize data pipelines, data storage, and analytics solutions. Your expertise in Azure technologies will contribute to building scalable, efficient, and secure data systems that support data-driven decision-making and enable advanced analytics.

Responsibilities

• Data Architecture and Design: Collaborate with data architects and stakeholders to design and implement scalable data architectures on the Azure platform. Define data storage, data ingestion, data transformation, and data processing strategies that align with business requirements and best practices.
• Data Pipeline Development: Develop robust and scalable data pipelines for ingesting, processing, and transforming data from various sources using Azure services such as Azure Data Factory, Azure Databricks, or Azure Logic Apps. Ensure data pipelines are reliable, performant, and scalable to handle large volumes of data.
• Data Security and Governance: Implement data security and privacy measures to ensure compliance with regulations and industry best practices. Configure access controls, encryption, and auditing mechanisms for data protection. Establish data governance practices and enforce data quality standards.
• Performance Optimization: Optimize data processing and query performance by fine-tuning Azure services, utilizing appropriate data partitioning and indexing techniques, and implementing caching mechanisms. Monitor and troubleshoot performance issues to ensure optimal system performance.
• Collaboration and Leadership: Collaborate with cross-functional teams, including data scientists, business analysts, and software engineers, to understand data requirements and deliver effective data solutions. Provide technical guidance and mentorship to junior data engineers. Lead and contribute to data engineering projects and initiatives.
• Continuous Learning and Innovation: Stay updated with the latest Azure technologies and data engineering trends. Evaluate and recommend new Azure services, tools, or methodologies that can enhance data engineering capabilities and drive innovation.