- 6+ years of experience into Azure.
- Good exposure in working with Azure data bricks – Pyspark, Spark SQL, Scala (Lazy evaluation and delta tables, Parquet formats, Working with large and complex datasets)
- Strong experience with DWH using Synapse (distribution of tables (hash, replicate, round robin - scenario). Partitioning of data and Polybase.
- ETL (Incremental loading .. SCD1, SCD2, Upserts)
- Data factory (pipeline orchestration, scheduling and different types of triggers).
- Logic apps (added advantage)
- Experience in Cloud Scale Analytics Platform, Federred Mesh Strategy for Lakehouse.
- Good exposure in data layers – Gold layer, Silver layer & Bronze layer.
- Design, develop, and maintain data pipelines and workflows to ingest, process, and deliver data efficiently and reliably.
- Implement workflow automation solutions to streamline data processing, transformation, and integration tasks.
- Collaborate with data stakeholders to gather and understand requirements for data workflows and automation processes.
- Ensure data governance standards are met by implementing automation for data quality checks, data lineage, and metadata management.
- Monitor and optimize data workflows for performance, scalability, and reliability.
- Troubleshoot and resolve data pipeline issues promptly to minimize downtime and data loss.
- Collaborate with data architects and data scientists to ensure data infrastructure meets analytical needs.
- Develop and maintain documentation for data workflows, governance automation, and best practices.
- Stay current with emerging technologies and industry trends in data engineering and governance.
Azure Lead DE - Bengaluru, India - Quantzig
Description
About Quantzig:
Quantzig is a global analytics and advisory firm with offices in the US, UK, Canada, China, and India. we have assisted our clients across the globe with end-to-end advanced analytics, visual storyboarding , Machine Learning and data engineering solutions implementation for prudent decision making. We are a rapidly growing organization that is built and operated by high-performance champions
If you have what it takes to be the champion with business and functional skills to take ownership of an entire project end-to-end, help build a team with great work ethic and a drive to learn, you are the one we're looking for. The clients love us for our solutioning capability, our enthusiasm and we expect you to be a part of our growth story.
Find out more about the company:
We have developed exceptional expertise in advanced analytics and business intelligence solutions for transforming global organizations. Harness the power of our intelligent, actionable insights to solve complex problems and inspire innovation, change, and growth across your organization value chain.
What do we look for?