- Design, Develop, and maintain ETL pipelines using Pyspark in Azure Databricks using delta tables.
- Create build from GitHub and release pipeline for Ingestion and Databricks using Azure DevOps / Harness
- Monitor the Performance of ETL Jobs, resolve any issues that arose, and improve the performance metrics as needed.
- Diagnose system performance issues related to data processing and implement the solution to address them.
- Collaborate with other teams to make sure successful integration of data pipelines into larger system architecture requirements.
- Maintain integrity and quality across all pipelines and environments.
- Bachelor's degree in Computer Science, Mathematics, Statistics, Finance, related technical field, or equivalent work experience
- Should have a robust background in Software development with experience in ingesting, transforming, and storing data from large datasets using Pyspark in Azure Databricks with robust knowledge of distributed computing concepts.
- Must have hands-on experience in designing and developing ETL Pipelines in Pyspark in Azure Databricks with robust python scripting exposure like list comprehensions, Dictionary variables, etc.
- Must have a minimum of 5 years of exposure and good proficiency in data warehousing concepts.
- Proficient in SQL and database Design concepts.
- Hands-on experience with good proficiency in Delta table and delta file operations like merge, Insert override, Partition overrides, etc.
- Hands-on Experience in CICD in Azure DevOps/Harness and ADF/Stone branch for orchestration.
- Knowledge of Azure cloud computing platform with Azure Synapse, ADLS.
- Knowledge of GitHub and build management.
- Exposure to any Informatica ETL tool is a plus.
- Passion for data and experience working within a data-driven organization
- You care about what you do, and what we doFIND your futureAXA XL, the P&C and specialty risk division of AXA, is known for solving complex risks. For mid-sized companies, multinationals, and even some inspirational individuals we don't just provide re/insurance, we reinvent it.
Assistant Scientist –Data Engineering - Bengaluru, India - AXA Group
Description
Assistant Scientist –Data EngineeringBangalore, Karnataka India & Gurgaon, Haryana-India
AXA XL recognizes data and information as critical business assets, both in terms of managing risk and enabling new business opportunities.
This data should not only be high quality, but also actionable – enabling AXA XL's executive leadership team to maximize benefits and facilitate sustained dynamic advantage.
Our Chief Data Office also known as our Innovation, Data Intelligence & Analytics team (IDA) is focused on driving innovation by optimizing how we leverage data to drive strategy and create a new business model – disrupting the insurance market.
As we develop an enterprise-wide data and digital strategy that moves us toward a greater focus on the use of data and data-driven insights, we are seeking an Assistant Scientist for our Data Engineering team.
The role will support the team's efforts towards creating, enhancing, and stabilizing the Enterprise data lake through the development of the data pipelines.
This role requires a person that is a team player and can work well with team members from other disciplines to deliver data in an efficient and strategic manner.
DISCOVERyour opportunityWhat will your essential responsibilities include?How? By combining a comprehensive and efficient capital platform, data-driven insights, leading technology, and the best talent in an agile and inclusive workspace, empowered to deliver top client service across all our lines of business − property, casualty, professional, financial lines and specialty.
With an innovative and flexible approach to risk solutions, we partner with those who move the world forward.Inclusion & Diversity