Data Engineer - Bangalore/Hyderabad/Chennai/Pune, India - Nilasu consulting

Description

• Design, develop, and deploy productionready data pipelines using Databricks, Kafka, and Snowflake.
• Collaborate with data analysts, software engineers, and stakeholders to understand data requirements and develop effective data migration strategies, API configurations, and backend architecture.
• Extract, transform, and load (ETL) data from various sources, ensuring data integrity and accuracy through data mapping, transformation, and validation procedures.
• Implement data quality checks, data profiling, and data cleansing techniques to maintain high data consistency.
• Partner with the data governance team to adhere to data privacy and compliance regulations.
• Leverage cloud platforms (AWS, Azure, Google Cloud) for data management solutions.
• Work effectively with various database management systems (SQL Server, PostgreSQL, DynamoDB, MongoDB).
• Design and build scalable realtime data processing pipelines using Apache Kafka.
• Utilize APIs and microservices to ensure seamless data connectivity across applications and 5+ years of experience leading data migration efforts, preferably in the healthcare industry.
• Expertise in Databricks, Kafka, and Snowflake technologies.
• Familiarity with Azure Data Factory (ADF) (willing to be trained is a plus).
• Experience with data governance practices and compliance regulations.
• Strong understanding of ETL processes and data warehousing concepts.
• Proficiency in data quality checks, profiling, and cleansing techniques.
• Familiarity with cloudbased data solutions (AWS, Azure, Google Cloud) is a plus.
• Knowledge of various database management systems (SQL Server, PostgreSQL, DynamoDB, MongoDB).
• Understanding of data analytics and visualization tools (e.g., Tableau, Power BI) is a plus.
• Experience with Git version control (GitHub).
• Ability to design complex relational databases and manage them efficiently.
• Strong understanding of APIs and microservices architecture.