Data Engineer - India - avua

Description

We are seeking a talented Data Engineer to join our clients dynamic team. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines and systems to support our data-driven initiatives.

Responsibilities:

• Design, build, and maintain robust, scalable data pipelines to collect, process, and store large volumes of structured and unstructured data from various sources.
• Develop and implement efficient data models and architectures to support data analysis and reporting requirements. Optimize data storage and retrieval for performance and cost-effectiveness.
• Integrate data from multiple sources, including databases, APIs, and streaming platforms, ensuring data consistency, accuracy, and reliability.
• Implement data quality checks and validation processes to ensure data accuracy, completeness, and consistency. Troubleshoot and resolve data quality issues as they arise.
• Identify and address performance bottlenecks in data processing and storage systems. Optimize query performance and resource utilization for improved efficiency.
• Monitor data pipelines and systems for performance and reliability. Proactively identify and address issues to minimize downtime and ensure data availability.
• Document data pipelines, architectures, and processes to facilitate knowledge sharing and collaboration across teams. Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and deliver integrated solutions.
• Stay updated on emerging technologies, tools, and best practices in data engineering. Evaluate and incorporate new technologies and techniques to enhance data processing capabilities and efficiency.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer or similar role, with a strong background in designing and implementing data pipelines and systems.
• Proficiency in programming languages such as Python, Java, or Scala, with experience in data processing frameworks like Apache Spark or Apache Flink.
• Solid understanding of database systems, data modeling, and SQL query optimization.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud, including services like Amazon S3, EMR, Redshift, or Azure Data Lake.
• Familiarity with containerization and orchestration tools such as Docker and Kubernetes is a plus.
• Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues and optimize performance.
• Excellent communication and collaboration skills, with the ability to work effectively in a team environment.

Preferred Qualifications:

• Experience with big data technologies such as Hadoop, Kafka, or Hive.
• Knowledge of machine learning frameworks and techniques.
• Certification in cloud computing or data engineering (e.g., AWS Certified Big Data - Specialty, Google Professional Data Engineer).

Note: This is for one of our clients, not avua directly