Big Data Engineer - Bangalore, India - Talentxo
Description
Job Description :
Analyze and optimize Hive queries :
Identify bottlenecks and opportunities for improvement in Hive scripts.
Apply best practices for query optimization, data partitioning, and materialized views.
Tame the TPC-H & TPC-DS benchmarks :
Deeply understand the TPC-H and TPC-DS benchmarks and leverage them to evaluate and compare Presto performance at different scales (1K & 1TB / 10K & 10TB).
Prestissimo power user :
Implement and configure Prestissimo for optimized query execution in our data environment.
Fine-tune Prestissimo configurations to maximize performance and resource utilization.
Hive whisperer :
Master the art of sizing and tuning Hive configurations for optimal performance.
Manage RAM allocation, adjust Hive settings, and implement best practices for data storage and access.
Automation Architect :
Develop and maintain automation scripts to reproduce your Hive and Presto configurations and optimization techniques.
Ensure repeatability and ease of deployment for future changes. (Bonus points for MinIO knowledge)
Collaborate and communicate :
Work closely with data analysts, scientists, and engineers to understand data processing needs and translate them into efficient execution plans.
Share your knowledge and findings through clear and concise communication.
Qualifications :
- 10+ years of experience as a Big Data Engineer or related role.
- Strong expertise in HiveQL and PrestoSQL.
- Experience with TPC-H/TPC-DS benchmarks and performance optimization techniques.
- Understanding of distributed file systems and data storage optimizations.
- Scripting skills in Python or similar languages for automation tasks. (Familiarity with MinIO is a plus)
- Excellent analytical and problem-solving skills.
- Strong communication and collaboration skills.