Aws Hadoop Administrator - Uttar Pradesh, India - Valiance Analytics

Description

We are seeking a skilled AWS Hadoop Administrator to join our team and manage our Hadoop infrastructure on the AWS platform.

In this role, you will be responsible for deploying, configuring, monitoring, and maintaining our Hadoop clusters to ensure optimal performance, scalability, and reliability.

As an integral member of our IT operations team, you will collaborate closely with data engineers, developers, and other stakeholders to support our big data initiatives and ensure smooth operation of our data processing required:
Hands-on experience on installation of Hadoop ecosystem on AWS such as EMR,S3,EC2,IAM policy, MWAA/Airflow,Hadoop,yarn,spark, Mlflow.

Experience on Linux enviroment and bash scriptingResponsibilities: 1.

Deploy and configure Hadoop clusters on the AWS cloud platform including MWAA,MLflow and Apache-Nifi, ensuring proper integration with related AWS services such as EC2, S3, EMR, and IAM.

• Monitor Hadoop cluster performance, health, and resource utilization using monitoring tools and AWS CloudWatch, and proactively address any issues to maintain optimal performance.
• Manage Hadoop cluster capacity, scaling resources up or down as needed to accommodate changing workload demands.
• Implement and enforce security best practices for Hadoop clusters on AWS, including access control, encryption, and compliance with regulatory requirements.
• Perform regular backups and disaster recovery procedures to ensure data integrity and minimize downtime.
• Collaborate with data engineers and developers to optimize Hadoop jobs and data processing workflows for performance and efficiency.
• Troubleshoot and resolve issues related to Hadoop cluster operation, performance, and data processing.
• Stay current with advancements in Hadoop technology and AWS services, and evaluate and recommend new tools and techniques to enhance our big data infrastructure.
• Document system configurations, procedures, and best practices, and provide training and support to other team members as needed.
• Participate in on-call rotation and respond to incidents outside of regular business hours as necessary

Qualifications:
1. Bachelor's degree in computer science, engineering, or a related field, or equivalent work experience.

• Proven experience as a Hadoop administrator, preferably in an AWS environment.
• Strong understanding of Hadoop architecture, components, and ecosystem technologies such as HDFS, YARN, MapReduce, Hive, Spark, and Kafka.
• Experience deploying and managing Hadoop clusters on AWS using services such as EMR, EC2, S3, and IAM.
• Solid understanding of cloud computing principles and best practices, particularly in AWS.
• Proficiency in Linux operating systems and shell scripting.
• Strong analytical and problem-solving skills, with the ability to troubleshoot complex issues and drive them to resolution.
• Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
• AWS certification(s) such as AWS Certified Solutions Architect or AWS Certified SysOps Administrator is a plus.
• Experience with containerization technologies such as Docker and orchestration tools like Kubernetes is desirable.