Bishwambar Sahoo

Highly skilled Hadoop Solutions Engineer with extensive knowledge on multiple Hadoop distributions

(Apache Hadoop, CDH, HDP, CDP).

Seven years of experience in Hadoop for Big Data and 3 years in AWS cloud Technology.

Extensive experience in working on customer facing roles and worked with varied kind of different customers.

Previously Worked on other Hadoop roles ie as Administrator, and as R&D engineer CDH, HDP and CDP capabilities. 

May 2021- till now, Kyndryl, Noida, India 

Cloudera Data Engineer(Junior Architechcture)

• Attend client call to understand requirement and provide the solution.
• Building the Hadoop cluster as per client requirement (HDP, Apache Open Source, CDH & CDP)
• Worked on all the client use cases, reviewing and formatting SOW document.
• Documenting the client use cases as per Data Engineering aspects.
• Migrating Kafka, Hive, HDFS, Hue from CDH/HDP/Apache Hadoop to CDP.
• Exploring and preparing the new features of eco system.
• Working on AWS Big Data and Database migration tools and given solution as per client requirements.
• Working on new use case and given demo to client/management team.
• Explored and worked on spark migration and published refactoring guide.
• Working on relevant Big Data tool like (spark/hive workload migration work) and compare between CDH, HDP and CDP.
• Migrated data from CDH/HDP/Apache Hadoop to CDP cluster.
• Good understanding to implement NIFI, NIFI registry, Kafka, Streaming replication manager/Streams messing manager.
• Good exposure to setup apache atlas for data lineage.
• Individual player to setup/upgrade any distribution cluster like apache open Hadoop/CDH/HDP/CDP cluster.
• Migrate kafka topics/data from CDH/HDP/Apache Hadoop to CDP cluster.
• Migrate workflows (Nifi/hue) from CDH/HDP to cdp cluster.
• Good knowledge on AWS Big data tools (Glue, Kinesis, Athena, Redshift, EMR, Quicksight) and working on use cases for data migration using mentioned tools.
• Experience in working on version controller tools like Github.
• Hands on experience in configuration of Network architecture on aws with VPC Internet gateway, NAT & Route table.
• Enable AWS site to site VPN access.
• Implementing VPC peering and Transit gateway for some POC.
• In depth knowledge of AWS cloud service like compute, network, storage &Identity  & access management.
• Extensively work on CI/CD pipeline for code deployment EC2 instances by using (code commit, code build, code deploy and pipeline).
• Extensively work on CI/CD pipeline for code deployment ECS Faregate by using (code commit, code build, code deploy and pipeline).
• Deployed ECS cluster 
• Hosting static website on S3 using Route 53.
• Hands experience to perform patching on aws ec2 instance using system manager.
• Integrate windows active directory with AWS directory service.
• Enable SSO for user management.
• Hands on experience on AWS Glue, AWS Glue studio,Notebook, Quicksight, Athena & Redshift.
• Good exposure to migrate homogeneous/heterogenous database migration to AWS RDS using DMS(Data Migration service).
• Good knowledge of other AWS services in AWS like cloudfront, Glacier, Cloud trail and Simple Notification Service(SNS)
• Good understanding to implement portfolio to distribute the product to end users.

July 2019- April 2021, Metlife, Noida, India 

Assistant Manager (BigData)

Role &Responsibilities:

• Capacity planning and cluster management.
• Setup Mysql galera cluster.
• Setup cluster in SDLC Environment(DEV,QA and Prod).
• Manage resources of cluster in best manner.
• HDP and Ambari upgraded as per application team requirement.
• Communicated all issues and participated in weekly strategy meetings.
• Integrated AD with required components like ambari,ranger& zeppelin.
• Setup the required services (kafka/hive/hbaseetc) in secure cluster.
• Inolved with application to work on hive view and hive masking.
• Setup kerberos and enable kerberos from ambari.
• Create onboarding script to onboard AD users/group in dev/qa/prod.
• Created cluster checkout script to run sample jobs for(MR,hive,hdfs,spark,kafkaetc)
• Setup zeppelin and integrate with AD.
• Upgraded HDP and ambari from 3.1.0 to 3.1.5 in dev/qa/prod.
• Configured spark in hdp 3.1 to access hive catalog.
• Configure TEZ UI to view application ID to view hive query and status.
• Enabled DR in between DR and Prod cluster.
• Enabled communication in between DR and Prod.
• Created script to perform distcp, capture the logs of application id and share the validation output.
• Helping application team to resolve their hive/spark/hbase/kafka jobs.
• Enabled HA for HMS,HS2,NM and RM.
• Involving with application team to work on demo session.
• Working on all instant issue and resolve in timely manner.
• Involved to work with application team to write their data into hive from spark shell in hdp 3.1
• On boarded the AD users and group to access haadoop cluster.
• Installed Kerberos and enabled Kerberos in dev/qa/prod cluster and helped user to use keytab and principal.

Aug 2018- June 2019,Wipro Limited, Greater Noida, India 

Tech Lead(BigData)

Role &Responsibilities:

• Worked collaboratively to manage build outs of large data clusters.
• Helped design big data clusters and administered them.
• Worked both independently and as an integral part of the development team.
• Communicated all issues and participated in weekly strategy meetings.
• Provisioning, installing, configuring, monitoring, and maintaining HDFS, Yarn, Flume, Sqoop, Oozie, HBase, Pig, Hive, Ambari,HortonWorks HDP2.5, Cloudera Manager(CDH5/6). 
• Involved in Hadoop Cluster environment administration that includes adding and removing cluster nodes, cluster capacity planning, cluster Monitoring, Troubleshooting 
• Performed & validate the hadoop cluster with the help of benchmarking tool (TeraGen/TesaSort/TeraValidate), TestDFSIO, Nnbench, Mrbench, Hibench. 
• Ongoing administration of Hadoop infrastructure and managing BigData operations 24/7
• Environment setup for new projects in Taco/PROD/DEV/UAT cluster.
• Working on real time issues.
• Migrating solr and perquisites steps before upgrading CDH.
• Upgrade(CDH and Cloudera Manager)on prod/dev/uat/taco environment.
• Set up new ecocystem like kafka, apache kudu and nifi etc.
• Transfer RDBMS to hive using sqoop.
• Implemented for Real-time processing with Apache Kafka/Nifi.
• Using Nifi to transfer the data on real time from RDBMS to Hive using nifi.
• Set up oozie shell action to use spark 2.3 and spakr 2.4.
• Replacing new spark version with old one.
• Implemented YARN with dynamic resource pools
• Enabled high availability for HDFS, YARN, Hue, Hive, Impala and OOZIE services with LB/VIP.
• Added edges nodes to the cluster to use by app team.
• Decommission/re-commission nodes of cluster.
• Performance tuning of Map-Reduce jobs, hive/impala queries
• Install and configure the ranger and create the hdfs& user policy. 
• Installation & configure the Kerberos and authentication all the local user by using the Kerberos 
• Keytab deployment on all the nodes of cluster.
• Installation & configure the oozie and created the workflow 
•  Configure & install the spark 
•  Implemented the ansible installation & configure the playbooks for apache, user management , package management etc
•  Implemented ambari blueprint for the hadoop cluster Provisioning 
• Set up new cluster as per business requirement.
• Perform cluster check out for new cluster and newly added data nodes.
• Deploy High availability for Name node and resource manager and fail over validation.
• Set up HA for (Oozie,HS2,HMS,Sentry etc)
• Sentry implementation and sentry High Availability and database management using sentry roles.
• Set up Keberos , create principal and keytabs.
• Integrate Kerberos with cloudera manager.
• Manage principal and keytabs for each users/components.
• Troubleshoot Kerberos issue.
• Hbase implementation and name space management.
• Implementing new components like kudu and testing the feature of tool.
• Compare the features between impala,hive and kudu.
• TLS/SSL implementation for each components of cluster using cloudera manager.
• Testing oozie sample job, troubleshoot real time jobs and oozie HA.
• Quota management and create yarn queue as per special request of application team.
• Attend webex call with clouder team if any escalation require.

Dec’16- July 2018, InceDo Technologies Pvt.Ltd. Gurgaon

Senior System Administrator (Bigdata Hadoop)

Role &Responsibilities:

• Monitor all the critical and minor alerts of clusters.
• Expert level knowledge of Cloudera Hadoop components such as HDFS, Sentry, HBase, Hue, Spark, Hive, YARN, ZooKeeper and HA.
• Cluster maintenance as well as creation and removal of nodes using tools like Cloudera Manager Enterprise.
• Perform cluster check out for new cluster and newly added data nodes.
• Set up High Availability for Namenode.
• Troubleshoot real time Name node issues.
• Resource Manager High Availability.
• Create dedicated pool as per the application team requirement.
• Hive Server2 and Hive Metastore High Availability.
• Created schema and give the permission to use by application team.
• Check hive health issue and real time issue.
• Set up HS2 load balancer and check out load usage of HS2.
• Set up Sentry and High Availability.
• Create roles and assign to the group on the basis of application need.
• Configure HA and its high availability.
• Implementing new components like kudu and testing the feature of tool.
• Compare the features between impala,hive and kudu.
• TLS/SSL implementation for each components of cluster using cloudera manager.
• Testing oozie sample job, troubleshoot real time jobs and oozie HA.
• Quota management and create yarn queue as per special request of application team.
• Set up Keberos , create principal and keytabs.
• Integrate Kerberos with cloudera manager.
• Manage principal and keytabs for each users/components.
• Troubleshoot Kerberos issue.
• Performance tuning of Hadoop clusters and Hadoop Map Reduce routines.
• Monitor Hadoop cluster connectivity and security.
• Attend webex call with clouder team if any escalation require.

Dec’15 - Dec16  O-Zone Network Pvt. Ltd. Gurgaon 

Hadoop and Linux Administrator

Role &Responsibilities:

• Implemented and validate the Hortonworks HDP 2.3 Ambari 2.1 on dell SEREVR.
• Responsible for implementation and ongoing administration of Hadoop infrastructure.
• Aligning with the systems engineering team to propose and deploy new hardware and software environments required for Hadoop and to expand existing environments.
• Working with data delivery teams to setup new Hadoop users. This job includes setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig and MapReduce access for the new users.
• Cluster maintenance as well as creation and removal of nodes using tools like Cloudera Manager Enterprise, Dell Open Manage and other tools.
• Performance tuning of Hadoop clusters and Hadoop MapReduce routines.
• Screen Hadoop cluster job performances and capacity planning
• Monitor Hadoop cluster connectivity and security
• Manage and review Hadoop log files.
• File system management and monitoring.
• HDFS support and maintenance.
• Diligently teaming with the infrastructure, network, database, application and business intelligence teams to guarantee high data quality and availability.
• Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades when required.
• Point of Contact for Vendor escalation
• The most essential requirements are: They should be able to deploy Hadoop cluster, add and remove nodes, keep track of jobs, monitor critical parts of the cluster, configure name-node high availability, schedule and configure it and take backups.
• Good knowledge of Linux as Hadoop runs on Linux.
• Hands on experience with the Hadoop stack; MapReduce, Sqoop, Pig, Hive, Flume) 
• Hands on experience in Hive
• End-to-end performance tuning of Hadoop clusters and Hadoop Map/Reduce routines against very large data sets
• Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
• Experience with integration of data from multiple data sources

March’11 - Nov’2015, iYogi Technical Services Pvt. Ltd. Gurgaon 

Floor Tech Lead

Role &Responsibilities:

• Managing user account creation, deletion, and configuration.
• Configuration and Troubleshooting of Ms Outlook & Outlook Express.
• Troubleshooting & installation of local area network
• Installation, Configure and Maintain Computer hardware, Wireless network, Application Software and Other Peripherals.
• Installation, Configure and Troubleshooting Application Software.
•  Installation, deployment, support and maintenance of Servers (RHEL & Windows)
• User/Group Administration
• NIS
• yum server
• Internal SAMBA – File Server
• Manage Log Files - Syslog (Linux) management with Log rotate utility
• Configure NTP servers/clients
• Managing Permissions of files/folders; grant Sudo access; Advanced permissions - ACLs
• Backups of databases & restoration as per requirement
• SSH , Telnet, FTP 
• Performance Tuning / Monitoring of Servers
• Kickstart Installation for complete hands-free installation of workstations, with custom profiles, begin/finish scripts, custom package suites.
• Backing-up Volumes & Files using Snapshots, cpio, dump, tar etc.
• File System Management with third party tools like GParted
• Working with LVMs
• Basic SQUID administration
• Mail services – Sendmail, POP3 mail, SSL Encapsulated IMAP Access (IMAPS)
• YUM client / server (multiple) configuration
• Troubleshooting networks, Linux systems, and applications to identify and correct malfunctions and other operational problems

March’08 to Feb'2011, iYogi Technical Service Pvt.Ltd. Gurgaon

Technical Associate.

Role &Responsibilities:

• Troubleshooting and installation of  Windows 98, Windows XP, Windows Vista (32 Bit & 64 Bit),Windows 7 (32 Bit & 64 Bit), Windows 8.
• Troubleshooting and Configuration of Email Client Like- Outlook Express, MS Outlook 2000, 2003 & 2010, Incredimail, Mozilla Thunderbird.
• Troubleshooting, Configuration and installation of all type wired and wireless printer like- HP, Cannon, Dell, Lexmark, Kodak, Brother etc.
• Troubleshooting and installation of all type Multimedia Player like- iTunes, Real Player, Windows Media Player, etc.
• Troubleshooting of Browser like- Internet explorer, Mozilla Firefox, Google Chrome etc.
• Troubleshooting and Installation of Antivirus for all version like- Microsoft Security Essentials, McAfee, Kaspersky, AVG, Norton, CA etc.
• Troubleshooting and installation of MS Office packages like 2000, 2003, 2007, 2010 and 2013.
• Upgrade and troubleshoot of windows media player.

Update and troubleshoot of windows update

• BSc (2003-2006) Utkal University.
• 12th Sc (2001-03) Council of Higher Secondary Education, Orissa.
• 10th (2001) Board of Secondary Education, Orissa.