BIG DATA ENGINEER - Bengaluru, India - Coforge

    Coforge
    Coforge Bengaluru, India

    2 weeks ago

    coforge background
    Description
    Job Description


    Job description Role and responsibilities : -Creating Project Technical Documentation - Designing Solution architecture, and work on Data Ingestion, Preparation and Transformation.

    Debugging the production failures and identifying the solution.- Developing efficient frameworks for development and testing using (AWS Dynamo DB, EKS, Kafka) to enable seamless data ingestion process on to the Hadoop platform.- Enabling Data Governance and Data Discovery on Hadoop Platform- Building data processing framework using Spark, HQL- Exposure of Security Framework with Kerberos, Ranger, Atlas- Exposure of Data Pipeline Automation using DevOps tools- Exposure of Job Monitoring framework along validations automation- Exposure of handling structured, Un Structured and Streaming data Technical skills requirements : - Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred)- Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR- Hands-on experience of programming language like Scala with Spark.- Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases- Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred- Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie- Hands on working experience of data processing at scale with event driven systems, message queues (Kafka FlinkSpark Streaming)- Hands on working Experience with AWS Services like EMR, Kinesis, S3, CloudFormation, Glue, API Gateway, Lake Foundation- Hands on working Experience with AWS Athena - Data Warehouse exposure on Apache Nifi, Apache Airflow, Kylo - Operationalization of ML models on AWS (e.g.

    deployment, scheduling, model monitoring etc.)- Feature Engineering Data Processing to be used for Model development- Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)

    - Experience building data pipelines for structured unstructured, real-time batch, events synchronous asynchronous using MQ, Kafka, Steam processing- Hands-on working experience in analysing source system data and data flows, working with structured and unstructured data- Must be very strong in writing SQL queries- Strengthen the Data engineering team with Big Data solutions- Strong technical, analytical, and problem-solving skillsPosted On


    Job description Role and responsibilities : -Creating Project Technical Documentation - Designing Solution architecture, and work on Data Ingestion, Preparation and Transformation.

    Debugging the production failures and identifying the solution.- Developing efficient frameworks for development and testing using (AWS Dynamo DB, EKS, Kafka) to enable seamless data ingestion process on to the Hadoop platform.- Enabling Data Governance and Data Discovery on Hadoop Platform- Building data processing framework using Spark, HQL- Exposure of Security Framework with Kerberos, Ranger, Atlas- Exposure of Data Pipeline Automation using DevOps tools- Exposure of Job Monitoring framework along validations automation- Exposure of handling structured, Un Structured and Streaming data Technical skills requirements : - Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred)- Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR- Hands-on experience of programming language like Scala with Spark.- Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases- Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred- Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie- Hands on working experience of data processing at scale with event driven systems, message queues (Kafka FlinkSpark Streaming)- Hands on working Experience with AWS Services like EMR, Kinesis, S3, CloudFormation, Glue, API Gateway, Lake Foundation- Hands on working Experience with AWS Athena - Data Warehouse exposure on Apache Nifi, Apache Airflow, Kylo - Operationalization of ML models on AWS (e.g.

    deployment, scheduling, model monitoring etc.)- Feature Engineering Data Processing to be used for Model development- Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)

    - Experience building data pipelines for structured unstructured, real-time batch, events synchronous asynchronous using MQ, Kafka, Steam processing- Hands-on working experience in analysing source system data and data flows, working with structured and unstructured data- Must be very strong in writing SQL queries- Strengthen the Data engineering team with Big Data solutions- Strong technical, analytical, and problem-solving skills

    Skills Required

    AWS EMRPosted On AWS EMR Location

    Bangalore / Greater Noida / Pune / HyderabadPosted On Bangalore / Greater Noida / Pune / Hyderabad