Data Engineer - Bengaluru / Bangalore - Virtusa

    Virtusa
    Virtusa Bengaluru / Bangalore

    4 hours ago

    Full time ₹900,000 - ₹2,500,000 (INR) per year *
    Description
    • We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques.
    • The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices. You will work closely with other data engineers to build solutions that drive impactful business insights.

    Responsibilities

    • Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy.
    • Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP.
    • Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements.
    • Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes.
    • Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline.
    • Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem.
    • Education and Experience
    • Bachelors or Masters degree in Computer Science, Data Engineering, Information Systems, or a related field.
    • 3+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform.

    Technical Skills

    • PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques.
    • Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase.
    • Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala).
    • Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools.
    • Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks.
    • Scripting and Automation: Strong scripting skills in Linux.
    * This salary range is an estimation made by beBee
  • Work in company

    Data Engineer

    Only for registered members

    Weiterhin suchen wir einen DevOps-Experten mit 7 bis 12 Jahren Erfahrung in relevanten Technologien, Konfiguration und Wartung von Kubernetes-Umgebungen einschließlich GitOps (ArgoCD), Ressourcen-Templating mit Helmcharts und Kustomize · ...

    Bengaluru

    3 weeks ago

  • Work in company

    Data Engineer

    Only for registered members

    Wir suchen einen DevOps-und Data Engineering Experten mit 7-12 Jahren Erfahrung. · Kubeflow Thingsboard Apache Superset Dremio usw. · ...

    Bengaluru, Karnataka

    3 weeks ago

  • Work in company

    Data Engineer

    Only for registered members

    Sviluppare pipeline dati utilizzando Python e SQL. · ...

    Bengaluru, Chennai, Hyderabad

    2 weeks ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    The Amazon IN Platform Development team is looking to hire a rock star Data/BI Engineer to build for pan Amazon India businesses. · ...

    Bengaluru, Karnataka

    1 week ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    Design and implement data infrastructure for analytics needs. · Interface with other teams to extract and load data from various sources using SQL and AWS technologies. · ...

    Bengaluru

    1 week ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    +Job summary · A rock star Data/BI Engineer to build for pan Amazon India businesses. · +Responsibilities · Design, implement and support an data infrastructure for analytics needs of large organization,Interface with other technology teams to extract, transform, and load data fr ...

    Bengaluru Full time

    2 weeks ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    Amazon IN Platform Development team is looking to hire a rock star Data/BI Engineer to build for pan Amazon India businesses. · ...

    Bengaluru

    2 weeks ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    We are looking to hire a rock star Data/BI Engineer to build for pan Amazon India businesses. · The BI Engineer will play a key role in contributing to the success of each focus area by partnering with respective business owners and leveraging data to identify areas of improvemen ...

    Bengaluru, Karnataka

    1 week ago

  • Work in company

    Data Engineer Data Engineer

    Only for registered members

    · At ANZ, we're applying new ways technology and data can be harnessed as we work towards a common goal: to improve the financial wellbeing and sustainability of our millions of customers. · ...

    Bengaluru Full time

    1 week ago

  • Work in company

    Data Engineer for Data

    Only for registered members

    We invite you to join our digital transformation journey. In this role, you will have the opportunity to boost your skills and become the heart of an innovative engine that is contributing to global impact and improvement. · ...

    Bengaluru Full time

    1 week ago

  • Work in company

    Data Engineer, senior data engineer

    Only for registered members

    Data Engineer role involves participating in the software development lifecycle for enterprise data projects, designing & maintaining robust ETL processes & data pipelines using modern platforms. · ...

    Bengaluru, Hyderabad

    5 days ago

  • Work in company

    Data Engineer, Data Engineering Solutions

    Only for registered members

    We are experts in data working to make it cost-effective trustworthy and understandable. We build pipelines processing billions of events daily and are stewards of canonical data warehouses and datasets delivering products for Stripe Users while embedding with teams to build thei ...

    Bengaluru, Karnataka

    1 month ago

  • Work in company

    Data Engineer, Data Engineering Solutions

    Only for registered members

    We're looking for a manager to drive the Data Engineering Solutions Team in solving high-impact,cutting-edge data problems. · Deliver cutting-edge data pipelines that scale to users' needs,focusing on reliability and efficiency. · ...

    Bengaluru Full time

    1 month ago

  • Work in company

    Data Engineer/Senior Data Engineer

    Only for registered members

    Requirements · Job Title: Data Engineer – II / Senior Data Engineer · Job Location: Bengaluru, India · Job Summary: We are a leading Software as a Service (SaaS) company that specializes in the transformation of data in the US healthcare industry through cutting-edge Artificial I ...

    Bengaluru

    5 days ago

  • Work in company

    Data Engineer, Data Engineering Solutions

    Only for registered members

    Who we are · About Stripe · Stripe is a financial infrastructure platform for businesses.Millions of companies—at the world's largest enterprises to the most ambitious startups—use Stripe to accept payments, grow their revenue, and accelerate new business opportunities. · ...

    Bengaluru

    1 month ago

  • Work in company

    IT engineer Data Engineering

    Only for registered members

    Govern data acquisition and ingestion architecture for enterprise-wide data & analytics integration. · Establish reusable, scalable, and governed interfaces to all source systems across business domains. · ...

    Bengaluru

    1 month ago

  • Work in company

    Data Engineer

    confidential

    Education: Bachelors Degree in Computer Science, Information Technology, or a related engineering field · Certification: Microsoft Certified Data Engineer Associate or Fabric Certified Associate · Job Summary: · We are looking for a highly motivated and technically proficient Dat ...

    Bengaluru / Bangalore $1,000,000 - $1,200,000 (USD) per year Full time

    4 days ago

  • Work in company

    Data Engineer

    Only for registered members

    We are looking for a skilled and experienced Data Engineer with over 5 years of experience in data engineering and data migration projects. · ...

    Bengaluru

    1 month ago

  • Work in company

    Data Engineer

    Only for registered members

    Data engineer, data analyst for 10+ years with experience in cloud platforms and statistical insight. · ...

    Bengaluru, Mumbai, Pune

    1 week ago

  • Work in company

    Data Engineer

    Only for registered members

    Data Engineer/ Lead Data Engineer will define and drive enterprise data engineering strategy next-generation unified analytics foundation spanning Digital, Stores, and Marketplace channels. · Define target-state data architecture for unified analytics platform using Databricks, S ...

    Bengaluru, Karnataka ₹900,000 - ₹2,500,000 (INR) per year

    5 days ago

  • Work in company

    Data Engineer

    Only for registered members

    We are seeking an experienced Senior Data Engineer with a strong background in Azure-based data engineering and big data technologies to join our growing team. · Design, develop, and maintain robust scalable efficient data pipelines ETL processes on Azure cloud platform. · Build ...

    Bengaluru

    3 weeks ago

Jobs
>
Data engineer
>
Jobs for Data engineer in Bengaluru