- We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques.
- The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices. You will work closely with other data engineers to build solutions that drive impactful business insights.
- Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy.
- Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP.
- Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements.
- Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes.
- Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline.
- Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem.
- Education and Experience
- Bachelors or Masters degree in Computer Science, Data Engineering, Information Systems, or a related field.
- 3+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform.
- PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques.
- Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase.
- Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala).
- Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools.
- Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks.
- Scripting and Automation: Strong scripting skills in Linux.
-
Weiterhin suchen wir einen DevOps-Experten mit 7 bis 12 Jahren Erfahrung in relevanten Technologien, Konfiguration und Wartung von Kubernetes-Umgebungen einschließlich GitOps (ArgoCD), Ressourcen-Templating mit Helmcharts und Kustomize · ...
Bengaluru3 weeks ago
-
Wir suchen einen DevOps-und Data Engineering Experten mit 7-12 Jahren Erfahrung. · Kubeflow Thingsboard Apache Superset Dremio usw. · ...
Bengaluru, Karnataka3 weeks ago
-
Sviluppare pipeline dati utilizzando Python e SQL. · ...
Bengaluru, Chennai, Hyderabad2 weeks ago
-
The Amazon IN Platform Development team is looking to hire a rock star Data/BI Engineer to build for pan Amazon India businesses. · ...
Bengaluru, Karnataka1 week ago
-
Design and implement data infrastructure for analytics needs. · Interface with other teams to extract and load data from various sources using SQL and AWS technologies. · ...
Bengaluru1 week ago
-
+Job summary · A rock star Data/BI Engineer to build for pan Amazon India businesses. · +Responsibilities · Design, implement and support an data infrastructure for analytics needs of large organization,Interface with other technology teams to extract, transform, and load data fr ...
Bengaluru Full time2 weeks ago
-
Amazon IN Platform Development team is looking to hire a rock star Data/BI Engineer to build for pan Amazon India businesses. · ...
Bengaluru2 weeks ago
-
We are looking to hire a rock star Data/BI Engineer to build for pan Amazon India businesses. · The BI Engineer will play a key role in contributing to the success of each focus area by partnering with respective business owners and leveraging data to identify areas of improvemen ...
Bengaluru, Karnataka1 week ago
-
· At ANZ, we're applying new ways technology and data can be harnessed as we work towards a common goal: to improve the financial wellbeing and sustainability of our millions of customers. · ...
Bengaluru Full time1 week ago
-
We invite you to join our digital transformation journey. In this role, you will have the opportunity to boost your skills and become the heart of an innovative engine that is contributing to global impact and improvement. · ...
Bengaluru Full time1 week ago
-
Data Engineer role involves participating in the software development lifecycle for enterprise data projects, designing & maintaining robust ETL processes & data pipelines using modern platforms. · ...
Bengaluru, Hyderabad5 days ago
-
We are experts in data working to make it cost-effective trustworthy and understandable. We build pipelines processing billions of events daily and are stewards of canonical data warehouses and datasets delivering products for Stripe Users while embedding with teams to build thei ...
Bengaluru, Karnataka1 month ago
-
We're looking for a manager to drive the Data Engineering Solutions Team in solving high-impact,cutting-edge data problems. · Deliver cutting-edge data pipelines that scale to users' needs,focusing on reliability and efficiency. · ...
Bengaluru Full time1 month ago
-
Requirements · Job Title: Data Engineer – II / Senior Data Engineer · Job Location: Bengaluru, India · Job Summary: We are a leading Software as a Service (SaaS) company that specializes in the transformation of data in the US healthcare industry through cutting-edge Artificial I ...
Bengaluru5 days ago
-
Who we are · About Stripe · Stripe is a financial infrastructure platform for businesses.Millions of companies—at the world's largest enterprises to the most ambitious startups—use Stripe to accept payments, grow their revenue, and accelerate new business opportunities. · ...
Bengaluru1 month ago
-
Govern data acquisition and ingestion architecture for enterprise-wide data & analytics integration. · Establish reusable, scalable, and governed interfaces to all source systems across business domains. · ...
Bengaluru1 month ago
-
Education: Bachelors Degree in Computer Science, Information Technology, or a related engineering field · Certification: Microsoft Certified Data Engineer Associate or Fabric Certified Associate · Job Summary: · We are looking for a highly motivated and technically proficient Dat ...
Bengaluru / Bangalore $1,000,000 - $1,200,000 (USD) per year Full time4 days ago
-
We are looking for a skilled and experienced Data Engineer with over 5 years of experience in data engineering and data migration projects. · ...
Bengaluru1 month ago
-
Data engineer, data analyst for 10+ years with experience in cloud platforms and statistical insight. · ...
Bengaluru, Mumbai, Pune1 week ago
-
Data Engineer/ Lead Data Engineer will define and drive enterprise data engineering strategy next-generation unified analytics foundation spanning Digital, Stores, and Marketplace channels. · Define target-state data architecture for unified analytics platform using Databricks, S ...
Bengaluru, Karnataka ₹900,000 - ₹2,500,000 (INR) per year5 days ago
-
We are seeking an experienced Senior Data Engineer with a strong background in Azure-based data engineering and big data technologies to join our growing team. · Design, develop, and maintain robust scalable efficient data pipelines ETL processes on Azure cloud platform. · Build ...
Bengaluru3 weeks ago
Data Engineer - Bengaluru / Bangalore - Virtusa
Description
Responsibilities
Technical Skills
-
Data Engineer
Only for registered members Bengaluru
-
Data Engineer
Only for registered members Bengaluru, Karnataka
-
Data Engineer
Only for registered members Bengaluru, Chennai, Hyderabad
-
Data Engineer, IN Data Engineering
Only for registered members Bengaluru, Karnataka
-
Data Engineer, IN Data Engineering
Only for registered members Bengaluru
-
Data Engineer, IN Data Engineering
Full time Only for registered members Bengaluru
-
Data Engineer, IN Data Engineering
Only for registered members Bengaluru
-
Data Engineer, IN Data Engineering
Only for registered members Bengaluru, Karnataka
-
Data Engineer Data Engineer
Full time Only for registered members Bengaluru
-
Data Engineer for Data
Full time Only for registered members Bengaluru
-
Data Engineer, senior data engineer
Only for registered members Bengaluru, Hyderabad
-
Data Engineer, Data Engineering Solutions
Only for registered members Bengaluru, Karnataka
-
Data Engineer, Data Engineering Solutions
Full time Only for registered members Bengaluru
-
Data Engineer/Senior Data Engineer
Only for registered members Bengaluru
-
Data Engineer, Data Engineering Solutions
Only for registered members Bengaluru
-
IT engineer Data Engineering
Only for registered members Bengaluru
-
Data Engineer
Full time confidential- Bengaluru / Bangalore
-
Data Engineer
Only for registered members Bengaluru
-
Data Engineer
Only for registered members Bengaluru, Mumbai, Pune
-
Data Engineer
Only for registered members Bengaluru, Karnataka
-
Data Engineer
Only for registered members Bengaluru