- Design, develop, and maintain data pipelines using DataBricks and PySpark to process and manipulate large scale datasets.
- Proven experience in optimizing Apache Spark batch processing workflows.
- Extensive experience in building and maintaining streaming data pipelines.
- Optimize and finetune existing DataBricks jobs and PySpark scripts for enhanced performance and reliability.
- Troubleshoot issues related to data pipelines, identify bottlenecks, and implement effective solutions.
- Implement best practices for data governance, security, and compliance within DataBricks environments.
- Work closely with Data Scientists and Analysts to support their data requirements and enable efficient access to relevant datasets.
- Stay updated with industry trends and advancements in DataBricks and PySpark technologies to propose and implement innovative solutions.
- Demonstrated expertise in optimizing systems for lowlatency and highthroughput performance.
- Proficiency in using Spark SQL and DataFrame API for dynamic data transformations.
- Experience with using programming languages such as Python or Scala to implement advanced filtering logic in Databricks notebooks or scripts.
- Familiarity with the principles of distributed systems and their application in message broking.
- Collaborate with cross functional teams to gather requirements, understand data needs, and implement scalable solutions.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- 4 to 5 years of proven experience as a Data Engineer with a strong emphasis on DataBricks.
- Proficiency in PySpark and extensive hands on experience in building and optimizing data pipelines using DataBricks.
- Solid understanding of different components within DataBricks such as clusters, notebooks, jobs, and libraries.
- Strong knowledge of SQL, data modeling, and ETL processes.
- Ability to analyzxcellent communication skills with the ability to collaborate with cross functional teams.
-
Data Engineer
2 days ago
Uplers New Delhi, IndiaUplers - Uplers, originated as an outsourcing firm in 2012, evolved to offer innovative solutions beyond expertise.Uplers is a pioneering talent matchmaking platform that bridges the gap between global companies and top-tier Indian professionals, redefining remote hiring through ...
-
Data Engineer
3 days ago
Muoro Delhi, IndiaResponsibilities : · - Design, develop, and maintain robust data pipelines using Apache Spark, Python, and other relevant technologies. · - Implement data ingestion processes from various sources, ensuring data quality and integrity. · - Optimize and tune data pipelines for perfo ...
-
Data Engineer
1 day ago
Macropace technologies Delhi, IndiaRole : Data Engineer · Job Description : · We are seeking a highly skilled Data Engineer to join our team. · The ideal candidate will have a strong foundation in SQL and Snowflake, with a passion for transforming and optimizing data pipelines. · As a Data Engineer, you will p ...
-
Data Engineer
1 day ago
Huquo Consulting Delhi, IndiaPosition : Data Engineer · Location : Gurgaon · Experience : Min 4+ years · Role & Responsibilities Overview : · - A seasoned data team member with a good grasp of both business and technical aspects. · Good familiarity with data stack · - Interact with the other members of the d ...
-
Data Engineer
3 days ago
Resources valley Delhi, IndiaJob Description : · Ownership of Data Engineering solution development, including ingestion, transformation, loading, modelling, curation, profiling, migration, and validation of data, also maintaining security, access control, compliance, and SLAs · - Design and create data mode ...
-
Data Engineer
1 week ago
Terminus Delhi, IndiaRole · We are looking for a Data Engineer to join our team who will be involved data management and transformation at scale and optimize to prepare the data pipe for the AI /Analytical technology stack. The ideal candidate is an experienced data pipeline builder and data wrangle ...
-
Data Engineer
1 day ago
SKYGATE CONSULTING Delhi, IndiaData Engineer (Snowflake) · Job Description : · As part of the team, you will be responsible for building and running the data pipelines and services that are required to support business functions/reports/dashboard.. We are heavily dependent on BigQuery/Snowflake, Airflow, Stitc ...
-
Data Engineer
6 days ago
Blackstraw Delhi, IndiaData Engineer · Company Name: · Office Location: Delhi (Work from Office) · Job Type: Full-time · Experience: 4+ Years · Job Description: We are looking for Data Engineers to join our team. You will use various methods to transform raw data into useful data systems. For exa ...
-
Data Engineer
1 week ago
SIENT TECHOLOGIES PVT LTD New Delhi, IndiaCompany Overview: · SIENT Technologies is a pioneering technology firm at the forefront of innovation, dedicated to revolutionizing industries through the power of advanced solutions. Committed to excellence, we specialize in creating transformative technology that enhances effic ...
-
Data Engineer
2 days ago
Reliable Group Delhi, IndiaPOSITION DESCRIPTION : · WHAT YOU'LL DO : · - Architect, Design, document & implement the data pipelines to feed data models for subsequent consumption in Snowflake using dbt, airflow. · - Ensure correctness and completeness of the data being transformed via engineering pipelines ...
-
Data Engineer
3 days ago
Triumph Ladders Pvt. Ltd. Delhi Division, IndiaTitle - Data Engineer · Exp years · Location - Delhi · Notice period days · Responsibilities : · - Collaborate with Data Scientists to test and scale new algorithms through pilots and later industrialize · the solutions at scale to the comprehensive fashion network of the Group ...
-
Data Engineer
2 days ago
Vbeyond Delhi NCR/Mumbai/Noida, India permanentJOB SUMMARY : · We are seeking a skilled and experienced Data Engineer with strong developments skills and solid track record in terms of implementing scalability, availability and excellence when building data pipelines and platforms for our royalties and revenue share initiati ...
-
Data Engineer
2 days ago
KARYA CONSULTANTS PRIVATE LIMITED Delhi, IndiaAs a GCP Data Engineer, you will be a key player in designing, implementing, and optimizing data solutions on the Google Cloud Platform. · We are seeking a professional with hands-on experience in data engineering and a strong understanding of GCP services. · Key Responsibilities ...
-
Data Engineer
1 day ago
TransOrg Analytics Delhi, IndiaRoles and Responsibilities : · 1. Design and implement data engineering projects. · 2. Integrate multiple data sources to create data lake/data mart Perform data ingestion and ETL processes using SQL, Scoop, Spark or Hive · 3. Knowledge of new components and various emerging tec ...
-
Data Engineer
1 day ago
INSIGHT GLOBAL SOLUTIONS Delhi, IndiaData Engineer · Must Haves : · - Bachelors Degree in science, technology, or another related field. · - 3+ years of data engineering experience. · - 3+ years of experience with Cloud platforms like GCP, Azure, Airflow. · - 4+ years of data warehouse, relational database, and SQL ...
-
Data Engineer
2 days ago
WovV Technologies Delhi, IndiaJob Description : Data Engineer · Location : Remote · Experience : 5+ years. · Durations : 6 months with a very likely extension · Mandatory Technology : DBT, SQL, Snowflake,data integration practices, and Tableau · Primary Skills : Expertise in transaction refunds and a comprehe ...
-
Data Engineer
1 week ago
RS Consultants Delhi, IndiaAbout the job : · Responsibilities : · - Build, document, test and launch new data extract, transform and load (ETL) processes for one or more core data sets. · - Collaborate with Business Intelligence team and other stakeholders for data requirements · - Collaborate with the Dat ...
-
Data Engineer
1 day ago
HS Solutions Inc Delhi, IndiaAbout the job : · Tech stack : · BigQuery , any ETL tool (Informatica, Talend, DataStage), Dataflow, Dataproc. · - 3-5 years' Experience in Data warehouse and Data lake implementation. · - 1-2 years of experience in Google Cloud Platform (especially Big Query). · - 1-2 years of w ...
-
Data Engineer
1 day ago
Vision Beyond Resources India Pvt. Ltd. Delhi, IndiaRole : Data : UK Shift Range : 7+ Requirements : · Excellent understanding of data structures with 5+ years of experience in enterprise-wide data platforms design and implementation. · - Strong knowledge of common design patterns used in Data Integrations and Warehousing. · - Pro ...
-
Data Engineer
3 days ago
USEReady Technology Pvt. Ltd. Any Location/Bangalore, India permanentRole Responsibility : · - Translate functional specifications and change requests into technical specifications. · - Translate business requirement document, functional specification, and technical specification to related coding. · - Develop efficient code with unit testing and ...
Data Engineer - Delhi, India - HARP Technologies and Services
Description
Job Location : Pune & Hyderabad (Initial 1 -2 months hybrid and then complete remote job. Client will offer accommodation(stay), food & travel exp. )
Exp range : 6+ years
Shift timings : General 9am - 6pm IST or 10am - 7pm IST
Mandatory skills : Data engineering (4+years), Databricks (3.5+ years), Pyspark (3+ years), Python (preferred) or bash, Data pipeline, Azure, ETL.
Job Type :
C2H long term (within 6 months based on your performance/cultural fit you might be onboarded by the end client as a permanent employee)
We are seeking a skilled Data Engineer with 5+ years of experience specializing in DataBricks. The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks.
As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.