Data Engineer - Pune, India - EDGESOFT

EDGESOFT Pune, India

Found in: Talent IN 2A C2 - 4 days ago

permanent Technology / Internet

Description

Data Engineer - DBX

We are seeking a skilled Data Engineer with 4 to 5 years of experience specializing in DataBricks. The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks.

As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.

Responsibilities :

Design, develop, and maintain data pipelines using DataBricks and PySpark to process and manipulate large scale datasets.
Proven experience in optimizing Apache Spark batch processing workflows.
Extensive experience in building and maintaining streaming data pipelines.
Optimize and finetune existing DataBricks jobs and PySpark scripts for enhanced performance and reliability.
Troubleshoot issues related to data pipelines, identify bottlenecks, and implement effective solutions.
Implement best practices for data governance, security, and compliance within DataBricks environments.
Work closely with Data Scientists and Analysts to support their data requirements and enable efficient access to relevant datasets.
Stay updated with industry trends and advancements in DataBricks and PySpark technologies to propose and implement innovative solutions.
Demonstrated expertise in optimizing systems for lowlatency and highthroughput performance.
Proficiency in using Spark SQL and DataFrame API for dynamic data transformations.
Experience with using programming languages such as Python or Scala to implement advanced filtering logic in Databricks notebooks or scripts.
Familiarity with the principles of distributed systems and their application in message broking.
Collaborate with cross functional teams to gather requirements, understand data needs, and implement scalable solutions.

Requirements :

Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
5 to 10 years of proven experience as a Data Engineer with a strong emphasis on DataBricks.
Proficiency in PySpark and extensive hands on experience in building and optimizing data pipelines using DataBricks.
Solid understanding of different components within DataBricks such as clusters, notebooks, jobs, and libraries.
Strong knowledge of SQL, data modeling, and ETL processes.
Ability to analyzxcellent communication skills with the ability to collaborate with cross functional teams.
Experience working in e complex problems, propose solutions, and implement them effectively.
Ea cloud environment (AWS, Azure, or GCP) is a plus.

Preferred Skills :

Certifications or training in DataBricks or PySpark.
Familiarity with other big data technologies (Hadoop, Spark, Kafka, etc.).
Experience with version control systems (Git, SVN, etc.) and CI/CD pipelines.
Knowledge of at least one repository like GIT and software tool like JIRA

)

Data Engineer

Found in: Talent IN 2A C2 - 6 days ago

Perficient Pune, India

Job Description: Data Engineer · Experience Total: 4- 7 · Location: Pune · Qualification: · Advanced working SQL knowledge and performance tuning · Good knowledge of Relational Model, Dimensional Data Modelling, Data Warehouse · Experience working with relational databases, fo ...
Data Engineer

Found in: Appcast Linkedin IN C2 - 1 day ago

Tech Mahindra Pune, India

Title: Reltio GCP Data Engineer · Location: Pune · Experience: 6+years · NP: Immediate to 15days · Company Profile: · Tech Mahindra represents the connected world, offering innovative and customer-centric information technology experiences. We #Rise together to create sustainable ...
Data Engineer

Found in: Talent IN 2A C2 - 1 day ago

Tech Mahindra Pune, India

GCP Data Engineer – 6 years of experience in GCP and Big Query, Cloud Storage, · Strong python programming experience in using Google Cloud python client libraries, · Strong SQL and data analysis skills, · Proficient ETL/ELT experience in workflows, data modelling and data pipeli ...
Data Engineer

Found in: Talent IN 2A C2 - 5 days ago

ACS International India Pvt. Ltd. (ACSII) Pune, India

ACS-I India is looking for an 8+ years experienced Data Engineer with data modeling experience who will support a rapidly growing Business Data & Analytic Platform focused on providing data product solutions that empower our core business users to turn data into actionable insigh ...
Data Engineer

Found in: Appcast Linkedin IN C2 - 4 days ago

Sigmaways Inc Pune, India

This position is a Hybrid Role candidate should be 2 days onsite in Pune Office ( Prefer Local candidates). · Position : Data Engineer · Type : Long Term Contract Role · Location : Pune / Hybrid · Job Description: · This position is for Data warehouse group . · This position is a ...
Data Engineer

Found in: Appcast Linkedin IN C2 - 3 days ago

Atyeti Inc Pune, India

Designation: Data Engineer · Experience: 5+Years · Location: Pune · Skills: GCP / AWS, Python, Spark, Hive, Hadoop and Big Data · Bachelor's degree in computer science, information systems, or other related field or equivalent work experience. · 5+ years of high-tech industry int ...
Data Engineer

Found in: Talent IN 2A C2 - 5 hours ago

PeopleLogic Pune, India

Company ProfileWe are hiring for a multi-national company providing IT services. It provides software and consultancy services in different industries like Finance, Education, Insurance, Retail, Energy, Life Science, and Healthcare.Experience Range : 10-15Desired Competencies Mus ...
Data Engineer

Found in: Talent IN 2A C2 - 2 days ago

ACS International India Pvt. Ltd. (ACSII) Pune, India

ACS-I India is looking for a 5+ years experienced Data Engineer who will support a rapidly growing Business Data & Analytic Platform focused on providing data product solutions that empower our core business users to turn data into actionable insights. The data engineer will work ...
Data Engineer

Found in: Talent IN 2A C2 - 3 days ago

PeopleLogic Pune, India

Company Profile · We are hiring for a multi-national company providing IT services. It provides software and consultancy services in different industries like Finance, Education, Insurance, Retail, Energy, Life Science, and Healthcare. · Experience Range : 10-15 · Desired Compet ...
Data Engineer

Found in: Talent IN 2A C2 - 2 days ago

Coditas Pune, India

We are looking for data engineers who have the right attitude, aptitude, skills, empathy, compassion, and hunger for learning. Build products in the data analytics space. A passion for shipping high-quality data products, interest in the data products space; curiosity about the b ...
Data Engineer

Found in: Talent IN 2A C2 - 3 days ago

HyrEzy Talent Solutions Pune, India permanent

Job Description and Responsibilities : · Solicit business, functional, non-functional and technical requirements through interviewing and requirements gathering process. · Analyze and document above requirements and data definitions, perform data analysis, assist in change manage ...
Data Engineer

Found in: Talent IN 2A C2 - 5 hours ago

Mindpool Technologies Pune, India

Mindpool Technology is hiring for Data Engineer for Pune Location · Job Description Our Technical team is looking for a Data Engineer to part of EDWBI platform team to help build Data Validation framework and related validation reports leveraging existing EDWBI platform along wit ...
Data Engineer

Found in: Talent IN 2A C2 - 3 days ago

TMF Group Pune, India

About TMF Group · TMF Group is a leading provider of critical administrative services, helping clients invest and operate safely around the world. Our 10,000 experts and 120 offices in 85 jurisdictions worldwide serve corporates, financial institutions, asset managers, private cl ...
Data Engineer

Found in: Talent IN 2A C2 - 4 days ago

SGS Technical Services Pvt. Ltd Pune, India permanent

Experience : Total 4+ Years of experience · Notice Period : Immediate to 15 days · Location : Pune · Job Title : Data Engineer (Snowflake and Matillion) · Job Description : · Overview : · Design, implement, and optimize data pipelines and workflows using Snowflake and Matillio ...
Data Engineer

Found in: Talent IN 2A C2 - 5 hours ago

Virtusa Pune, India

Data Engineer - CREQ184791 Description 8-10years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real t ...
Data Engineer

Found in: Talent IN 2A C2 - 4 days ago

Huquo Pune/Hyderabad, India permanent

Must-Have : · - 4+ years of experience in Data Engineering experience on Azure and Databricks · - Strong knowledge of Databricks, Azure Data Factory, Azure Blob, and Azure SQL is required · - Experience Developing and tuning Spark Application and SparkSQL queries- · - Expertise t ...
Data Engineer

Found in: Adzuna IN C2 - 3 days ago

Nanotech Soft-App IT Solution Pune/Hyderabad, India permanent

In this role, you will: · - Technical leadership for a team of engineers that focus on development, deployment and operations. · - Lead and contribute to multiple pods with moderate resource requirements, risk, and/or complexity. · - Interface technically with a range of stakehol ...
Data Engineer

Found in: Talent IN 2A C2 - 4 days ago

ADVANSOFT Pune/Bangalore/Chennai, India permanent

Skills : · - Hadoop · - Python · - Spark · - PySpark · - ETL (Extract, Transform, Load) · Roles & Responsibilities : · - Data Ingestion: Develop and maintain data pipelines for ingesting raw data from various sources into the Hadoop ecosystem. · - Data Processing: Utilize Python ...
Data Engineer

Found in: Talent IN 2A C2 - 5 hours ago

Synechron Pune, India

Good day, · We have immediate opportunity for Data Analytics Engineer. · Job Role: · Data Analytics EngineerJob Location: · Kharadi, PuneExperience- · 8 Years to 12 Years.Notice Period: · upto 30 days. · About Company:At Synechron, we believe in the power of digital to transform ...
Data Engineer

Found in: Talent IN 2A C2 - 2 days ago

Promaynaov Advisory Services Pvt Ltd Pune, India

Role : Data Engineer · Location : Pune/Hyderabad · In this role, you will : · Principal Responsibilities : · As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects o ...