Databricks Specialist - Bengaluru, India - Affine

    Affine
    Affine Bengaluru, India

    Found in: Appcast Linkedin IN C2 - 1 week ago

    Default job background
    Description

    Objective:

    As a Data Engineer, you will be responsible for building effective data solutions that will cater to the needs of Data Analysts, Data Scientists, and other Data Consumers across the organization. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. They assess a wide range of requirements and apply relevant database techniques to create a robust architecture and will ensure delivering of high quality ETL solutions.

    Roles and Responsibilities:

    • Hands on experience in Databricks and related services(atleast 2 projects)
    • Should be comfortable in building and optimizing performant data pipelines which include data ingestion, data cleansing and curation into a datawarehouse, database, or any other data platform
    • Must have solid understanding of data structures and algorithms
    • Maintenance and optimization of existing processes
    • Experience in writing production ready code in Python and test, participate in code reviews to maintain and improve code quality, stability, and supportability
    • Experience in end to end development and deployment on any cloud platform
    • Leading the client calls to flag off any delays, blockers, escalations and collate all the requirements
    • Expertise in requirement gathering, technical design and functional documents
    • Experience in Agile/Scrum practices
    • Experience in leading other developers and guiding them technically
    • Experience in deploying data pipelines using automated CI/CD approach
    • Ability to write modularized reusable code components
    • Proficient in identifying data issues and anomalies during analysis
    • Strong analytical and logical skills
    • Must be able to comfortably tackle new challenges and learn
    • Must have strong verbal and written communication skills

    Technical Capabilities

    Must Have –

    • Databricks
    • Pyspark
    • SQL
    • Any ETL (Azure Data Factory (ADF)/Airflow)
    • DataLake and Delta Lake

    Good to Have-

    • Git/similar version control tool
    • Basic knowledge in CI/CD, Microservices

    Education: UG/PG preferably in Computer Science/Information Technology or any other related fields