Data Engineer – Senior Associate - Bengaluru, India - PwC

    PwC
    PwC Bengaluru, India

    2 weeks ago

    PwC background
    Description
    & SummaryA career in Products and Technology is an opportunity to bring PwC's strategy to life by driving products and technology into everything we deliver. Our clients expect us to bring the right people and the right technology to solve their biggest problems; Products and Technology is here to help PwC meet that challenge and accelerate the growth of our business. We have skilled technologists, data scientists, product managers and business strategists who are using technology to accelerate change.

    Our team collaborates with product strategy and product managers to govern readiness standards in achieving principles (compliance, privacy, security) by design for what PwC's technology assets require to be successful in the market.

    They provide guidance for product development across the lifecycle (ideation / strategy through commercialization / monetization).

    Additionally, they facilitate market readiness for technology assets overall, as changes occur to assets or market conditions throughout the asset's life cycle.

    Data Engineer (Azure Data Lake, Spark & Databricks)


    Required Knowledge and Skills:6-9 years of experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.

    High Proficiency in at least two of Scala, Python, Spark applied to large scale data setsExpertise with big data technologies (Spark, Data Lake, Delta Lake, Hive)Knowledge of batch and streaming data processing techniquesUnderstanding of the Data Lifecycle Management process to collect, access, use, store, transfer, delete data.

    Expert level knowledge of using SQL to write complex, highly optimized queries across large volumes of data.

    Hands-on object-oriented programming experience using Scala, Python, R, or JavaProficient with Azure DevOps, managing backlogs and creating/maintaining pipelinesExperience working in an Agile environment and applying it in Azure DevOpsAt Effectual, Data engineers are responsible for designing, building, and maintaining datasets that can be leveraged in data projects.

    Obsession for service observability, instrumentation, monitoring, and alertingKnowledge or experience in architectural best practices in building data lakesResponsibilitiesYou will be responsible for designing and building optimized data pipelines using emerging technologies, in a cloud environment, for the purpose of driving analytic insights.

    Create the conceptual, logical and physical data models.

    Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of sources like API, Blob Storage, and no SQL DatabaseDesign, develop, test, deploy, maintain, and improve data integration pipeline for data streaming.

    Develop pipeline objects using Apache Spark / Pyspark / PythonDesign and develop data pipeline architectures using Databricks, Spark and cloud Services.

    Load and performance test data pipelines built using the above-mentioned technologies.
    Good to havePassionate about testing strategy, problem solving, learning new skills, sharing expertise and knowledge.

    Always Be LearningProduct / Engineering MindsetEducationDegrees/Field of Study required:

    Degrees/Field of Study preferred:
    CertificationsRequired SkillsOptional SkillsDesired LanguagesTravel RequirementsNot Specified

    Available for Work Visa Sponsorship?No

    Government Clearance Required?No

    Job Posting End Date