Senior Data Engineer - Pune, India - QAAgility Technologies

    QAAgility Technologies
    QAAgility Technologies Pune, India

    1 week ago

    Default job background
    Technology / Internet
    Description

    Senior Data Engineer - (Google BigQuery, Apache Dataflow and Shell Scripting)

    We are looking for a talented Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, implementing, and maintaining robust data pipelines and infrastructure to support our data-driven initiatives. You will work closely with cross-functional teams to ensure that our data solutions meet the needs of the business.

    Experience: 5-8 Yrs

    Location: Pune, Bengaluru ,Hyderabad

    Mandatory Skills:

    • Proficiency in Google BigQuery for data warehousing and analytics.
    • Experience with Apache Dataflow (Apache Beam) for building and managing data processing pipelines.
    • Strong shell scripting skills for automation and scheduling tasks.
    • Familiarity with Google Cloud Composer for workflow orchestration and management.
    • Preferred experience with BI tools such as Power BI and Microstrategy for data visualization and reporting.

    Key Responsibilities:

    • Design, develop, and deploy data pipelines using Google BigQuery and Apache Dataflow to process large-scale datasets efficiently.
    • Write and optimize shell scripts for automating data pipeline workflows and scheduling tasks.
    • Utilize Google Cloud Composer for orchestrating complex workflows and managing dependencies.
    • Collaborate with data analysts and business stakeholders to understand data requirements and translate them into technical solutions.
    • Implement best practices for data governance, data quality, and data security within our data infrastructure.
    • Troubleshoot and resolve issues related to data processing, performance, and scalability.
    • Work closely with the BI team to integrate data pipelines with preferred BI tools such as Power BI and MicroStrategy for reporting and visualization.
    • Stay current with industry trends and emerging technologies in data engineering and recommend opportunities for innovation and improvement.
    • Document data pipelines, workflows, and technical specifications for knowledge sharing and future reference.