Platform Data Engineer - Bengaluru / Bangalore - Clifyx Technology

    Clifyx Technology
    Clifyx Technology Bengaluru / Bangalore

    3 days ago

    Full time
    Description

    Key Responsibilities:

    • Design & Implement Data Architecture: Design, implement, and maintain the overall data platform architecture ensuring the scalability, security, and performance of the platform.
    • Data Technologies Integration: Select, integrate, and configure data technologies (cloud platforms like AWS, Azure, GCP, data lakes, data warehouses, streaming platforms like Kafka, containerization technologies).
    • Infrastructure Management: Setup and manage the infrastructure for data pipelines, data storage, and data processing across platforms like Kubernetes and Airflow.
    • Develop Frameworks & Tools: Develop internal frameworks to improve the efficiency and usability of the platform for other teams like Data Engineers and Data Scientists.
    • Data Platform Monitoring & Observability: Implement and manage monitoring and observability for the data platform, ensuring high availability and fault tolerance.
    • Collaboration: Work closely with software engineering teams to integrate the data platform with other business systems and applications.
    • Capacity & Cost Optimization: Involved in capacity planning and cost optimization for data infrastructure, ensuring efficient utilization of resources.

    Tech Stack Requirements:

    • Apache Iceberg (version : Experience in managing table formats for scalable data storage.
    • Apache Spark (version 3.4 and above): Expertise in building and maintaining batch processing and streaming data processing capabilities.
    • Apache Kafka (version 3.9 and above): Proficiency in managing messaging platforms for real-time data streaming.
    • Role-Based Access Control (RBAC): Experience with Apache Ranger (version for implementing and administering security and access controls.
    • RDBMS: Experience working with near real-time data storage solutions, specifically Oracle (version 19c).
    • Great Expectations (version 1.3.4): Familiarity with implementing Data Quality (DQ) frameworks to ensure data integrity and consistency.
    • Data Lineage & Cataloging: Experience with Open Lineage and DataHub (version for managing data lineage and catalog solutions.
    • Trino (version 4.7.0): Proficiency with query engines for batch processing.
    • Container Platforms: Hands-on experience in managing container platforms such as SKE (version 1.29 on AKS).
    • Airflow (version : Experience using workflow and scheduling tools for orchestrating and managing data pipelines.
    • DBT (Data Build Tool): Proficiency in using ETL/ELT frameworks like DBT for data transformation and automation.
    • Data Tokenization: Experience with data tokenization technologies like Protegrity (version 9.2) for ensuring data security.

    Desired Skills:

    • Domain Expertise: Familiarity with the Banking domain is a plus, including working with financial data and regulatory requirements.
  • Work in company

    Data Engineer

    Only for registered members

    Weiterhin suchen wir einen DevOps-Experten mit 7 bis 12 Jahren Erfahrung in relevanten Technologien, Konfiguration und Wartung von Kubernetes-Umgebungen einschließlich GitOps (ArgoCD), Ressourcen-Templating mit Helmcharts und Kustomize · ...

    Bengaluru

    1 month ago

  • Work in company

    Data Engineer

    Only for registered members

    Wir suchen einen DevOps-und Data Engineering Experten mit 7-12 Jahren Erfahrung. · Kubeflow Thingsboard Apache Superset Dremio usw. · ...

    Bengaluru, Karnataka

    3 weeks ago

  • Work in company

    Data Engineer

    Only for registered members

    Sviluppare pipeline dati utilizzando Python e SQL. · ...

    Bengaluru, Chennai, Hyderabad

    3 weeks ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    Design and implement data infrastructure for analytics needs. · Interface with other teams to extract and load data from various sources using SQL and AWS technologies. · ...

    Bengaluru

    1 week ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    Amazon IN Platform Development team is looking to hire a rock star Data/BI Engineer to build for pan Amazon India businesses. · ...

    Bengaluru

    2 weeks ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    The Amazon IN Platform Development team is looking to hire a rock star Data/BI Engineer to build for pan Amazon India businesses. · ...

    Bengaluru, Karnataka

    2 weeks ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    +Job summary · A rock star Data/BI Engineer to build for pan Amazon India businesses. · +Responsibilities · Design, implement and support an data infrastructure for analytics needs of large organization,Interface with other technology teams to extract, transform, and load data fr ...

    Bengaluru Full time

    2 weeks ago

  • Work in company

    Data Engineer Data Engineer

    Only for registered members

    · At ANZ, we're applying new ways technology and data can be harnessed as we work towards a common goal: to improve the financial wellbeing and sustainability of our millions of customers. · ...

    Bengaluru Full time

    1 week ago

  • Work in company

    Senior Data Engineer

    Only for registered members

    Le rôle consiste à concevoir et mettre en œuvre des solutions de lac de données et lakehouse scalables sur AWS pour les teas & analytics workloads. · AWS Core Services : S3, Lambda, DynamoDB, · RDS (PostgreSQL), API Gateway, CloudFormation, · CloudWatch. · Data Engineering Tools ...

    Bengaluru

    1 week ago

  • Work in company

    Data Engineer, IN Data Engineering

    Only for registered members

    We are looking to hire a rock star Data/BI Engineer to build for pan Amazon India businesses. · The BI Engineer will play a key role in contributing to the success of each focus area by partnering with respective business owners and leveraging data to identify areas of improvemen ...

    Bengaluru, Karnataka

    2 weeks ago

  • Work in company

    Senior Data Engineer

    Only for registered members

    Safran est un groupe international de haute technologie opérant dans les domaines de l'aéronautique (propulsion, équipements et intérieurs), de l'espace et de la défense. · ...

    Bengaluru, Karnataka ₹10 - ₹15 (INR) per year

    1 week ago

  • Work in company

    Data Engineer for Data

    Only for registered members

    We invite you to join our digital transformation journey. In this role, you will have the opportunity to boost your skills and become the heart of an innovative engine that is contributing to global impact and improvement. · ...

    Bengaluru Full time

    2 weeks ago

  • Work in company

    Data Engineer, Data Engineering Solutions

    Only for registered members

    Who we are · About Stripe · Stripe is a financial infrastructure platform for businesses.Millions of companies—at the world's largest enterprises to the most ambitious startups—use Stripe to accept payments, grow their revenue, and accelerate new business opportunities. · ...

    Bengaluru

    1 month ago

  • Work in company

    Staff Data Engineer, Data Engineering

    Only for registered members

    Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be pa ...

    Bengaluru

    2 days ago

  • Work in company

    Staff Data Engineer, Data Engineering

    Only for registered members

    · Company Description · Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the ...

    Bengaluru, India

    1 day ago

  • Work in company

    Staff Data Engineer, Data Engineering

    Only for registered members

    Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be pa ...

    Bengaluru, Karnataka ₹900,000 - ₹1,200,000 (INR) per year

    1 day ago

  • Work in company

    Data Engineer, senior data engineer

    Only for registered members

    Data Engineer role involves participating in the software development lifecycle for enterprise data projects, designing & maintaining robust ETL processes & data pipelines using modern platforms. · ...

    Bengaluru, Hyderabad

    1 week ago

  • Work in company

    Data Engineer, Data Engineering Solutions

    Only for registered members

    We are experts in data working to make it cost-effective trustworthy and understandable. We build pipelines processing billions of events daily and are stewards of canonical data warehouses and datasets delivering products for Stripe Users while embedding with teams to build thei ...

    Bengaluru, Karnataka

    1 month ago

  • Work in company

    Data Engineer/Senior Data Engineer

    Only for registered members

    Requirements · Job Title: Data Engineer – II / Senior Data Engineer · Job Location: Bengaluru, India · Job Summary: We are a leading Software as a Service (SaaS) company that specializes in the transformation of data in the US healthcare industry through cutting-edge Artificial I ...

    Bengaluru

    1 week ago

  • Work in company

    Staff Data Engineer, Data Engineering

    Only for registered members

    · Company Description · Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the ...

    Bengaluru ₹900,000 - ₹1,200,000 (INR) per year Full time

    2 days ago

  • Work in company

    Data Engineer, Data Engineering Solutions

    Only for registered members

    We're looking for a manager to drive the Data Engineering Solutions Team in solving high-impact,cutting-edge data problems. · Deliver cutting-edge data pipelines that scale to users' needs,focusing on reliability and efficiency. · ...

    Bengaluru Full time

    1 month ago

Jobs
>
Bengaluru