DevOps Engineer - Bengaluru, Karnataka

Job summary

You'll be the engineer who makes data move reliably. You'll build Azure DevOps CI/CD that orchestrates ADF and Databricks Workflows to deliver data into a Medallion (Bronze/Silver/Gold) architecture, and you'll own Terraform-based IaC that provisions secure, scalable Azure and Databricks resources across environments.

Responsibilities

• Build CI/CD for data ingestion o Author and maintain Azure DevOps (ADO) YAML pipelines to version, validate, and deploy ADF pipelines and Databricks Workflows/Jobs. o Implement parameterized, reusable pipeline templates (multi-stage, approvals, gates) for dev/test/prod with artifact promotion. o Orchestrate batch and streaming patterns (e.g., ADF triggers + Databricks Jobs/Auto Loader) into Bronze ? Silver ? Gold layers. o Integrate code quality, unit tests, and data checks (pytest, dbx/tests, Great Expectations or equivalent) in CI.
• Provision cloud & lakehouse with IaC o Design Terraform modules and workspaces to provision Azure (RGs, VNets/PE, Key Vault, Storage, Event Hubs, Azure SQL, Monitor/Log Analytics) and Databricks (workspaces, UC metastore/catalogs/schemas, grants, clusters/cluster policies, Jobs, Repos/Asset Bundles). o Manage Terraform state (remote backends), environment promotion, and drift detection; enforce Policy-as-Code (Azure Policy, Sentinel/OPA). o Implement secure networking (VNet injection, Private Link, firewall rules), identity (AAD groups, service principals, workload identities), and secrets management (Key Vault ? Databricks secret scopes).
• Operate and improve the platform o Set up observability for pipelines (Azure Monitor, Log Analytics, Databricks metrics & audit logs), SLOs, alerting, and incident runbooks. o Optimize spend (autoscaling, spot/On-Demand mix, job cluster vs warehouse usage, caching, cluster policies). o Champion Git strategies (trunk/PR), code reviews, and release tagging; document standards and how-to guides.

Job description