Azure Devops engineer with DataBricks

Location: Weehawken, NJ – Onsite

Contract - W2 only, H1b Transfers are acceptable

Azure Cloud, Terraform, Databricks, Certifications in Azure (e.g., Azure Solutions Architect, Azure DevOps Engineer).

Job Description:

We are seeking a highly skilled and motivated Senior DevOps Engineer with Financial Industry experience and extensive experience in Databricks, Infrastructure management, Terraform, DevOps, and Azure. The successful candidate will be responsible for designing, implementing, and managing robust cloud solutions that support our business needs. This role requires a deep understanding of cloud architecture, automation, and continuous integration/continuous deployment (CI/CD) practices.

Responsibilities:

Design and implement scalable and secure cloud infrastructure using Azure services.
Manage and optimize cloud resources to ensure high availability and performance.
Monitor and troubleshoot cloud infrastructure and services.
Develop and manage Databricks environments for data processing and analytics.
Implement ETL pipelines using Databricks and integrate with other data sources.
Optimize Databricks performance and cost.
Write and maintain Terraform scripts to automate infrastructure provisioning.
Implement infrastructure as code (IaC) practices to ensure reproducibility and consistency.
Collaborate with development teams to integrate Terraform into CI/CD pipelines.
Implement and manage CI/CD pipelines for automated deployments and testing.
Use tools like Jenkins, GitHub Actions, or Azure DevOps for continuous integration and continuous deployment.
Ensure security and compliance in DevOps workflows.
Utilize Azure services such as Azure VMs, Azure Kubernetes Service (AKS), Azure SQL Database, Azure Blob Storage, etc.
Implement and manage Azure Networking, including VNETs, VPNs, and ExpressRoute.
Manage Azure Active Directory and role-based access control (RBAC).
Work closely with cross-functional teams including developers, data scientists, and IT operations.
Document infrastructure designs, configurations, and procedures.
Provide training and support to team members on cloud technologies and best practices.

Requirements:

· 14+ years of experience in cloud engineering with a focus on Azure.

· Proven experience with Databricks and data engineering.

· Strong knowledge of Terraform and infrastructure as code principles.

· Hands-on experience with DevOps tools and practices.

· Experience in containerization and orchestration tools like Docker and Kubernetes.

· Proficiency in scripting languages such as Python, PowerShell, or Bash.

· Solid understanding of networking concepts and cloud security best practices.

· Excellent problem-solving skills and the ability to work in a fast-paced environment.

· Certifications in Azure (e.g., Azure Solutions Architect, Azure DevOps Engineer).

· Familiarity with Big Data technologies and frameworks.

iPivot

New Jobs

DevOps Engineer with Terraform & DataBriks