Job Description:
Seeking a hands on and experienced Data DevOps Engineer to join our team and take ownership of automating, deploying, and managing CI/CD pipelines for a multi-product environment, including Azure Data Factory (ADF), WhereScape, Fabric, Semarchy MDM, Microsoft Purview, and other data platforms. The ideal candidate will have a strong background in Azure DevOps, Python scripting, and configuration management, with a proven ability to build and maintain automated deployment pipelines across complex environments.
This role requires expertise in managing data infrastructure, ensuring best practices for CI/CD, and troubleshooting technical issues related to databases, web services, Kafka, Spark, Fabric, and Databricks. The candidate will play a critical role in streamlining release management, ensuring seamless migration of product releases from development to upper environments, and maintaining robust exception management processes.
Key Responsibilities:
CI/CD Pipeline Automation:
- Design, develop, and maintain automated CI/CD pipelines for migrating product releases across multiple environments (Dev, Test, UAT, Prod).
- Implement and manage pipelines for Azure Data Factory (ADF), WhereScape, Fabric, Semarchy MDM, and other data platforms.
- Automate deployment processes for products which use databases, Kafka, Spark, and other data infrastructure components.
Infrastructure Management:
- Manage and optimize data infrastructure to ensure scalability, reliability, and performance.
- Configure and manage connections to databases, web services, Kafka, Spark, Fabric, and Databricks.
- Abstract and automate infrastructure configurations to enable repeatable and scalable deployments.
Scripting and Development:
- Develop and maintain Python scripts for automation, deployment, and configuration management.
- Write and maintain Windows and Unix shell scripts for infrastructure and deployment tasks.
- Collaborate with development teams to integrate scripts and automation into the CI/CD process.
Release and Configuration Management:
- Implement best practices for release management, configuration management, and exception management.
- Ensure version control, change management, and auditability across all deployments.
- Troubleshoot and resolve deployment issues across environments.
Technical Support and Troubleshooting:
- Provide technical expertise in configuring and managing connections to various data platforms and services.
- Diagnose and resolve technical issues related to pipelines, infrastructure, and deployments.
- Collaborate with cross-functional teams to ensure smooth delivery of product releases.
Tooling and Best Practices:
- Utilize Azure DevOps to build, test, and deploy pipelines.
- Stay updated with industry best practices and emerging technologies in DevOps, data engineering, and cloud platforms.
- Advocate for and implement DevOps best practices across the organization.
Qualifications:
- Proven experience in DevOps roles, with a focus on Azure Data Factory (ADF), WhereScape, Fabric, Semarchy MDM, and Microsoft Purview.
- Strong expertise in Azure DevOps for building and managing CI/CD pipelines.
- Proficiency in Python scripting for automation and deployment tasks.
- Experience with Windows and Unix shell scripting.
- Experience with databases, Kafka, Spark, Fabric, and Databricks.
- Knowledge of configuration management, release management, and exception management practices.
- Ability to troubleshoot and resolve technical issues related to data infrastructure and deployments.
- Strong understanding of cloud platforms (Azure preferred) and data engineering concepts.
- Excellent communication and collaboration skills, with the ability to work in a cross-functional team environment.
Preferred Skills:
- Experience with Semarchy MDM and Microsoft Purview for data governance and master data management.
- Familiarity with Kubernetes and Docker for containerization and orchestration.
- Knowledge of IaC (Infrastructure as Code) tools such as Terraform or ARM templates.
- Understanding of data security and compliance best practices.
