- Lead the design, development, and implementation of innovative platform automation solutions for the Databricks data lake on AWS.
- Automate the provisioning, configuration, and management of core AWS services (S3, IAM, EC2, VPC, Lambda, CloudWatch, SNS, SQS, ELB, Route53) using Infrastructure as Code (Terraform).
- Develop robust automation scripts and tools using Python for Databricks workspaces, clusters, jobs, and security configurations, leveraging Databricks APIs and CLI.
- Design, implement, and maintain CI/CD pipelines (e.g., GitHub Actions) for continuous delivery of infrastructure, platform, and automation code.
- Implement and enforce security best practices, compliance, and cost optimization strategies across the AWS and Databricks data platform.
- Integrate and automate the management of AWS data services (Redshift, DynamoDB, Glue, Athena) within the Databricks ecosystem.
- Drive continuous improvement and innovation in platform operations, monitoring, and performance.
- Collaborate with data engineers, data scientists, governance, and operations teams to deliver automated solutions that enhance productivity, platform stability and improve customer experience.
Required Skills & Qualifications:
- 5-7 years of professional experience in cloud platform engineering, DevOps, or data platform automation.
- AWS (Mandatory): Extensive hands-on experience with S3, IAM, EC2, API Gateway, Step Functions, VPC, Lambda, CloudWatch, SNS, SQS, ELB, and Route53.
- DevOps and Automation (Mandatory)
- Proven expertise in Infrastructure as Code (IaC) with Terraform.
- Advanced Python scripting for platform automation, API integration, and data pipeline orchestration.
- A passion for identifying and implementing innovative automation solutions.
- Solid understanding of CI/CD principles and practical experience with tools like GitHub Actions.
- Proficiency with Git for version control.
