Total Required Experience in Years: 10+ Years
Job Description:The client is seeking a senior-level AWS Data Engineer with strong Databricks expertise to design, build, and maintain scalable cloud-based data pipelines and health insurance applications. The role focuses on architecting data engineering solutions from scratch, implementing ETL/ELT processes, supporting enterprise integrations, and ensuring data quality and reliability. The ideal candidate should have strong AWS cloud expertise, Databricks experience, and the ability to collaborate across distributed Agile teams.
Key Responsibilities
- Design, build, and maintain scalable data pipelines using AWS and Databricks
- Implement ETL/ELT processes for loading and transforming data into data lakes and downstream systems
- Ensure data quality, reliability, observability, and performance
- Build and automate unit, integration, and performance testing
- Integrate internal systems, vendors, mobile applications, and data marts
- Support retail health insurance applications
- Monitor pipelines and proactively resolve production issues
- Participate in SAFe/Agile ceremonies and collaborate with cross-functional teams
Additional Responsibilities
- Support migration of on-premise data pipelines to Databricks cloud environments
- Work with Delta Lake, Apache Iceberg, and Apache Hudi technologies
- Implement partitioning strategies and metadata management
- Handle JSON/XML, document databases, and real-time ingestion processes
- Apply Lakehouse architecture, dimensional modeling, and Data Vault concepts
- Provide technical leadership and architecture recommendations
Required Skills
- 10+ years designing and architecting solutions from scratch
- Strong AWS expertise: Glue, ECS/EKS, Fargate, Aurora, DocumentDB, S3, Lambda, SNS, IAM, ECR
- Strong Databricks expertise: Delta Live Tables (DLT), Spark, SQL, Python/PySpark, Java, Databricks workflows or Apache Airflow
- Strong knowledge of ETL/ELT, data integration, and data warehousing principles
- Experience in DevOps practices, performance tuning, and design patterns
- Familiarity with Tableau, Alation, and Collibra
- Strong analytical, communication, and problem-solving skills
Qualifications & Certifications
- Bachelors degree in Information Systems or related field
- Advanced degree (MBA or equivalent) preferred
- Equivalent experience accepted
- AWS/Databricks certifications preferred but not mandatory
