Job Title: Lead Managed Service Data Engineer
Job Type: Full-time
We are seeking a
Lead Managed Service Data Engineer to join our dynamic team. This is a leadership position, where you will guide a team of data engineers while working on complex data pipelines, cloud infrastructure, and managed service offerings. The ideal candidate will have strong technical expertise, particularly with
PySpark,
Databricks,
Azure,
SQL, and
DevOps, along with the ability to mentor, inspire, and lead teams in delivering high-quality solutions.
Key Responsibilities
Leadership & Team Management:
- Lead, mentor, and manage a team of data engineers, providing guidance on best practices, architecture design, and technical approaches.
- Oversee the design, development, and implementation of data pipelines and systems for managed services clients.
- Collaborate with other team leads, project managers, and cross-functional stakeholders to ensure alignment of technical solutions with business requirements.
- Foster a culture of continuous learning, improvement, and collaboration within the team.
Technical Strategy & Architecture
- Architect, design, and implement scalable, high-performance data pipelines using PySpark and Databricks in an Azure cloud environment.
- Ensure solutions are optimized for cost, performance, and scalability in Azure cloud platforms, including services like Azure Data Lake, Azure Synapse Analytics, and Azure SQL Database.
- Lead the deployment of data engineering workflows with integration into the overall DevOps pipeline for continuous integration/continuous deployment (CI/CD).
Data Engineering & Development
- Develop robust data pipelines and ETL processes to move, transform, and load data from diverse sources into the data lake, data warehouse, or other storage solutions.
- Build and optimize SQL queries and databases for efficient data retrieval and analysis.
- Apply PySpark and Databricks to build distributed data processing solutions, including data ingestion, transformation, and aggregation.
- Design and implement automated data monitoring and alerting systems to ensure the health of data systems and pipelines.
Collaboration & Stakeholder Management
- Work closely with business analysts, data scientists, and other stakeholders to understand requirements and translate them into technical solutions.
- Provide guidance on data governance, security practices, and compliance standards as they relate to managed services and cloud infrastructure.
- Act as a point of contact for technical escalations and support, ensuring client satisfaction with service delivery.
Required Qualifications & Skills
- Education: Bachelor's or Master's degree in Computer Science, Engineering, or a related field, or equivalent work experience.
- Experience:
- 5+ years of experience as a Data Engineer with a proven track record of working with PySpark, Databricks, Azure, and SQL.
- 2+ years in a leadership or team management role, with experience mentoring and guiding engineers on technical best practices.
- Solid experience in cloud-based data architectures, specifically with Azure services (e.g., Azure Data Lake, Azure Synapse, Azure SQL Database, Azure Data Factory).
- Experience in designing and deploying data pipelines in a production environment with a focus on scalability, reliability, and performance.
- Experience with DevOps practices, CI/CD pipelines, and infrastructure automation (e.g., using Azure DevOps, Terraform, Git, etc.).
- Technical Skills:
- Expertise in PySpark for big data processing and working with Databricks for collaborative data engineering.
- Advanced SQL skills for querying and optimizing databases, working with large datasets.
- Strong knowledge of cloud data services, especially Azure (Data Factory, Data Lake, Synapse, etc.).
- Familiarity with DevOps tools and practices to manage infrastructure and deployments (e.g., GitHub, Azure DevOps, Jenkins).
- Leadership Skills:
- Proven ability to lead technical teams, provide mentorship, and promote a collaborative work environment.
- Strong problem-solving skills and a proactive approach to resolving technical challenges.
- Excellent communication skills, both written and verbal, to interface with technical and non-technical stakeholders.
Preferred Qualifications
- Certifications: Azure Data Engineer or Azure Solutions Architect certifications are a plus.
- Familiarity with DataOps principles and tools.
- Experience in handling data security, compliance, and governance frameworks (e.g., GDPR, HIPAA).
- Familiarity with additional tools or languages such as Python, Terraform, or Kubernetes.
