Job Description:
About the Role:
We are looking for an Azure Data Engineer with minimum 6+ years of experience in Designing, developing and maintaining ETL/ELT pipelines using Azure Data Factory to ingest, transform and load data from diverse sources.
Requirements:
- Implementing and managing data storage solutions in Azure Data Lake, ensuring efficient data structuring, partitioning, and access controls.
- Developing complex data transformations using PySpark and Python for data processing on distributed systems, ensuring high performance and scalability.
- Writing optimized and efficient SQL queries to retrieve, manipulate, and aggregate large datasets from databases like Azure SQL, Synapse Analytics or other relational systems.
- Ensuring data quality and consistency through proper error handling, logging, and data validation checks within the ETL processes.
- Collaborate with data architects and business stakeholders to understand data requirements, build data models and design optimal solutions.
- Monitoring, troubleshooting, and optimizing data pipelines, ensuring data availability, performance, and cost-effectiveness.
- Implementing data security, governance, and compliance best practices for data storage and processing in the cloud environment.
- Contributing to DevOps practices, including CI/CD pipelines, for deploying data solutions using Azure DevOps or similar tools.
