Azure Data Lead

Role Overview: As a Data Engineering Lead, you will be responsible for designing, developing, and maintaining scalable data pipelines and lakehouse architectures using Azure Databricks (PySpark, SQL). You will lead a team of data engineers, oversee data governance, and ensure optimal data transformation and organization through the implementation of Medallion Architecture. Additionally, you will collaborate with stakeholders to deliver well-governed, reusable data models and analytics solutions, while staying ahead of industry trends in big data, ETL, and lakehouse technologies. Your role will also involve supporting machine learning and AI initiatives by optimizing data pipelines for ML and AI workloads. Key Responsibilities: - Lead the design, development, and maintenance of scalable data pipelines and lakehouse architectures using Azure Databricks (PySpark, SQL). - Architect and implement Medallion Architecture (Bronze, Silver, Gold layers) to optimize data transformation and organization. - Oversee data governance, access control, and lineage using Databricks Unity Catalog. - Manage and mentor a team of data engineers, providing technical guidance and career development opportunities. - Establish and maintain Delta Lake tables for reliable, scalable ACID transactions and incremental processing. - Optimize Spark jobs and Databricks clusters for performance, scalability, and cost efficiency. - Lead ingestion and transformation strategies from diverse sources such as APIs, files, databases, and cloud platforms. - Partner with stakeholders to deliver well-governed, reusable data models and analytics solutions. - Drive implementation of data quality frameworks, error handling mechanisms, and monitoring practices. - Enforce data security, governance, and compliance requirements across all Azure and cloud data assets. - Utilize Azure Synapse Analytics, Key Vault, and Azure Monitor to build resilient, secure, and observable data platforms. - Own CI/CD and deployment best practices using Git and Azure DevOps. - Stay informed about industry trends in big data, ETL, and lakehouse technologies. - Collaborate with data science and AI teams to support ML, generative AI, and LLM initiatives. - Guide data consultation efforts for enterprise digital transformation. Qualifications Required: - Bachelors or Masters degree in Computer Science, Engineering, or related field. - 8+ years of experience in data engineering, with 2+ years in a technical lead or supervisory role. - Advanced expertise in Databricks, PySpark, SQL, Azure cloud data engineering, and related tools. - Experience in architecting and leading end-to-end pipeline solutions leveraging Medallion architecture, Unity Catalog, and Delta Lake. - Strong leadership, communication, and mentoring skills. - Deep understanding of big data principles, data lakehouse architectures, and cloud ETL orchestration. - Hands-on experience with Azure Data Lake Storage, Data Factory, Synapse Analytics, Key Vault, Sql Server, and Azure Monitor. - Proficiency in CI/CD pipelines, automation with Git and Azure DevOps, and code quality enforcement. - Strong analytical mindset and strategic problem-solving skills. - Certifications in Azure/Data Engineering (Databricks, Azure) are highly desired. - Ability to consult on data strategy and architecture for digital transformation initiatives. - Knowledge of data preparation, feature engineering, and optimization for ML and AI workloads. - Experience collaborating cross-functionally with data science, analytics, and business teams on ML/AI use cases. Company Information: At Next, we are driven by the mission of enabling our customers to transform their aspirations into tangible outcomes. Our culture is defined by agility, innovation, and a commitment to progress, with a hands-on leadership style that prioritizes results and fosters growth. Perks of working with us include clear objectives, engagement with customers and leadership, continuous learning opportunities, and comprehensive family health insurance coverage. Join our passionate team and tailor your growth with us! Role Overview: As a Data Engineering Lead, you will be responsible for designing, developing, and maintaining scalable data pipelines and lakehouse architectures using Azure Databricks (PySpark, SQL). You will lead a team of data engineers, oversee data governance, and ensure optimal data transformation and organization through the implementation of Medallion Architecture. Additionally, you will collaborate with stakeholders to deliver well-governed, reusable data models and analytics solutions, while staying ahead of industry trends in big data, ETL, and lakehouse technologies. Your role will also involve supporting machine learning and AI initiatives by optimizing data pipelines for ML and AI workloads. Key Responsibilities: - Lead the design, development, and maintenance of scalable data pipelines and lakehouse architectures using Azure Databricks (PySpark, SQL).