Lead Data Engineer / Platform Engineer

As a Lead Data / Platform Engineer at our company, you will play a crucial role in architecting, building, and scaling a modern lakehouse and data platform stack powered by Spark, Trino, Kubernetes, and other open-source technologies. You will be responsible for leading the architecture and implementation of distributed data platforms, designing and building high-performance pipelines, and implementing strong data governance frameworks. Your role will involve mentoring engineers, ensuring delivery of high-quality software, and establishing best practices for reliability, performance, and cost optimization. **Key Responsibilities:** - Lead architecture and implementation of distributed data platforms - Design and build high-performance pipelines using Spark, PySpark, and Spark SQL - Build and optimize federated query systems using Trino - Design scalable orchestration workflows using Apache Airflow - Implement strong data governance frameworks - Own Kubernetes-based OSS stack deployment - Architect ML data pipelines for feature engineering and model training - Drive SSO integrations across platform components - Mentor engineers and enforce engineering excellence standards - Establish best practices for reliability, performance, and cost optimization **Required Technical Expertise:** **Core Data Engineering** - Strong hands-on experience with Apache Spark, PySpark, and Spark SQL - Experience working with TB-scale datasets - Query engine expertise with Trino - Strong Python programming skills - Deep understanding of distributed systems and data modeling **Orchestration & Pipelines** - Production experience with Apache Airflow - Designing reliable DAGs with observability - CI/CD for data pipelines **Data Governance & Quality** - Experience with data catalog tools - Data contracts and schema enforcement - Data quality frameworks and lineage tracking **ML & Data Science Exposure** - Experience supporting ML pipelines - Feature engineering and model lifecycle exposure - Understanding of MLflow or equivalent tools **Platform & DevOps** - Kubernetes production deployments - Infrastructure as Code using Terraform - Managing OSS stacks via Helm charts - Performance tuning and cluster management **Security & Identity** - Experience implementing SSO (OAuth2 / OIDC) - Understanding of RBAC and multi-tenant security architectures **AI & Agentic Systems Expertise** - Experience building Agentic AI systems and multi-agent workflows - Hands-on experience with LangChain and LangGraph frameworks - Strong understanding of Prompt Engineering techniques - Experience building Text-to-SQL, code generation, or AI assistant workflows - Knowledge of Retrieval-Augmented Generation (RAG) architectures - Experience integrating LLM APIs - Understanding of model evaluation, guardrails, and AI observability - Experience deploying AI/ML services on Kubernetes **Leadership Expectations:** - Lead and mentor a team of engineers - Drive delivery with clear milestones - Make scalable, long-term architectural decisions - Communicate effectively with stakeholders - Raise the bar for engineering quality As a Lead Data / Platform Engineer at our company, you will play a crucial role in architecting, building, and scaling a modern lakehouse and data platform stack powered by Spark, Trino, Kubernetes, and other open-source technologies. You will be responsible for leading the architecture and implementation of distributed data platforms, designing and building high-performance pipelines, and implementing strong data governance frameworks. Your role will involve mentoring engineers, ensuring delivery of high-quality software, and establishing best practices for reliability, performance, and cost optimization. **Key Responsibilities:** - Lead architecture and implementation of distributed data platforms - Design and build high-performance pipelines using Spark, PySpark, and Spark SQL - Build and optimize federated query systems using Trino - Design scalable orchestration workflows using Apache Airflow - Implement strong data governance frameworks - Own Kubernetes-based OSS stack deployment - Architect ML data pipelines for feature engineering and model training - Drive SSO integrations across platform components - Mentor engineers and enforce engineering excellence standards - Establish best practices for reliability, performance, and cost optimization **Required Technical Expertise:** **Core Data Engineering** - Strong hands-on experience with Apache Spark, PySpark, and Spark SQL - Experience working with TB-scale datasets - Query engine expertise with Trino - Strong Python programming skills - Deep understanding of distributed systems and data modeling **Orchestration & Pipelines** - Production experience with Apache Airflow - Designing reliable DAGs with observability - CI/CD for data pipelines **Data Governance & Quality** - Experience with data catalog tools - Data contracts and schema enforcement - Data quality frameworks and lineage