Data Platform Engineer

Role Overview: You will be responsible for designing and managing the foundational infrastructure for one of Weekday's clients. This role goes beyond simply maintaining pipelines as you will be leading the design and management of the data lakehouse, real-time stream processing frameworks, OLAP stores, self-service ETL and query frameworks, data movement APIs, reverse-ETL pipelines, and job orchestration layers. Key Responsibilities: - Take full ownership of the data lakehouse, including its architecture, ingestion from CDC sources, scalability, and reliability - Develop and manage real-time stream processing frameworks for applications like anomaly detection, customer 360 views, and live supply chain signals - Design and scale OLAP stores to support real-time and batch processing for internal analytics and AI/ML pipelines - Create self-service ETL and query frameworks for data consumers - Implement cost observability measures to reduce compute, storage, and query expenses - Build data movement APIs and reverse-ETL pipelines for efficient data delivery at scale - Establish stable job orchestration layers that remain consistent under scale Qualifications Required: - 312 years of experience in data engineering, with at least 17 years focused on building or managing a data platform - Hands-on expertise with tools like Spark, Hudi/Delta Lake, Kafka, Airflow, Debezium, Presto/Trino, DBT, Airbyte - Comfortable working with the AWS data ecosystem - Experience in managing daily processing of terabytes and billions of events - Demonstrated ability to reduce infrastructure costs and provide impact metrics - Proficiency in Java, Python, or Scala - Experience with OLAP engines like Pinot, Druid, or ClickHouse (Bonus) - Experience with data movement or reverse-ETL APIs (Bonus) - Familiarity with feature stores or data catalog tools (Bonus) [Additional Company Details]: The data platform you will be working on powers AI for supply chain decisions of Fortune 500 companies. You will directly witness the real business impact of your work and join a small team with high ownership and the challenge of working at true scale. Role Overview: You will be responsible for designing and managing the foundational infrastructure for one of Weekday's clients. This role goes beyond simply maintaining pipelines as you will be leading the design and management of the data lakehouse, real-time stream processing frameworks, OLAP stores, self-service ETL and query frameworks, data movement APIs, reverse-ETL pipelines, and job orchestration layers. Key Responsibilities: - Take full ownership of the data lakehouse, including its architecture, ingestion from CDC sources, scalability, and reliability - Develop and manage real-time stream processing frameworks for applications like anomaly detection, customer 360 views, and live supply chain signals - Design and scale OLAP stores to support real-time and batch processing for internal analytics and AI/ML pipelines - Create self-service ETL and query frameworks for data consumers - Implement cost observability measures to reduce compute, storage, and query expenses - Build data movement APIs and reverse-ETL pipelines for efficient data delivery at scale - Establish stable job orchestration layers that remain consistent under scale Qualifications Required: - 312 years of experience in data engineering, with at least 17 years focused on building or managing a data platform - Hands-on expertise with tools like Spark, Hudi/Delta Lake, Kafka, Airflow, Debezium, Presto/Trino, DBT, Airbyte - Comfortable working with the AWS data ecosystem - Experience in managing daily processing of terabytes and billions of events - Demonstrated ability to reduce infrastructure costs and provide impact metrics - Proficiency in Java, Python, or Scala - Experience with OLAP engines like Pinot, Druid, or ClickHouse (Bonus) - Experience with data movement or reverse-ETL APIs (Bonus) - Familiarity with feature stores or data catalog tools (Bonus) [Additional Company Details]: The data platform you will be working on powers AI for supply chain decisions of Fortune 500 companies. You will directly witness the real business impact of your work and join a small team with high ownership and the challenge of working at true scale.