Manager, Data Platform Engineering

As an experienced Manager - Data Platform Engineering, you will be responsible for leading and overseeing the operations, enhancements, and scaling of the enterprise-wide Data Platform. The platform includes Apache NiFi, Apache Spark, and MinIO (object storage). Your role will also involve driving integration and adoption of Kubernetes, Apache Airflow, and AI to ensure high availability, scalability, automation, and operational excellence across ingestion, processing, and storage workflows. **Key Responsibilities:** - Lead and manage the engineering team responsible for maintaining, optimizing, and upgrading platform components such as NiFi, Spark, and MinIO. - Drive adoption and rollout of Kubernetes (orchestration) and Airflow (workflow management). - Partner closely with DevOps to maintain CI/CD pipelines and containerized environments. - Collaborate with cross-functional teams (Data Science, Analytics, Application Development) to ensure smooth data ingestion and processing. - Establish best practices, automation, and monitoring for platform reliability, scalability, and security. - Manage incident response, perform RCA, and continuously improve operations. - Plan and oversee platform upgrades, migrations, and capacity planning. - Promote collaboration across engineering groups to ensure cohesive enhancements. - Track KPIs to measure platform health and engineering team performance. - Mentor and grow the engineering team, fostering a culture of innovation and accountability. **Qualifications Required:** - Bachelors or Masters degree in Computer Science, Engineering, or related field. - 12+ years of experience in data engineering, data platform operations, or distributed systems. - Strong experience managing and scaling data platforms with components like NiFi, Spark, and object stores like MinIO. - Strong knowledge of Linux/Unix administration, scripting, and automation. - Java/Scala/Python development experience. - Hands-on or conceptual understanding of Kubernetes. - Familiarity with orchestration tools such as Apache Airflow. - Deep understanding of data ingestion, ETL/ELT concepts, and distributed data pipelines. - Experience leading and mentoring engineering teams. - Strong communication skills across technical and non-technical audiences. This job also prefers candidates with previous experience in NiFi/Spark pipeline development and exposure to AI/ML systems or experience integrating data platforms with AI pipelines. As an experienced Manager - Data Platform Engineering, you will be responsible for leading and overseeing the operations, enhancements, and scaling of the enterprise-wide Data Platform. The platform includes Apache NiFi, Apache Spark, and MinIO (object storage). Your role will also involve driving integration and adoption of Kubernetes, Apache Airflow, and AI to ensure high availability, scalability, automation, and operational excellence across ingestion, processing, and storage workflows. **Key Responsibilities:** - Lead and manage the engineering team responsible for maintaining, optimizing, and upgrading platform components such as NiFi, Spark, and MinIO. - Drive adoption and rollout of Kubernetes (orchestration) and Airflow (workflow management). - Partner closely with DevOps to maintain CI/CD pipelines and containerized environments. - Collaborate with cross-functional teams (Data Science, Analytics, Application Development) to ensure smooth data ingestion and processing. - Establish best practices, automation, and monitoring for platform reliability, scalability, and security. - Manage incident response, perform RCA, and continuously improve operations. - Plan and oversee platform upgrades, migrations, and capacity planning. - Promote collaboration across engineering groups to ensure cohesive enhancements. - Track KPIs to measure platform health and engineering team performance. - Mentor and grow the engineering team, fostering a culture of innovation and accountability. **Qualifications Required:** - Bachelors or Masters degree in Computer Science, Engineering, or related field. - 12+ years of experience in data engineering, data platform operations, or distributed systems. - Strong experience managing and scaling data platforms with components like NiFi, Spark, and object stores like MinIO. - Strong knowledge of Linux/Unix administration, scripting, and automation. - Java/Scala/Python development experience. - Hands-on or conceptual understanding of Kubernetes. - Familiarity with orchestration tools such as Apache Airflow. - Deep understanding of data ingestion, ETL/ELT concepts, and distributed data pipelines. - Experience leading and mentoring engineering teams. - Strong communication skills across technical and non-technical audiences. This job also prefers candidates with previous experience in NiFi/Spark pipeline development and exposure to AI/ML systems or experience integrating data platforms with AI pipelines.