Senior/Lead Data Engineer

You will be responsible for building systems for the collection & transformation of complex data sets for use in production systems. Additionally, you will collaborate with engineers on building & maintaining back-end services while implementing data schema and data management improvements for scale and performance. Providing insights into key performance indicators for the product and customer usage will also be a key responsibility. You are expected to serve as the team's authority on data infrastructure, privacy controls, and data security. Furthermore, collaborating with appropriate stakeholders to understand user requirements and supporting efforts for continuous improvement, metrics, and test automation are essential tasks. You will also be required to maintain operations of the live service as issues arise on a rotational, on-call basis and verify whether the data architecture meets security and compliance requirements and expectations. Moreover, you should be able to quickly learn and adapt to Java/Scala and SQL at a rapid pace. Qualifications Required: - Bachelor's degree in computer science, computer engineering, or a related field, or equivalent experience. - 3+ years of progressive experience demonstrating strong architecture, programming, and engineering skills. - Firm grasp of data structures, algorithms with fluency in programming languages like Java, Python, Scala. - Strong SQL language skills with the ability to write complex queries. - Experience with Airflow like orchestration tools. - Demonstrated ability to lead, partner, and collaborate cross-functionally across multiple engineering organizations. - Experience with streaming technologies such as Apache Spark, Kafka, Flink. - Backend experience including Apache Cassandra, MongoDB, and relational databases such as Oracle, PostgreSQL AWS/GCP with solid hands-on experience of 4+ years. - Strong communication and soft skills. - Knowledge and/or experience with containerized environments, Kubernetes, Docker. - Experience in implementing and maintaining highly scalable microservices in Rest, Spring Boot, GRPC. - Appetite for trying new things and building rapid POCs. Key Responsibilities: - Design, develop, and maintain scalable data pipelines to support data ingestion, processing, and storage. - Implement data integration solutions to consolidate data from multiple sources into a centralized data warehouse or data lake. - Collaborate with data scientists and analysts to understand data requirements and translate them into technical specifications. - Ensure data quality and integrity by implementing robust data validation and cleansing processes. - Optimize data pipelines for performance, scalability, and reliability. - Develop and maintain ETL (Extract, Transform, Load) processes using tools such as Apache Spark, Apache NiFi, or similar technologies. - Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime. - Implement best practices for data management, security, and compliance. - Document data engineering processes, workflows, and technical specifications. - Stay up-to-date with industry trends and emerging technologies in data engineering and big data. You will be responsible for building systems for the collection & transformation of complex data sets for use in production systems. Additionally, you will collaborate with engineers on building & maintaining back-end services while implementing data schema and data management improvements for scale and performance. Providing insights into key performance indicators for the product and customer usage will also be a key responsibility. You are expected to serve as the team's authority on data infrastructure, privacy controls, and data security. Furthermore, collaborating with appropriate stakeholders to understand user requirements and supporting efforts for continuous improvement, metrics, and test automation are essential tasks. You will also be required to maintain operations of the live service as issues arise on a rotational, on-call basis and verify whether the data architecture meets security and compliance requirements and expectations. Moreover, you should be able to quickly learn and adapt to Java/Scala and SQL at a rapid pace. Qualifications Required: - Bachelor's degree in computer science, computer engineering, or a related field, or equivalent experience. - 3+ years of progressive experience demonstrating strong architecture, programming, and engineering skills. - Firm grasp of data structures, algorithms with fluency in programming languages like Java, Python, Scala. - Strong SQL language skills with the ability to write complex queries. - Experience with Airflow like orchestration tools. - Demonstrated ability to lead, partner, and collaborate cross-functionally across multiple engineering organizations. - Experience with streaming technologies such as Apache Spark, Kafka, Flink. - Backend experience including Apache Cassandra, Mo