Senior Data Engineer - ETL/Python

As a Senior Data Engineer specializing in AI & Analytics, your role is crucial in designing, developing, and optimizing end-to-end data pipelines to support AI-driven engineering. You will collaborate with various teams to ensure high-impact solutions are delivered seamlessly. Your responsibilities will include: - Designing, developing, and optimizing end-to-end data pipelines for ingestion, transformation, and orchestration. - Building and maintaining high-performance data models, ETL processes, and storage layers for large datasets. - Integrating AI capabilities by implementing LLM workflows, embeddings, vector stores, and inference services. - Collaborating with engineering, data, and product teams to ensure data quality, reliability, lineage, and security. - Developing APIs, microservices, and interfaces supporting analytics, reporting, and AI-driven features. - Implementing monitoring, logging, and automation frameworks for robust data operations. - Conducting performance optimization of pipelines, queries, and data infrastructure. - Evaluating and recommending modern tools and technologies in cloud, big data, and AI domains. In order to excel in this role, you are required to have: - 5-8 years of hands-on experience as a Data Engineer or in a related role. - Strong programming skills in Python, TypeScript, SQL, Spark, and Databricks. - Experience with Jupyter Notebooks for data exploration and prototyping AI/data workflows. - Knowledge of cloud platforms like Azure, AWS, GCP, data lakes/lakehouses, and distributed computing. - Practical experience integrating AI/LLM APIs, vector databases, and RAG workflows. - Understanding of data warehousing, schema design, metadata management, and large-scale processing. - Proficiency in orchestration tools like Airflow, Prefect, and Dagster. - Familiarity with containerization and deployment using Docker and Kubernetes. - Experience in building CI/CD pipelines for data and AI systems. - Strong analytical and problem-solving skills. Preferred qualifications include exposure to AI agent frameworks, LLM prompt automation, and retrieval pipelines, along with an understanding of enterprise security, RBAC, and compliance requirements. Prior experience working in large-scale enterprise data environments is also beneficial. Moreover, your soft skills will play a significant role in your success, including excellent communication, collaboration, simplifying complex technical concepts, high ownership mindset, and the ability to work independently as a high-impact individual contributor. In this role, you will contribute to the development of next-generation data and AI capabilities, enabling intelligence, automation, and insights across key business initiatives. This position offers significant opportunities for innovation and technical impact. As a Senior Data Engineer specializing in AI & Analytics, your role is crucial in designing, developing, and optimizing end-to-end data pipelines to support AI-driven engineering. You will collaborate with various teams to ensure high-impact solutions are delivered seamlessly. Your responsibilities will include: - Designing, developing, and optimizing end-to-end data pipelines for ingestion, transformation, and orchestration. - Building and maintaining high-performance data models, ETL processes, and storage layers for large datasets. - Integrating AI capabilities by implementing LLM workflows, embeddings, vector stores, and inference services. - Collaborating with engineering, data, and product teams to ensure data quality, reliability, lineage, and security. - Developing APIs, microservices, and interfaces supporting analytics, reporting, and AI-driven features. - Implementing monitoring, logging, and automation frameworks for robust data operations. - Conducting performance optimization of pipelines, queries, and data infrastructure. - Evaluating and recommending modern tools and technologies in cloud, big data, and AI domains. In order to excel in this role, you are required to have: - 5-8 years of hands-on experience as a Data Engineer or in a related role. - Strong programming skills in Python, TypeScript, SQL, Spark, and Databricks. - Experience with Jupyter Notebooks for data exploration and prototyping AI/data workflows. - Knowledge of cloud platforms like Azure, AWS, GCP, data lakes/lakehouses, and distributed computing. - Practical experience integrating AI/LLM APIs, vector databases, and RAG workflows. - Understanding of data warehousing, schema design, metadata management, and large-scale processing. - Proficiency in orchestration tools like Airflow, Prefect, and Dagster. - Familiarity with containerization and deployment using Docker and Kubernetes. - Experience in building CI/CD pipelines for data and AI systems. - Strong analytical and problem-solving skills. Preferred qualifications include exposure to AI agent frameworks, LLM prompt automation, and retrieval pipelines, along with an understanding of enterpri