Lead - Market Intelligence & Data Engineering

You will be responsible for leading Market Intelligence and Data Engineering initiatives focused on collecting and analyzing external marketplace data. This role will require you to design scalable data collection systems, build reliable data pipelines, and deliver insights that support strategic business decisions. **Key Responsibilities:** - Design and maintain scalable data collection systems to gather structured information from online marketplaces and public data sources. - Collect and monitor data such as product listings, pricing, promotions, product rankings, reviews, and competitor activity. - Develop systems capable of operating across multiple e-commerce platforms and geographic markets. - Build real-time or near-real-time data pipelines for monitoring price changes, promotions, and marketplace trends. - Ensure reliable data collection while managing platform constraints such as rate limits and regional restrictions. - Transform and normalize data from multiple sources into a unified and structured data model. - Develop systems to map products, brands, and categories across platforms. - Implement monitoring processes to maintain data quality, reliability, and freshness. - Enhance datasets with taxonomy, categorization, and relevant metadata. **Decision Support for Business Functions:** Provide data insights that support teams across areas such as: **Marketing:** - Competitive signals, search performance insights, and category visibility. **Pricing:** - Monitoring market prices, discounts, and competitive positioning. **Inventory & Buying:** - Demand signals, category trends, and seasonal patterns. **Market Expansion:** - Identifying opportunities across regions and categories. **Integration with Internal Data Systems:** - Collaborate with internal data and analytics teams to combine external market intelligence with operational data. - Develop data products and dashboards that support business decision-making. - Maintain reliable, production-grade pipelines for both batch analytics and real-time insights. **Continuous Improvement & Innovation:** - Expand data coverage across additional platforms and markets. - Explore new data sources such as social insights or market signals. - Apply AI/ML techniques to improve data extraction, matching, and predictive insights. **Qualification Required:** - 6 - 10 years of experience in data engineering, web crawling, or market data infrastructure. - Strong expertise in Python and SQL. - Experience with web crawling frameworks such as Scrapy, Playwright, Puppeteer, or similar tools. - Experience building data pipelines using tools such as Airflow, Dagster, or similar orchestration frameworks. - Experience working with real-time data processing tools such as Kafka, Flink, or similar platforms. - Experience deploying scalable systems on cloud infrastructure (AWS preferred). - Strong understanding of data quality, monitoring, and observability practices. **Preferred Experience:** - Experience working with e-commerce or marketplace data environments. - Knowledge of large-scale data collection and structured data systems. - Exposure to AI/ML techniques for data extraction and analysis. You will be responsible for leading Market Intelligence and Data Engineering initiatives focused on collecting and analyzing external marketplace data. This role will require you to design scalable data collection systems, build reliable data pipelines, and deliver insights that support strategic business decisions. **Key Responsibilities:** - Design and maintain scalable data collection systems to gather structured information from online marketplaces and public data sources. - Collect and monitor data such as product listings, pricing, promotions, product rankings, reviews, and competitor activity. - Develop systems capable of operating across multiple e-commerce platforms and geographic markets. - Build real-time or near-real-time data pipelines for monitoring price changes, promotions, and marketplace trends. - Ensure reliable data collection while managing platform constraints such as rate limits and regional restrictions. - Transform and normalize data from multiple sources into a unified and structured data model. - Develop systems to map products, brands, and categories across platforms. - Implement monitoring processes to maintain data quality, reliability, and freshness. - Enhance datasets with taxonomy, categorization, and relevant metadata. **Decision Support for Business Functions:** Provide data insights that support teams across areas such as: **Marketing:** - Competitive signals, search performance insights, and category visibility. **Pricing:** - Monitoring market prices, discounts, and competitive positioning. **Inventory & Buying:** - Demand signals, category trends, and seasonal patterns. **Market Expansion:** - Identifying opportunities across regions and categories. **Integration with Internal Data Systems:** - Collaborate with internal d