Web Scraping and Automation Developer

As a Data Engineer specializing in AI & Web Scraping Automation with 5+ years of experience, you will play a crucial role in leveraging your expertise to lead the development and implementation of web scraping solutions in a remote work environment. Your primary responsibilities will involve building scalable ETL pipelines using Python and AWS, developing and maintaining web scraping systems using tools like Scrapy, Selenium, and BeautifulSoup, integrating OpenAI / Azure OpenAI APIs for structured data extraction, building workflow automation with tools like n8n, and implementing orchestration using Dagster or Airflow. Additionally, you will be expected to manage structured storage using PostgreSQL, S3, and Parquet, as well as lead and mentor junior engineers. **Key Responsibilities:** - Strong hands-on experience in Python-based web scraping and crawling using Requests, Scrapy, Selenium, and Playwright. - Deep hands-on experience working with JavaScript-heavy and dynamic websites using Selenium, Playwright, or similar browser automation frameworks. - Proven expertise in handling large-scale enterprise data scraping across complex, high-traffic platforms. - Lead and mentor a web scraping team to build and scale enterprise-grade crawling solutions using Python, Requests, Scrapy, Selenium, and Playwright. - Robust understanding of protocols, cookies, sessions, headers, tokens, and browser storage (local/session storage). - Hands-on experience with proxy rotation, IP management, routers, rate limiting, fingerprinting, and anti-bot evasion techniques. - Ability to design and maintain scalable, reliable, and fault-tolerant crawling architectures. **Qualifications:** - 610 years of hands-on experience in web scraping and crawling using Python. - Strong practical knowledge of Requests, Scrapy, Selenium, and Playwright, Python, AWS (RDS/S3/EC2), OpenAI API, ETL design. - Proven experience scraping JavaScript-heavy, dynamic, and access-controlled enterprise websites. - Deep understanding of cookies, sessions, headers, proxies, IP rotation, routers, and anti-detection strategies. - Experience working with SQL/NoSQL databases for structured data storage. - Strong debugging, system design, and problem-solving skills. - Ability to independently own and scale scraping systems in a remote environment. Please note that immediate joiners are preferred for this full-time remote position. If you meet the qualifications and are ready to take on the challenges of leading web scraping initiatives in the realm of Data & AI, we look forward to receiving your application. As a Data Engineer specializing in AI & Web Scraping Automation with 5+ years of experience, you will play a crucial role in leveraging your expertise to lead the development and implementation of web scraping solutions in a remote work environment. Your primary responsibilities will involve building scalable ETL pipelines using Python and AWS, developing and maintaining web scraping systems using tools like Scrapy, Selenium, and BeautifulSoup, integrating OpenAI / Azure OpenAI APIs for structured data extraction, building workflow automation with tools like n8n, and implementing orchestration using Dagster or Airflow. Additionally, you will be expected to manage structured storage using PostgreSQL, S3, and Parquet, as well as lead and mentor junior engineers. **Key Responsibilities:** - Strong hands-on experience in Python-based web scraping and crawling using Requests, Scrapy, Selenium, and Playwright. - Deep hands-on experience working with JavaScript-heavy and dynamic websites using Selenium, Playwright, or similar browser automation frameworks. - Proven expertise in handling large-scale enterprise data scraping across complex, high-traffic platforms. - Lead and mentor a web scraping team to build and scale enterprise-grade crawling solutions using Python, Requests, Scrapy, Selenium, and Playwright. - Robust understanding of protocols, cookies, sessions, headers, tokens, and browser storage (local/session storage). - Hands-on experience with proxy rotation, IP management, routers, rate limiting, fingerprinting, and anti-bot evasion techniques. - Ability to design and maintain scalable, reliable, and fault-tolerant crawling architectures. **Qualifications:** - 610 years of hands-on experience in web scraping and crawling using Python. - Strong practical knowledge of Requests, Scrapy, Selenium, and Playwright, Python, AWS (RDS/S3/EC2), OpenAI API, ETL design. - Proven experience scraping JavaScript-heavy, dynamic, and access-controlled enterprise websites. - Deep understanding of cookies, sessions, headers, proxies, IP rotation, routers, and anti-detection strategies. - Experience working with SQL/NoSQL databases for structured data storage. - Strong debugging, system design, and problem-solving skills. - Ability to independently own and scale scraping systems in a remote environment. Please note that immediate joiners are preferred for this full-time re