Web Scraping Engineer · Bengaluru

AIMLEAP is looking for a Web Scraping Engineer to design and manage automated data extraction systems and web crawlers.

Key Responsibilities

• Build robust web crawlers and spiders using Python libraries such as Scrapy, BeautifulSoup, and Selenium.
• Extract large volumes of structured and unstructured data from complex web architectures.
• Handle dynamic content rendering, JavaScript execution, and single-page applications.
• Implement proxy rotation, user-agent management, and CAPTCHA handling mechanisms.
• Design data cleaning, validation, and transformation pipelines using Python and SQL.
• Monitor scraping pipelines to quickly identify and resolve website structure changes.
• Ensure extracted datasets meet strict quality, schema, and completeness standards.
• Store and deliver processed data in standard formats like JSON, CSV, and relational databases.
• Optimize crawlers for performance, execution speed, and bandwidth efficiency.
• Collaborate with data analysts and engineers to integrate scraped feeds into business databases.
• Document crawler architecture, extraction workflows, and system runbooks.