Lead Data Engineer

Overview

Responsibilities:

• Design and develop custom web crawlers to extract data.
• Build scalable scraping systems that comply with legal and ethical scraping practices.
• Clean, normalize, and store scraped data into structured databases .
• Integrate APIs where applicable to enrich scraped data.
• Maintain and update crawling scripts to adapt to dynamic websites.
• Ensure data quality, deduplication, and build rules to avoid banned IPs.

Preferred Skills:

• Python
• Database knowledge (PostgreSQL, MongoDB, or ElasticSearch)
• Experience with cloud platforms (AWS, GCP) and managing data pipelines
• Experience in IP rotation, CAPTCHA solving, and anti-bot evasion
• Basic understanding of GDPR/CCPA and data compliance
• Experience in B2B lead generation workflows

Job Types: Full-time, Part-time

Pay: From ₹499,856.55 per year

Expected hours: 40 per week

Benefits:

• Health insurance
• Paid sick time
• Paid time off
• Work from home

Schedule:

• Day shift

Supplemental Pay:

• Performance bonus
• Yearly bonus

Education:

• Bachelor's (Required)

Location:

• Remote (Required)

Work Location: Remote