SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
iCapital is seeking an experienced Python Developer to design and build robust, secure, and scalable web scraping solutions that interact with complex web applications and secured portals. This role focuses on architecting advanced scraping pipelines using Python, Playwright for browser automation, and AWS cloud-native services.
Key responsibilities include:
- Design and implement advanced scraping solutions using Python, Playwright, and AWS services
- Automate interactions with JavaScript-heavy and authentication-secured websites, handling multi-factor authentication (MFA), CAPTCHAs, and session/token-based login flows
- Architect serverless scraping pipelines using AWS Lambda, Step Functions, S3, CloudWatch, and Secrets Manager
- Build systems that scale to support high volumes of data extraction with fault tolerance, retries, and intelligent logging
- Integrate and manage complex workflows across multiple portals, APIs, and data sources
- Contribute to architectural decisions, tooling, and best practices
Required qualifications:
- 5+ years of Python development experience with strong focus on automation and data extraction
- Proven expertise in web scraping using Playwright, Selenium, Scrapy, BeautifulSoup, or requests
- In-depth experience handling multi-step authentication flows, including MFA, CAPTCHA solving, and session/cookie management
- Proficiency deploying and managing scraping workloads in AWS (Lambda, S3, IAM, CloudWatch, Secrets Manager)
- Experience with asynchronous programming, headless browsers, and JavaScript-rendered content
- Solid understanding of web protocols (HTTP, HTTPS, cookies, headers) and ability to reverse-engineer network calls
- Comfortable working with APIs, JSON/XML, and data transformation
- Bachelor's or Master's degree in Computer Science, Software Engineering, or equivalent professional experience
Preferred qualifications include CI/CD pipelines, Docker, infrastructure as code (CloudFormation, Terraform), data pipeline orchestration tools (Apache Airflow, AWS Step Functions), experience scraping secure enterprise portals, data engineering/ETL background, and Python testing frameworks.
The role is hybrid with four days in office and one day remote (Friday).
