1. use mobile phone proxies. Because of how mobile phone networks do NAT, basically it means that thousands of people share IPs and are much less like to get blocked.
2. Reverse engineer APIs if the data you want is returned in an ajax call.
3. Use a captcha solving service to defeat captchas. There's many and they are cheap.
4. Use an actual phone or get really good at convincing the server you are a mobile phone.
5. Buy 1000s of fake emails to simulate multiple accounts.
6. Experiment. Experiment. Experiment. Get some burner accounts. Figure out if they have request per min/hour/day throttling. See what behavior triggers a cloudflare captchas. Check if different variables such as email domain, useragent, voip vs non-voip sms based 2fa. your goal is to simulate a human. So if you sequentially enumerate through every document - that might be what get's you flagged.
1. use mobile phone proxies. Because of how mobile phone networks do NAT, basically it means that thousands of people share IPs and are much less like to get blocked.
2. Reverse engineer APIs if the data you want is returned in an ajax call.
3. Use a captcha solving service to defeat captchas. There's many and they are cheap.
4. Use an actual phone or get really good at convincing the server you are a mobile phone.
5. Buy 1000s of fake emails to simulate multiple accounts.
6. Experiment. Experiment. Experiment. Get some burner accounts. Figure out if they have request per min/hour/day throttling. See what behavior triggers a cloudflare captchas. Check if different variables such as email domain, useragent, voip vs non-voip sms based 2fa. your goal is to simulate a human. So if you sequentially enumerate through every document - that might be what get's you flagged.
Best of luck and happy scraping!