Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
mattfrommars
on Sept 1, 2020
|
parent
|
context
|
favorite
| on:
Applebot
I am mega curious how does Apple circumnavigate problem of getting flagged for crawling website. Doesn't CDN cloudfare eventually flag them?
If proxies, what is their source for proxies.
ricardo81
on Sept 1, 2020
[–]
Likely whitelisted as a 'good bot' in Cloudflare. If a bot isn't well known, doesn't respect robots.txt or crawls too fast then much more likely to get flagged, but applebot are none of those.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
If proxies, what is their source for proxies.