Are scrapers written on a per-website basis? Are there techniques to separate co... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		givemeethekeys on Feb 20, 2024 \| parent \| context \| favorite \| on: Web Scraping in Python – The Complete Guide Are scrapers written on a per-website basis? Are there techniques to separate content from menus / ads / filler / additional information, etc? How do people deal with design changes - is it by rewriting the scraper whenever this happens? Thanks!

spaniard89277 on Feb 20, 2024 | [–]

Yeah. I managed to abstract a bit the structure but in the end websites change.

staticautomatic on Feb 20, 2024 | [–]

Yeah it’s often gonna be a per site, lots of xpath queries, email me when it breaks kind of endeavor.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact