What Web Scraping Maintenance Actually Involves
A scraper is not a delivery, it is a system that has to keep agreeing with a website you do not control. Maintenance is most of the real cost of data collection, and it is the part quotes usually leave out.
What actually breaks
- Redesigns. The most common cause. One source in our portfolio went through several major rebuilds over the years.
- Changes to how the site handles automated traffic. On that same source, significant changes more than once.
- Markup drift. Not a redesign, just a field that moved or an element that got renamed. Small, frequent, and easy to miss.
- Structure of results. Pagination changes, a filter behaves differently, a section moves behind an interaction.
- Volume. What works at a thousand rows behaves differently at 200,000 a day. In one project the central difficulty was sustaining 150,000 lookups a day without interruption.
Why the failures are quiet
A collector that stops is easy to notice. The expensive failure is the one that keeps producing output: a filter that silently stopped applying, a category that dropped out, a field that is now empty for half the rows. The file arrives on time and looks normal in a spreadsheet preview.
This is why maintenance is not only repair work. It is checking that the output is still what it claims to be, run by run, and that means knowing what normal looks like for that source.
What monitoring means in practice
Comparing today's run against the shape of previous ones: row counts in the range they should be, fields populated at the rate they should be, sections all present, values in the formats they were yesterday. When something drifts, the collector is fixed before the next scheduled delivery rather than after somebody downstream notices.
On projects with several sources this matters more, not less. When five portals feed one structure, a change in any one of them can quietly distort the combined dataset.
Who does it in a vendor project
We do, and it is not billed as an extra. On an ongoing project, keeping the collector working through redesigns and access changes is what the monthly plan pays for. The client's side stays the same: the dataset arrives in the agreed shape on the agreed schedule.
Support response time in our plans is 48 to 72 hours. Plans start at 169 EUR per month with a limit of 100,000 rows per month and run to 549 EUR per month, with larger or unusual projects quoted individually. Current tiers are on the pricing page.
Why this is the argument for outsourcing, not the argument against
Building the first version is a known quantity. Maintenance is open-ended: it never finishes, it arrives on the source's schedule rather than yours, and it needs someone who remembers how this particular collector works. That is a poor fit for a team with its own roadmap and a good fit for someone whose whole job it is.
See the case studies for projects that have been running through exactly this, or custom development if you need something built and then kept alive.
