What Web Scraping Maintenance Actually Involves
A scraper is not a delivery, it is a system that has to keep agreeing with a website you do not control. Maintenance is most of the real cost of data collection, and it is the part quotes usually leave out.
What actually breaks
- Redesigns. The most common cause. One source in our portfolio went through several major rebuilds over the years.
- Changes to how the site handles automated traffic. On that same source, significant changes more than once.
- Markup drift. Not a redesign, just a field that moved or an element that got renamed. Small, frequent, and easy to miss.
- Structure of results. Pagination changes, a filter behaves differently, a section moves behind an interaction.
- Volume. What works at a thousand rows behaves differently at 200,000 a day. In one project the central difficulty was sustaining 150,000 lookups a day without interruption.
Why the failures are quiet
A collector that stops is easy to notice. The expensive failure is the one that keeps producing output: a filter that silently stopped applying, a category that dropped out, a field that is now empty for half the rows. The file arrives on time and looks normal in a spreadsheet preview.
This is why maintenance is not only repair work. It is checking that the output is still what it claims to be, run by run, and that means knowing what normal looks like for that source.
What monitoring means in practice
Comparing today's run against the shape of previous ones: row counts in the range they should be, fields populated at the rate they should be, sections all present, values in the formats they were yesterday. When something drifts, the collector is fixed before the next scheduled delivery rather than after somebody downstream notices.
On projects with several sources this matters more, not less. When five portals feed one structure, a change in any one of them can quietly distort the combined dataset.
Who does it in a vendor project
We do, and it is not billed as an extra. On an ongoing project, keeping the collector working through redesigns and access changes is what the monthly plan pays for. The client's side stays the same: the dataset arrives in the agreed shape on the agreed schedule.
Support response time in our plans is 48 to 72 hours. Plans start at 169 EUR per month with a limit of 100,000 rows per month and run to 549 EUR per month, with larger or unusual projects quoted individually. Current tiers are on the pricing page.
Why this is the argument for outsourcing, not the argument against
Building the first version is a known quantity. Maintenance is open-ended: it never finishes, it arrives on the source's schedule rather than yours, and it needs someone who remembers how this particular collector works. That is a poor fit for a team with its own roadmap and a good fit for someone whose whole job it is.
See the case studies for projects that have been running through exactly this, or custom development if you need something built and then kept alive.
Different failure, same morning: Scraper Broke After a Site Redesign.
Running at volume? Scraping at Scale Without Getting Banned.
Frequently asked questions
What actually breaks in a running scraper?
Four things, in roughly this order: the target changes its markup, the protection layer changes, pagination or filters move, and the output schema drifts because a field the source used to publish quietly disappears. None of them are exotic; all of them are unscheduled.
Why do scraper failures stay quiet?
Because a scraper that returns something looks like a scraper that works. Fewer rows, an empty column or slightly stale values do not raise an exception, do not stop the pipeline and do not page anyone. The failure surfaces downstream, in a report someone already acted on.
What does monitoring mean in practice?
Not uptime checks. It means asserting on the data: row counts inside expected bounds, field fill rates per column, and a small set of known records whose values are checked on every run. Anything that only tells you the job finished is not monitoring.
Is maintenance an argument against outsourcing?
It is the argument for it. Maintenance is the largest recurring cost in any scraping programme and the one that never appears in a build-versus-buy spreadsheet, because it is unplanned work landing on an engineer who was doing something else. On a managed service it is inside the fee.
Have a target list and a deadline?
Send the sites and the fields you need. You get a scoped answer and a price, not a discovery call.
