The Hidden Costs of In-House Web Scraping

Your engineer is probably right - a working scraper for one site is a week of work, sometimes an afternoon. The cost is not in the first version, it is in the second year: the layout changes nobody scheduled, the proxy bill nobody forecast, the anti-bot escalation that arrives without warning, the normalization work that turns five sources into one usable table, and the fact that the person doing all of it was hired to build your product.

The first version is the cheapest part

Building a crawler against a stable page is a solved problem. Fetch, parse, write rows. Any competent backend engineer can do it, and the demo will look convincing because it runs once, against today's HTML, on a laptop IP, with no schedule and nothing downstream.

What the demo does not include:

Running on a schedule, unattended, when the engineer is asleep or on holiday
Detecting that today's run returned far fewer rows than yesterday's, before someone downstream builds a decision on it
Handling the target site's redesign, which will happen and will not be announced
Pagination edge cases, region-specific variants, listings that render differently for logged-in users
Retries, backoff, partial-failure recovery, and the state to resume from

Each of these is small. Together they are the difference between a script and a data pipeline, and that gap is where in-house projects quietly consume engineering months. What the ongoing work actually involves is covered in web scraping maintenance.

Maintenance is not an event, it is a rate

Selectors that were stable for six months break in an afternoon because a marketing team shipped a new card component. Nothing errors out - the scraper runs happily and returns empty fields, which is worse than a crash because a crash gets noticed.

The honest framing: maintenance cost scales with the number of sources, not with the volume of data. One site refreshed daily is manageable. Twelve sites refreshed daily means something is broken most weeks, and the queue never reaches zero. The engineer who built the crawlers becomes the only person who can fix them, and every new source is another thing only they understand.

Budget this as a standing percentage of an engineer's time, not as a one-off. If you cannot say what that percentage is, you have not costed the project.

Infrastructure and the anti-bot arms race

Proxy spend is the line item that surprises people, because it is invisible until you scale: local testing costs nothing. Production at volume means residential or datacenter proxy pools, and those pools are usually priced by traffic rather than by row - which means image-heavy pages and JavaScript-rendered sites cost multiples of what a plain HTML listing costs. One of our property projects pulled 226K listings and 3.4M images from Immobilienscout24 across Germany and Mallorca; the images alone dominate the bandwidth profile of a job like that.

Then there is headless browser infrastructure. Sites that render client-side need a real browser per request, which turns a cheap HTTP job into a memory-hungry fleet. Add session management, fingerprint handling, CAPTCHA flows, and rate limits tuned per site so you are not hammering a target you depend on.

The arms-race part is what makes this hard to budget, because anti-bot posture is not static. A site that was trivially accessible in March can deploy a new protection layer in April, and your working pipeline is now a support ticket with no ETA. If you are already there, what to do when a scraper keeps getting blocked covers the practical options.

The hard part is usually normalization, not collection

This is the cost most in-house plans miss entirely, and it is often the largest one. Getting rows out of a site is mechanical. Making rows from different sites mean the same thing is a design problem that does not have a library.

We standardized five European property portals into a single schema at 230,000 rows a day. The collection work was the smaller half. The real work was reconciling what each portal means by a field: floor area measured differently, price that sometimes includes fees and sometimes does not, property types that do not map one-to-one, addresses at different granularity, currencies and units that must be resolved before anything can be compared. Do that wrong and you get a clean-looking dataset that produces confidently wrong answers.

Schema drift makes it ongoing rather than one-time. A source adds a field, renames a category, splits one attribute into two. Your unified schema now has a source that no longer fits, and every downstream consumer inherits the inconsistency. Someone has to own that mapping permanently. If you want to see what standardized output looks like, there is a real extracted data sample.

Data quality checks nobody budgets for

A scraper that fails loudly is a good day. The expensive failures are silent: a field that starts returning nulls, a price parsed as a string, duplicates from a pagination bug, a category page that quietly returns fewer results because of an A/B test you are not aware of.

Catching these needs a validation layer that nobody puts in the original estimate:

Row-count deltas against the previous run, with thresholds per source
Field-level completeness checks - how many rows actually carry a price, an address, an image
Type and range validation, so an absurd price gets flagged rather than stored
Deduplication logic that survives listings appearing under multiple URLs
Alerting that reaches a human, and a human who is responsible for answering it

That last item is the on-call burden. Scheduled data has a delivery time, and if the pipeline breaks overnight the business either waits or someone gets paged. Small teams rarely want to formalize an on-call rotation for a crawler, so it defaults to whoever built it, informally, forever.

When to build in-house, and what the alternative costs

Not every case belongs with a vendor. Build in-house when:

The extraction itself is your competitive advantage. If your product is the data pipeline - if the way you collect and interpret a source is the thing customers pay for - do not outsource your core.
You have one stable source and modest volume. A single site, a well-behaved structure, no aggressive anti-bot layer, a few thousand rows a week. That is a script, not a project, and paying a service for it is overkill.
You already have a data engineering team with real spare capacity. Not "an engineer who is interested" - a team that already runs scheduled pipelines, owns monitoring, and has an on-call rotation. For them the marginal cost of one more pipeline is genuinely low.

Outside those cases the arithmetic usually goes the other way, and faster the more sources you have. We work through the comparison in detail in build vs buy for web scrapers.

With a managed service you name the target sites and the fields; we build the crawlers, run them, maintain them when the sites change, and deliver structured data on your schedule as CSV, JSON, XLSX, into your database, or through an API. There is no infrastructure on your side and no code in your repo. Plans start from EUR 199 - see how web scraping pricing actually works for what drives the number up or down.

The comparison to make is not "vendor invoice versus zero". It is the vendor invoice versus a fraction of an engineer's salary, plus proxy and browser infrastructure, plus the delivery your roadmap gives up. That last cost never appears in a spreadsheet and is usually the biggest: an engineer maintaining crawlers is an engineer not shipping the product customers buy.

Speed matters when a project is time-boxed. iHerb supplements ran 250,000 products end to end in 3 days; a Switzerland private-listings feed into a client CRM went live in 7 days at 85K rows a day. Those timelines are hard to match while also learning proxy management from scratch.

If you want to sanity-check the estimate your team gave you, send us the sites and the fields and we will tell you what it takes - including when the answer is that you should build it yourself. Request a quote.