Scraping at Scale Without Getting Banned

If you are collecting at volume and keep hitting limits, the fix is almost always to request less and to request it more evenly - not to disguise the requests better. Scale on the legitimate side of web scraping is an engineering result: pacing and concurrency control, incremental collection so you re-fetch almost nothing, and schedules that keep your traffic small next to the target site's normal load. That is how one of our property projects sustains 230,000 rows a day across five European portals, and how we monitor 150K Allegro.pl EANs daily for the lowest price.

Pacing and concurrency are the single biggest lever

Look first at your request rate and concurrent connections per host. In most projects we see, blocks correlate with those two numbers far more than with anything else in the stack. A crawler that opens dozens of parallel connections to one domain looks like a problem to any operations team, whatever it says about itself. The same crawler, held to a handful of connections with a short delay between requests, usually goes unnoticed.

Rules that hold across targets:

Budget per host, not per job. If ten workers share a target, they need one shared rate limit, not ten individual ones.
Treat 429 and 503 as instructions, not as noise. Back off, and keep the lower rate afterwards. A retry loop that fires harder after a rejection is the fastest route to a permanent block.
Cap the burst, not just the average. A daily average of one request per second means nothing if it arrives as 20,000 requests in ten minutes.
Shed load automatically. When error rates rise, the crawler should cut its own concurrency instead of waiting for a person to notice the next morning.

The right number depends entirely on the target. A large marketplace absorbs traffic that would visibly load a regional property portal on modest hardware. There is no universal safe rate, and anyone who quotes you one has not looked at your targets.

Aggressive collection is usually an engineering failure, not a necessity

When a team tells us they have to hammer a site to get their data, the volume is nearly always self-inflicted. The usual causes are boring:

Re-crawling the entire catalogue every day when only a small share of records changed.
Re-fetching pages in order to re-parse them, because the raw responses were never stored.
Two or three teams inside the same company scraping the same target separately.
Fetching detail pages for records that the listing page already answered.

Fix those and the request count usually falls far enough that the pressure which made evasion look necessary goes with it. It is also what makes ongoing maintenance affordable: fewer requests means fewer moving parts to debug when the target changes its markup. If your blocks started recently and nothing about your volume changed, the cause is more likely a layout or protection change on the target than your rate, and that case is covered in what to do when a scraper keeps getting blocked.

Cache and collect incrementally so you request less

The cheapest request is the one you never send. Incremental collection is the difference between a project that scales and one that fights its targets every night.

Store the raw response, not only the parsed row. When you add a field or fix a selector, you re-parse from storage. Zero new requests.
Use conditional requests where the target supports them, and honour cache headers instead of ignoring them.
Let the listing pages do the filtering. Most catalogues expose price, date, or a status flag in the list view. Fetch the detail page only when something you care about moved.
Use sitemaps and change feeds when they exist. They are published precisely so that crawlers do not have to guess.
Separate the backfill from the refresh. A one-off historical pass and a daily delta are different jobs with different rates.

PropertyGuru Malaysia is a clean example: 25K new listings weekly, a job scoped to what is new rather than to re-reading the whole board. Backfills are the same logic in reverse. iHerb ran as 250,000 products end to end in 3 days and Zolo.ca sold listings as 60K rows in 2 days, both finite passes rather than open-ended daily crawls.

Schedule around the target site, not around your own convenience

Two projects can pull identical volumes and look completely different to the site owner. Concentrated traffic during a shopping peak is a load event. The same volume spread across quiet hours is background noise. On daily projects such as Booking.com end to end - hotels, flights, cars, attractions - or Opodo flights with full filter logic, the run window is a design decision, and we set it around the target rather than around when the report is convenient to read.

Be explicit about which hours are off limits, what your ceiling is per hour rather than per day, and what happens when a run overruns. A job still running when the next one starts will quietly double your rate against that host, which is a common way a well-behaved crawler becomes an aggressive one overnight.

Robots directives, published limits, and identifying yourself

Read robots.txt and follow it, including crawl-delay where it is set. Respect published rate limits and the terms that cover automated access. Send a user agent that identifies the crawler and gives a way to reach a human, and make sure someone actually reads that inbox. If a site operator asks you to slow down or to stay off a path, the answer is to slow down or stay off it, immediately and permanently.

This costs less than people expect, and being reachable is much of what separates a collector that runs for years from one that gets blocked and stays blocked. Our position on public data, personal data, GDPR and CCPA posture, confidentiality, and the specific things we will not do is set out on the compliance page, worth reading before you brief any vendor, including us.

What we do not do, and when to ask for a licence instead

ScrapeIt does not bypass logins, paywalls, or CAPTCHAs. That is a line, not a negotiating position. If a vendor offers to defeat those protections for you, treat it as a risk signal rather than a capability: you inherit the exposure, the provenance of the data becomes impossible to defend to your legal team or your customers, and the pipeline is fragile in a way no maintenance contract can fix.

The consequence is that scale has to come from engineering instead. 230,000 rows a day across five European portals, standardized into one schema, is a pacing, deduplication and scheduling problem. 150K EANs monitored daily on Allegro.pl for the lowest price is a scheduling and delta problem. Neither is an evasion problem, and neither would still be running if it had been solved that way. The same constraints apply to your own crawlers, so build versus buy is largely a question of who carries that discipline over time.

Sometimes public collection is the wrong tool and you should not hire anyone, us included, to do it. Go and negotiate with the site directly when:

The data sits behind a login or a paywall. There is no compliant scraping answer here. There may well be a commercial one.
You need near-real-time completeness. If your business breaks when a record is thirty minutes stale, you need a feed, not a crawler.
The site sells this data as a product. A licence is usually cheaper than the arms race, and it comes with a support contact.
Your required volume is a meaningful share of the site's traffic. At that point you are an infrastructure cost to them, and a conversation is more durable than a crawler.
An official API already covers the fields you need. Use it. Scraping around an available API is work you pay for twice.

Where public collection is the right tool, the design is decided by volume, refresh frequency, number of targets, and how stable those targets are - the same variables behind what a project actually costs. Send us your target list and volumes through the contact form and we will tell you whether those numbers work at a sane request rate, or whether you should be asking that site for an API instead.