You get files, not a tool to operate. Output comes as CSV, JSON, XLSX or a REST API, with the schema agreed before the first crawl and held stable afterwards so your loaders do not break.
Schedules run daily, weekly, or on a tighter cycle for a watchlist of lines where price and stock matter most. Delivery goes to S3, Google Cloud Storage, SFTP, a database you nominate, or an endpoint we host.
Every run is validated before it ships: product id and product code present, price parsed as a number, currency GBP, variant rows attached to a parent, and volumes compared against the previous run so a silent partial crawl is caught rather than delivered.
Each John Lewis product page becomes a flat row, and we keep the site's own naming so the output stays auditable against the live page.
A single crawl of John Lewis gives you a catalogue. A schedule gives you the retail behaviour behind it.
Price history comes first. The page shows a current price and a previous price, but not the path between them, and once a promotion ends the old figure is gone. Only your own series tells you how deep the cut went, how long it held, and which lines were discounted twice in a season.
Stock comes second. John Lewis states that shop stock figures on its site and app are refreshed every two hours, so availability is a genuinely moving field rather than a static flag. Watching it per colour and per size shows which sizes sell through first and which never move at all.
Range rotation is third and least visible. Products leave the range without a redirect trail you can read from outside. Comparing today's sitemap shards against yesterday's is how you catch a delisting on the day it happens, and how you separate a genuinely new line from an existing product that simply gained another colourway URL.
Seasonal clearance follows the UK retail calendar, so the special offers grids swell and empty around the January and mid-summer sales and again at Black Friday. Crawling only in quiet weeks will understate discount depth.
John Lewis trades as a UK department store across fashion, home, furniture, electricals, beauty and nursery, alongside own labels John Lewis & Partners, ANYDAY, KIN and AND/OR. Prices are quoted in GBP and the catalogue is built for a British audience, so the site reads as a clean sample of UK mid-market retail rather than a global feed.
A record is a product, not a single sellable item. Product URLs end in a numeric id after /p, as in /john-lewis-anyday-linen-blend-shirt/p6499247. Colour variants take their own path segment ahead of that id, so the same product is reachable as /product-slug/mustard/p6499247 and /product-slug/navy/p6499247. Deduplicating on the numeric id rather than on the URL is the first thing a crawler has to get right.
The page also carries a separate eight digit product code, the number John Lewis asks customers to quote when checking shop stock or reading a receipt. In the markup we have seen it sits in the description block under a description:code test id, and it is the field that joins web data to till and warehouse records.
Listings are faceted in Endeca style. Category landing pages sit at /electricals/c500001 and /women/c50000298, while filtered grids sit under /browse/ and end in /_/N- plus a dimension value, with each extra refinement appended after a Z, as in /browse/special-offers/click-collect/_/N-eeoZ1z0ktrg. Brand pages use the same scheme under /brand/.
Get a Quote
Developers
Customers worldwide
Pages extracted
Hours saved for our clients
€199 / one-time
setup fee - included
€169 / mo
setup fee €499
€229 / mo
setup fee €499
€349 / mo
setup fee €499
€549 / mo
setup fee €799
John Lewis reinstated its "Never Knowingly Undersold" promise on 9 September 2024, two years after withdrawing it, and now matches branded products against a fixed list of 25 named UK retailers including Argos, Currys, Boots, M&S and Next. That turns the site into a reference point rather than another shop. If you sell into that set, a John Lewis price is a price you will be measured against.
Brands supplying John Lewis use the rows for RRP and MAP compliance, checking that a listing shows the agreed price, the right colourways and the correct guarantee wording. Buying teams at competing retailers watch assortment instead: what enters the range, what moves to clearance, and how ANYDAY is priced against the John Lewis & Partners line above it.
Marketplace sellers and comparison sites need the same rows for feed enrichment, since the eight digit product code and the specification list give enough to match a John Lewis listing against their own catalogue. Analysts treat the range as a proxy for UK mid-market demand, because own-label and third party brands sit on the same shelf.
Scraping supplement products from iHerb.com with full details, including descriptions and packaging variations.
Learn More
Daily scraping of lowest prices for 150K products on Allegro.pl to support marketplace pricing and margin optimization.
Learn More
Regular monitoring of Ralph Lauren clothing, footwear, and accessories sold across Amazon subdomains: AE, DE, ES, FR, IT, NL, PL, UK.
Learn MoreLearn how to use web scraping to solve data problems for your organization
If you sell online, run a marketplace, or advise e-commerce clients, you already know why eBay matters: it’s one of the few places where big retailers compete side by side with thousands of small merchants and private sellers.
E-commerce teams do not just need “some” competitor data anymore. They need a continuous stream of real prices, discounts, stock levels, reviews, and seller behavior from the platforms that actually shape their markets.
Amazon provides valuable information gathered in one place: products, reviews, ratings, exclusive offers, news, etc. So scraping data from Amazon will help solve the problems of the time-consuming process of extracting data from e-commerce.
We are a managed service. We build the crawler, run it on your schedule, watch it when John Lewis changes its markup, and fix it without you filing a ticket. You never touch a proxy pool or a parser.
UK department store catalogues are heavy on variants and light on stable identifiers, and John Lewis shows both traits. We scope a pilot against your real field list and a sample of your own SKUs before anyone commits to a schedule.
Yes. We take the current and previous price from the price block on each product page, where John Lewis marks them up as price-now and price-prev, and we keep both the numeric value and the displayed GBP string. Price differs by colour and size on many lines, so we return a row per variant rather than a single price per product. On a schedule the run also stamps the date each price was observed, which is the only way to reconstruct a promotion after it has ended.
There are two numbers and they are not the same. The URL carries a numeric product id at the end of the path after /p, as in /john-lewis-anyday-linen-blend-shirt/p6499247. Separately, the description block shows an eight digit product code, and that is the number John Lewis asks customers to quote when checking stock in a shop or reading a receipt. We return both. The eight digit code is usually the better join key against supplier or EPOS records.
Not one you can license and build on. The site's own listing pages are assembled from an internal endpoint at /standard-plp/api/product-chunks, which public technical write-ups document as taking page, chunk, facetId, sortBy, price and priceBands parameters and returning a listing page in eight chunks. That is an internal implementation detail rather than a supported interface, and it changes without notice. We deliver a stable API of our own on top of the extracted data, so your integration does not move when John Lewis reorganises its front end.
It needs real engineering. A plain HTTP client frequently gets no usable response from www.johnlewis.com at all, and listing pages are assembled client side in eight chunks, so a naive fetch of page one returns a fraction of the products. robots.txt also disallows /search entirely and blocks faceted N- URLs beyond a set refinement depth, so a crawl has to be planned around the sitemap and the permitted /browse and /brand grids. We run headless browsers, keep request rates modest, and work within those directives.
Where the site exposes it, yes. John Lewis publishes shop level availability against the eight digit product code and states that those figures are updated every two hours, so we schedule to match rather than polling faster than the source refreshes. Online availability is simpler: we record whether each variant is orderable and whether the page has fallen back to its out of stock state with the "Email me when available" link. Both are captured per colour and per size.
Step 1 - Make a Request
You share your needs, expectations, and desired timeframe. We’ll suggest the best solution based on your request and budget.
Step 2 - Configuring Custom Web Crawlers
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Step 3 - Collect and Deliver
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Step 4 - Maintain and Support
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Scrapeit Sp. z o.o.
10/208 Legionowa str., 15-099, Bialystok, Poland
NIP: 5423457175
REGON: 523384582
