Members-Only Inventory Unlocked - How We Scraped 200K Rows of Fashion Catalog/Day

A client came to us with a short brief: collect the complete BestSecret.com catalog, and refresh all of it every single day. BestSecret is an online fashion retailer built around premium and luxury labels sold at discounted prices, which is exactly the kind of assortment that does not sit still. Almost all of the difficulty in that brief was hiding in the word "complete". The work ran from 2021 to early 2025.

Sources: bestsecret.com, Zalando Lounge, the Scotch & Soda outlet
Coverage: BestSecret's full inventory, Zalando Lounge, Scotch & Soda in three countries
Frequency: daily, a full run each time, from 2021 to early 2025
Volume: roughly 200,000 BestSecret rows per day, up to about 290,000 in a single day
Delivery: JSON files in the client's AWS S3 bucket, due before 03:00 UTC from January 2024
Setup and launch: 4 working days

The catalog lives behind a members-only door

BestSecret is not an open storefront. The inventory is visible to club members only, and membership is not something you sign up for - it comes by invitation. Without a valid member session, a crawler does not see a thin version of the store; it sees nothing at all.

We ran the collection under our own member accounts, registered through invitations. When access changed in November 2023, the run was going again the same day. The client never ran any part of the collection themselves.

A site that changed shape more than once

Over the years BestSecret went through several major updates, and a major update is rarely cosmetic from a scraper's point of view: page structures move, identifiers change, and the anti-bot systems changed too. In May 2023 the site's protection started ending sessions right after login; we rewrote the sign-in step and the run was back at its usual pace the same day. Blocking images, fonts and third-party analytics also roughly halved the traffic of a full pass.

It was a running service: when the site shifted, we adapted the collector, and the client kept getting the same dataset on the same schedule.

Zalando Lounge, a second members-only store, and Scotch & Soda

From the first deliveries in 2021 the daily files also covered Zalando Lounge, which likewise shows its products to signed-in members only. Its sales last a few days, so a file two or three days old is already out of date there.

The Scotch & Soda outlet went offline in September 2023 and came back at the end of November on a completely rebuilt site, so we wrote a new collector, sent a 75-product sample, 25 for each of the three countries the client needed, and shipped the first full dataset on 15 December 2023.

In January 2024 the client asked for all three files in their S3 bucket before 03:00 UTC, two to four hours earlier than before, in time for their early-morning processing. We moved every run earlier: Zalando Lounge and Scotch & Soda took three to four hours, and the long BestSecret pass was rescheduled into the same window.

Making a 200,000-row day unremarkable

The daily dataset came to about 200,000 rows. A full pass over the entire inventory had to start, finish, and be usable inside a single day, then run again tomorrow; a missed day was a permanent hole in the client's history of the catalog. A good run in March 2023 gathered 260,000 rows in 11 hours.

So the engineering goal was consistency rather than raw speed, including through the client's own changes. When they moved their data warehouse to a new AWS account in May 2023, we wrote every daily file into both the old and the new bucket from early June and switched the old one off only in late July, so the files kept arriving throughout the move.

What the client actually had

From 2021 to early 2025 the client had daily access to the full BestSecret inventory - approximately 200,000 rows per day from a catalog closed to anyone outside the club - plus Zalando Lounge and the Scotch & Soda outlet, in their own S3 bucket, without maintaining scraping infrastructure, handling redesigns or dealing with access questions. The client worked with the data. We kept the pipe open.

If you need a members-only catalog collected on a fixed schedule, see our BestSecret scraper, read about e-commerce data collection, or contact our team.