BIG W scraper for prices, tickets and store stock

BIG W product data we extract

A BIG W product page is built around one identifier, the article number. It is printed above the title as Article, repeated in the Specification table and closes the product address after /p/; a colour variant of a garment adds a suffix to it. We use it as the row key, so a product keeps its identity from run to run.

• Identity - article number, product URL, title, brand, the barcode (GTIN) carried by the page, and the breadcrumb from department down to the last category.
• Price - the current price in AUD, the SAVE amount and percentage off, the WAS price with its date on Price Drop lines, the RRP where one is shown, the Originally figure on Clearance lines, the Price after promotion on introductory offers and the unit price printed on everyday goods.
• Tickets - Special, Clearance, Price Drop, Big Deal, New, Flash Deal and multi-buy offers, plus the Lowest price in 14 days call-out that BIG W shows on Marketplace products.
• Seller - the sold and shipped by line: BIG W itself or a named Marketplace seller, with the returns wording that goes with each.
• Fulfilment - Free delivery and Limited free delivery flags, the Delivery only, Online only and Not available in store markers, delivery and pick up availability, and whether a delivery fee is added.
• Content - description, the Product Features list, the Specification table (brand, dimensions, weight, recommended age on toys) and image addresses.
• Ratings - the star rating on tiles and product pages and the Reviews section; review text on request, without reviewer names.
• Loyalty - the Everyday Rewards points a purchase collects.

Listing tiles on category, brand and seller pages repeat the price, ticket, star rating and fulfilment markers, so a price run can read listings alone and leave product pages for a slower cycle. Clothing with colour swatches and a size range is split into one row per option, because BIG W shows availability only once a colour and a size are chosen. Field documentation is delivered with every dataset.

BIG W price tracker: tickets, sellers and store stock

One pass gives a price list. BIG W itself looks back only a fortnight, and only for Marketplace products, with its Lowest price in 14 days label. To track BIG W prices across the whole range we key every run on the article number and keep each price, ticket and seller we see.

• Ticket calendar. BIG W describes a Special as a discount off the shelf price for a short period, usually a couple of weeks. Daily runs show the day a Special starts, how deep it is and the shelf price it returns to.
• Listing coverage. A category page loads 48 products at a time behind a Load more button; brand, collection and seller pages follow the same layout, and the largest departments show a rounded counter instead of an exact total. We walk the tree down to the smallest categories and use the Category, Price, Deals, Brand, Condition and Rating filters as slices.
• Marketplace sellers. Each seller has a page under Marketplace Sellers with its description, categories and range, so one seller, or BIG W-sold lines alone, can be isolated.
• Store stock. With a postcode set, a product page shows pick up, in-store and delivery availability for the chosen store. We repeat the check across the stores you name; the store finder adds each store's address, services and opening hours for this week and the next.
• Collections. Clearance, Price drop, New and Trending, pre-orders and catalogue pages are tracked as lists of their own.