Web scraping for sentiment analysis tends to go wrong before a model reads a single sentence. The scores look plausible, and then someone notices that the same review was counted on several country sites, that many of the "German" texts are machine translations, and that the sample came from the "most helpful" tab.
This post covers the data side: where opinion text lives on the public web, what a review record contains on each kind of site, which fields a model needs besides the text, and what breaks a dataset. For training data in general, see our post on web scraping for machine learning.
The examples come from our published projects. They are catalog and listing datasets, not review collections - one includes product ratings across eight Amazon storefronts - and the decisions that made them usable are the ones a review dataset depends on.
A product review on a marketplace such as Amazon, Walmart, Ozon, Shopee or Flipkart carries a star rating, the review text and the review date; many add a title, a verified purchase label and helpful votes, and the product ID comes from the page. Walmart keeps seller reviews on seller pages, apart from product reviews - see Walmart review and seller data; mixing them puts delivery complaints into a product score. Marketplaces often pool reviews across variants and print the variant - size, color, pack - on each review. Keep it as a column: "runs small" is about one size, not the product.
A Google Play review comes with text, star rating, date, helpful count and the developer reply. Rating and reviews change with the country, so each row needs the country and language it was collected in - see our Google Play data page. Join review dates to the app's update history: the usual first question is which release caused the drop.
Google Maps reviews belong to a place: star rating, review text, the owner reply with its date, and a place ID that keeps the branches of one chain apart. The trap is the date: Google prints a relative phrase such as "a month ago", so each row must carry the collection day. Yelp and Trustpilot reviews hang on a business page in the same way; the Google Maps listing and review fields have their own page.
A hotel review on Booking.com is a score out of ten plus two texts, what the guest liked and disliked; Tripadvisor uses a five-step scale, a title, one text and the date of stay. For travel review scraping, settle the scale and the date first: a March review of a January stay belongs to January in a seasonality study. Listing-level fields are on the Tripadvisor data page.
An employer review on Glassdoor has a rating, a date, the author's role and employment status, and two texts, Pros and Cons. Keep them as two columns: the site has split the polarity for you. We collect what a signed-out visitor sees and never log in; our Glassdoor data page lists those fields.
Reddit has no star rating. Structure replaces it: the comment, its parent, its depth, its score and its community. "Same here, returned mine" means nothing without the comment above it, so comment threads must arrive as a tree, with deleted comments kept as empty markers. See the Reddit data page.
A news article gives a headline, a body, a publication time, a section and an outlet. Sentiment here is about an entity, so the unit is the paragraph that names it, not the article. The trap is agency copy: one wire story republished by many outlets is one opinion, not many.
Which fields should a review dataset contain? Enough to check, slice and trust the scores:
The principle: store the observation, not an interpretation. Our Ralph Lauren monitoring project on Amazon recorded each product's position in search results as the listing page number and the slot number on that page rather than collapsing it into a single number. Reviews work the same way: keep "a month ago" next to the collection date and the score on the site's own scale, and derive tidy columns afterwards.
The Ralph Lauren project covered eight Amazon storefronts - amazon.ae, amazon.de, amazon.es, amazon.fr, amazon.it, amazon.nl, amazon.co.uk and amazon.pl - and the full list of ASINs by color and size; the delivered dataset came to 41,000 rows. Put reviews on that grid: the variants of one product share a review list, and a storefront adds a block headed "Top reviews from other countries", so a naive crawl collects one review many times. Amazon prints the country where each review was written next to its date; keep it as a column. How do you remove duplicate reviews from a dataset? Key every row on the review ID - or, where the page prints none, on text, date and rating together - and keep one copy with the list of places where it was seen.
Many sites show reviews from other markets machine-translated into the local language. Score those and you measure the translation engine; count them and one opinion appears once per language. Keep the original text, its language code and a flag for translated reviews. Syndicated reviews are the retail version: a review collected on the brand's own shop is displayed on several retailers, often with a line saying where it was first posted.
A rating can often be left without text, so the displayed average describes a wider group than your text sample. The opposite failure is text that only looks complete: long reviews fold behind a "read more" control, and a crawler that takes the visible part delivers openings without the complaint. The iHerb supplements catalog project, a dataset of 250,000 rows, had the same requirement: the description in its entirety rather than a truncated preview, because a file of first paragraphs looks complete only in a preview.
Many review pages open on a "top" or "most helpful" sort order, not on the newest reviews, and many sites stop paging after a limited depth. A crawl of the default view is a sample chosen by the ranking, not by you. Read newest first, walk each star filter separately, store the sort order in every row, and compare the stars you collected with the rating breakdown on the page.
The displayed score is not a plain mean of the reviews you can read. Amazon says on its product pages that the star rating is not a simple average and reflects how recent a review is; Glassdoor says recent reviews weigh more in the overall rating. Reviews also get edited or removed. A trend therefore needs dated snapshots of the average, the rating breakdown and the review count. Text can be backfilled; the score displayed on a given day cannot. Our weekly PropertyGuru project - about 25,000 rows per weekly dataset from a rolling seven-day window - runs on the rule its case states: anything posted inside the window but missed is a gap that cannot be repaired afterwards.
A brand search returns other brands. The Ralph Lauren specification included filtering out third-party brands unrelated to Ralph Lauren that surface in search results, because if those rows stay in the table, every figure built on top of it moves. In opinion text the stray rows are subtler: accessories matching the brand keyword, a review about the courier, a thread about a namesake company. Put the filter into the dataset definition.
We extract the data you need from any website to satisfy all your business requirements with 100% accuracy.
Every site describes an opinion its own way: five stars, a score out of ten, a recommended mark on Steam, one text or two, calendar dates or "last week". In our project for a real estate portal, five EU property sites - funda.nl, pararius.com, rentberry.com, rentola.com and zimmo.be, about 230,000 rows per day - are reconciled into one common structure before they reach the client, whose analytics pipeline ingests a single consistent format.
For reviews that structure needs a source column, the rating as printed, its scale, a normalized rating, one row per text part (title, positive, negative, body) and dates in one format. The case names the risk too: a normalization layer that is right on day one but drifts as the sources change is a deferred problem.
Five decisions to settle before the first run:
ScrapeIt is a managed service: we build, run and repair the crawlers, and anti-bot handling - pacing, proxy rotation, CAPTCHA handling, rendering - is part of the service. You describe the sources, the list and the fields; we answer with a free project assessment, and a sample can be ordered before the full run - see what a data sample contains. Runs are one-off or on the schedule you set, and field documentation comes with the dataset.
Scraper development and testing for the eight Amazon storefronts took 4 days and the crawl itself ran in 1 day; the iHerb scraper was set up and launched within 3 working days.
Formats: CSV, Excel (XLSX), JSON, JSONLines, XML. Channels: our API (an endpoint we host), a JSON feed, FTP, SFTP, Amazon S3, Google Cloud Storage, Dropbox, Google Drive, e-mail, a load into your database; an MCP server is available on request. Plans start from EUR 169 a month, or EUR 199 for a one-time extract.
At minimum the review ID, the product ID or place ID, the star rating with its scale, the review date, the collection date, the country site and a language code. Text alone can be scored, but not deduplicated, dated or checked.
We collect only publicly available data - everything a visitor can see on the site - and we collect it legally. Reviewer names and profile links are personal data: we handle personal data under GDPR, leave such fields out by default, never log in and never go behind paywalls.
Yes. Each country site is collected separately, and every row carries the country and the language code it was collected in. A review shown on several country sites can be matched on the review ID and kept once.
That depends on the decision the data feeds: daily around an app release, monthly for a quarterly brand report. Each run should store the average rating and the review count, which cannot be recovered later.
One row per review or comment with the fields above and documentation for each, plus a table of aggregates per product, place or app on each collection date. Scoring stays with your team and your model.
More on getting data out of the sources we work with
Lead scraping is a reliable way to get the right customer contacts to market your product, saves organizations time, and helps you better understand the audience you want to attract. Contacts include emails, phone numbers, or social media profiles.
Using web scraping to extract Google Maps data will help you quickly and efficiently find businesses in any industry, city, state, or region. And the extracted contact information, such as phone or email address, social networks, or links to web pages, will help to contact them.
Thanks to real estate web scraping, you can get exhaustive and up-to-date information about properties. Read this article to get to know the essence of this technology and learn how exactly you can benefit from it.
Let us take your work with data to the next level and outrank your competitors.
1. Make a request
You tell us which website(s) to scrape, what data to capture, how often to repeat etc.
2. Analysis
An expert analyzes the specs and proposes a lowest cost solution that fits your budget.
3. Work in progress
We configure, deploy and maintain jobs in our cloud to extract data with highest quality. Then we sample the data and send it to you for review.
4. You check the sample
If you are satisfied with the quality of the dataset sample, we finish the data collection and send you the final result.
Scrapeit Sp. z o.o.
10/208 Legionowa str., 15-099, Bialystok, Poland
KRS: 0000996672
NIP: 5423457175
REGON: 523384582
