The comment thread under an Ars article is frequently more informed than the article, and it is the part most collectors throw away.
Plans from €169 a month · Free project assessment · Reply within 1 business day
CSV · Excel · JSON · XML · FTP · SFTP · Amazon S3 · Google Cloud
API, JSON feed or CSV, XLSX and JSON files
ScheduleOne-time, monthly, weekly, daily or 3 times a day
Ars Technica pages, filters, fields and a schedule
Built, monitored and repaired by our team
API, JSON feed or CSV, XLSX and JSON files
ScrapeIt runs the collector as a managed service. You name the sections, products or vendors; we build the pipeline, reassemble multi page articles, collect threads with structure intact and hand back CSV, JSON, Excel or a push into your warehouse with entity extraction applied across both layers.
Cadence is moderate. Articles are long and published at a measured pace, but threads keep growing after publication, so we re-collect discussions on a schedule rather than capturing them once at publish time.
The dataset is metadata, entity extraction and publicly rendered text from articles and threads. Handles are identifiers, not people. We collect only publicly available data - everything a visitor can see on Ars Technica - and we collect it legally.
The article record covers canonical URL, headline, standfirst, section, author, publication timestamp, last updated timestamp, item type and the article identifier.
Body text is collected in full where publicly rendered, with word count, because on a long-form source the length is a meaningful attribute rather than a side effect. Multi page articles are reassembled into one record instead of arriving as fragments.
Comment threads are collected with structure intact: comment identifier, parent, author handle, timestamp, depth and text. Flattening the thread destroys the argument, and on this source the argument is a substantial part of what a client is buying.
Entity extraction runs over both the article and the thread. Products, vendors and technologies mentioned are normalised to identifiers, because the mention that matters is frequently in a reply comparing your product with an alternative rather than in the headline.
Commenter handles are treated as identifiers rather than people: no attempt to link them to real identities and no cross site profiling, and we say so before anybody asks.
Thread analysis is the output most vendor clients want. Which objections recur, which comparisons keep being drawn, how sentiment in the thread differs from the article's framing, and which technical claims get challenged by readers who tried it.
Product and vendor mention extraction runs across both layers and is normalised so that a product appearing under three names is one entity. In threads, naming is informal and inconsistent, which makes the normalisation more valuable here than on edited text.
Corpus building for technical analysis is a distinct use. Because articles are long and the subject coverage is deep, this source contributes disproportionately to a technical corpus relative to its item count, and clients doing topic modelling usually weight it accordingly.
Historical backfill is practical because URLs are stable and section pagination reaches back. Threads on older articles generally remain available, which means the historical discussion is collectable rather than lost.
Ars Technica publishes technology and science journalism at unusual length and depth, aimed at a readership that works in the field. Articles run long, cover implementation detail that most technology press skips, and attract comment threads from people who do the thing being written about.
That readership is the distinguishing feature for data work. Coverage here signals something different from coverage in a general technology outlet: it reaches the engineers and administrators who make purchasing recommendations rather than the executives who sign them.
Section fronts carry ordered selections, and each article page carries the headline, standfirst, author, section and both timestamps. Volume is moderate by technology press standards because the articles are long, which makes complete collection of a subject achievable rather than aspirational.
Comment threads are extensive and threaded, and they are where a large share of the technical substance sits - corrections, benchmarks from readers, operational experience that the article did not have room for.
Delivery: CSV, JSON or XLSX files, our API or a JSON feed. MCP server on request.
Full plan detailsFor most media sources the article is what you collect and the comments are noise. Here that is reversed often enough to change the design.
The readership is technical, so a thread under a product article typically contains operational experience, benchmark numbers, failure modes and direct comparisons with alternatives. For a vendor, that is unsolicited and unusually specific feedback from exactly the people who evaluate the product, arriving faster and more honestly than any survey.
Collecting it properly means preserving structure. A top level objection with thirty replies is a different object from thirty separate comments, and any summarisation that ignores depth will misread which criticism actually landed. We keep the tree.
The second reason is length. Long-form articles carry detail that short technology news does not, which makes this a better corpus for anything doing topic or technical analysis rather than headline counting. Multi page pieces have to be reassembled or the corpus is full of fragments.
The third is audience weighting. Coverage here reaches practitioners rather than executives, and a media dataset that treats all technology outlets as interchangeable misses that this one moves a different group of people.
Learn how to use web scraping to solve data problems for your organization
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
If you specialize in machine learning, you need to feed large amounts of data to the algorithms. Web scraping is the easiest and the most efficient method of collecting the data from all over the Internet.
ScrapeIt is a managed extraction company, not a tool you have to learn. Our team builds the pipeline, watches it as templates and comment systems change, and repairs it before your threads stop arriving.
You see a sample first, in your format, over the products or vendors you actually track, with a full thread included so you can judge whether the discussion is worth what it costs to collect.
Still unsure? Write to info@scrapeit.io
Because on this source they frequently carry more technical substance than the article: operational experience, benchmark numbers, failure modes and direct comparisons with alternatives, from the people who actually evaluate the product. It is unsolicited feedback from exactly the right audience, arriving faster and more honestly than a survey.
Yes: identifier, parent, author handle, timestamp, depth and text. A top level objection with thirty replies is a different object from thirty separate comments, and any summarisation that ignores depth will misread which criticism landed. Flattening the tree destroys the argument.
By reassembling them into one record. Collected page by page they arrive as fragments, word counts are wrong and any text analysis treats one article as several. On a long-form source that error is large.
We treat handles as identifiers, not as people. No attempt is made to link them to real identities and we do not build cross site profiles. The analysis clients actually need - which objections recur, which comparisons get drawn - requires none of that.
On a schedule after publication, not once at publish time. Threads keep growing for days, and a discussion captured at the moment the article went live is essentially empty. We re-collect until the thread settles.
You share your needs, expectations, and desired timeframe. We'll suggest the best solution based on your request and budget.
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Tell us which sites, which fields and how often. A couple of lines is enough to start.
Ars Technica data from €169/month. Free project assessment, reply within 1 business day.