Behind the paywall there is nothing we will take. In front of it there is more than most monitoring setups collect, and it is enough for the questions people actually ask.
Plans from €169/month · Free project assessment · Reply within 1 business day
ScrapeIt runs the collector as a managed service. You name the sections, topics and competitors you track; we build the pipeline, run it on your cadence and hand back CSV, JSON, Excel or a push into your warehouse. Deduplication against the other outlets in your feed is part of the work, not a separate project.
We are explicit about the boundary. We collect what the site renders publicly to an ordinary visitor: headline, summary, byline, section, timestamps, front position and whatever opening text is shown before the meter. We do not use subscriber credentials, we do not defeat the paywall, and we do not deliver full article bodies. Every row carries the paywall flag, so an opening is never counted as a full article.
We collect only publicly available data - everything a visitor can see on the New York Times site - and we collect it legally.
The article record covers canonical URL, headline, summary line, section and subsection, byline as published, publication timestamp, last updated timestamp, item type and the article identifier used in Times URLs.
The paywall flag is a first class field rather than an afterthought. Every row records whether the article was metered at the time of collection and how much text was publicly rendered. That keeps the dataset honest downstream: an analyst can see immediately which rows carry a full public text and which carry only an opening, instead of discovering it inside a model six weeks later.
Both timestamps are kept separately and never merged. The Times updates stories in place, often substantially, and the gap between published and updated is frequently the most interesting number on the row.
Front observations run alongside the articles. Each pass over a section front records the stories on it, their order, the headline shown at that moment and the time of the pass. This is where prominence and silent rewriting become visible, and it is not available from any archive after the fact.
Then the context fields: keywords and topic tags where published, lead image URL and caption, word count of the publicly rendered portion, outbound links, and the collection timestamp on every row.
Section fronts are the highest value surface here and the cheapest to collect. They are fully public, they carry order, and reading them on a schedule builds the prominence history that no archive can reconstruct later. For most clients the fronts plus the news sitemap cover the whole brief.
Topic pages extend the reach backwards. They gather everything the Times has published on a subject, a company or a person, and they paginate a long way. For a retrospective study of how coverage of an organisation developed, that is the practical entry point, and it runs as a one time job separate from the ongoing feed.
Front observations, topic page history and article records are joined on the article identifier used in Times URLs, so the client sees one table rather than three.
Live coverage and interactive features need typing rather than parsing. A live briefing is a stream of entries under one URL, and an interactive graphic has almost no body text at all. Both are recorded by type so that a word count comparison does not quietly treat them as short articles.
The New York Times runs a metered paywall, and the first thing to settle on any project here is what that leaves visible. Section fronts are public and complete: the world, business, technology and politics indexes list what the desk is running, in order, with headlines and summary lines. Article pages render the headline, the summary, the byline, the section, both timestamps and usually an opening portion of the text before the wall appears. The rest is not ours to take and we do not take it.
Our crawler reads the news sitemap for recently published items and the structured markup on article and section pages, so every new item arrives with its headline, section, byline and both timestamps.
Structurally the site is organised by section and by topic, with topic pages that gather everything on a subject and reach considerably further back than the fronts. Newsletters, live coverage and interactive features sit alongside conventional articles and behave differently enough that they are typed rather than lumped together.
The practical consequence is that a Times dataset is a metadata dataset with a text sample attached. For media monitoring, share of voice, section level prominence and revision tracking, that is sufficient.
Get a Quote
Developers
Customers worldwide
Pages extracted
Hours saved for our clients
€199 / one-time
setup fee - included
€169 / mo
setup fee €499
€229 / mo
setup fee €499
€349 / mo
setup fee €499
€549 / mo
setup fee €799
Delivery: CSV, JSON or XLSX files, our API or a JSON feed. MCP server on request.
Full plan details →The reflex is to skip paywalled publications because the body is unavailable. That skips the most influential outlet in the market on a technicality, and it misreads what monitoring actually needs.
Very little of the standard brief requires full text. Did we appear. In which section. How prominently, and for how long. Was the headline changed after publication. Who wrote it. How does our share of coverage compare with the competitor. Every one of those is answerable from the headline, summary, section, byline, timestamps and front position, all of which are public.
Prominence is where this source rewards the effort most. A story on the business front at seven in the morning is a different event from the same story reachable only through a topic page, and the Times moves stories around its fronts through the day. Only repeated observation with timestamps captures that, and it is the part clients most often discover they have been missing.
Revision tracking is the other reason. Times stories are updated in place under the same URL, sometimes with a materially different headline, and the page does not announce it. Keeping the headline at each observation alongside both timestamps makes the edit history visible. For anything with a legal or regulatory edge, that record matters more than the body text would.
Where full text genuinely is the requirement, the paywall flag shows which rows carry it and which carry only an opening, so a thin dataset is never called complete.
Learn how to use web scraping to solve data problems for your organization
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
If you specialize in machine learning, you need to feed large amounts of data to the algorithms. Web scraping is the easiest and the most efficient method of collecting the data from all over the Internet.
ScrapeIt is a managed extraction company, not a tool you have to learn. Our team builds the pipeline, watches it as the site changes, and repairs it before your dashboard goes quiet.
You see a sample first, in your format, over the sections and topics you actually track, with the paywall flag visible so you can judge the coverage before committing.
No. We do not use subscriber accounts and we do not defeat the meter. What we collect is what an ordinary visitor sees: headline, summary, byline, section, both timestamps, front position and whatever opening text renders before the wall, with the article marked as metered. Every row carries that flag, so an opening is never mistaken for a full article.
For most briefs, yes. Whether you appeared, in which section, how prominently and for how long, under which byline, and whether the headline changed afterwards are all answerable from public fields. Full text mainly matters for sentiment scoring and model training, and for those the paywall flag shows which rows carry the full public text.
Yes. The ScrapeIt New York Times API returns headline, section, byline, both timestamps, the paywall flag and front position as JSON from an endpoint we host, refreshed on the schedule you set, so it tells you what sat at the top of the business front at seven this morning. The same data also comes as CSV or XLSX files or straight into your database.
By observation. Each pass records the headline as shown at that moment along with both the published and updated timestamps, so a rewrite appears as a change between two observations. The site does not announce these edits, so the only way to have the history is to have been collecting while it happened.
Yes. We collect the headline, summary, byline, section, both timestamps, front position and the opening text that renders publicly, never with subscriber credentials, and we pace requests. We collect only publicly available data - everything a visitor can see on the New York Times site - and we collect it legally.
Step 1 - Make a Request
You share your needs, expectations, and desired timeframe. We’ll suggest the best solution based on your request and budget.
Step 2 - Configuring Custom Web Crawlers
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Step 3 - Collect and Deliver
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Step 4 - Maintain and Support
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
The New York Times data from €169/month. Free project assessment, reply within 1 business day.
Scrapeit Sp. z o.o.
10/208 Legionowa str., 15-099, Bialystok, Poland
KRS: 0000996672
NIP: 5423457175
REGON: 523384582
