Most of the duplication in a news dataset starts here. Collect the wire itself and forty republished copies stop being forty stories.
ScrapeIt runs the AP collector as a managed service. You name the hubs and subjects; we build the crawl, run it on your cadence and hand back CSV, JSON, Excel or a push into your warehouse. Where you also want pickup measurement we run the outlet side in the same pipeline and deliver the matched table rather than two datasets you have to join yourself.
Cadence is per source. Hubs on a live story justify frequent passes; quiet subjects run hourly or daily. Requests are paced deliberately rather than bursted.
We collect what the public site renders and no more. AP journalism is copyrighted and licensed commercially, so this dataset is built for analysis and never for republication; if redistribution is what you need, the route is a licence from AP and we will say so plainly. Bring the use case to your own counsel before the project starts, and we scope the work to what counsel signs off.
The story record covers canonical URL, headline, summary line, hub and topic assignment, byline and dateline, publication timestamp, last updated timestamp, item type, and the story identifier used in AP URLs.
The dateline is a field most collectors throw away and it is unusually useful here. Wire copy carries the place the reporting came from, which gives you a geography for the story that no section path provides. We keep it as written and parsed into place and country where it resolves cleanly.
Both timestamps are kept separately. Agency copy is updated aggressively as an event develops, often many times in a day under one URL, and the gap between first publication and last update is the clearest measure of how live a story still is.
Front and hub observations run alongside: which stories were promoted, in what order and at what time. On an agency site this is a genuine editorial signal, because what AP leads with tends to become what several hundred other outlets lead with a few hours later.
The field that makes this source pay for itself is the syndication key. Every AP story gets a stable identifier in our schema, and the near duplicate detection we run across the rest of your feed points back to it. A republished copy on a regional site arrives already linked to the wire original rather than counted as a separate story.
Hubs are the practical unit of collection here. Rather than crawling everything, most projects follow a set of hubs matching their subjects and read them on a schedule, which keeps the request volume modest and the relevance high. Hub membership is recorded on every row, so the same story can belong to several without being duplicated.
Pickup measurement is the piece clients most often ask for once they see it working. It needs two collections running together: AP for the originals and a set of outlets for the republished versions. We match on text similarity, headline overlap and the dateline, then report first seen time per outlet against the wire timestamp. The output is a table of who carried it and how quickly, which is exactly the question the count of mentions was standing in for.
Datelines support a geographic view that most news datasets lack. Filtering wire coverage by the place it was reported from, rather than by the section it landed in, is a different and often better cut for risk and political monitoring.
Historical depth on the public site is limited compared with a licensed archive. Hub pages paginate back a reasonable distance and article URLs stay stable, so a retrospective run is practical for recent history. For a deep archive with usage rights, the licensed route is the honest answer and we will point you at it.
The Associated Press is a cooperative news agency. Its journalism is written to be republished, and thousands of outlets do exactly that, sometimes verbatim, sometimes trimmed, sometimes under a local byline with a line of agency credit buried at the foot. That is why the same event turns up in a monitoring feed dozens of times looking like dozens of stories.
Its own site is organised around hubs rather than conventional sections. A hub gathers everything on a running subject, and the front carries an ordered selection of what the agency is leading with. Article pages are free to read, there is no meter, and the volume is substantial: the front alone renders tens of thousands of words at any moment.
Machine readable entry points exist and are worth using first. The news sitemap is a sitemap index pointing at a content file of recently published items, so knowing what is new is cheap. Article pages carry structured markup with the headline, publisher and both timestamps.
Separately from the public site, AP runs a commercial content programme with a developer portal for licensed members. If you need the wire feed itself, in full, with rights to use it, that is the correct route and we will say so. What we collect is the public site, and its value is different: it is the reference copy you match everything else against.
Get a Quote
Developers
Customers worldwide
Pages extracted
Hours saved for our clients
€199 / one-time
setup fee - included
€169 / mo
setup fee €499
€229 / mo
setup fee €499
€349 / mo
setup fee €499
€549 / mo
setup fee €799
Ask a monitoring platform how much coverage an event got and you will usually be told a number that is mostly repetition. One agency story about a company can appear on two hundred sites within an afternoon, each with its own URL, its own timestamp and often its own headline. Counted naively, that is two hundred stories and a very misleading chart.
Collecting AP directly is the cheapest way out of that, because it gives you the reference text. Near duplicate detection against a known original is far more reliable than clustering an undifferentiated pile of articles and hoping the groups come out sensibly. Once the original exists in the dataset, the republished copies attach to it and your volume chart starts describing events rather than republishing.
The second use is timing. Because AP is upstream, its publication timestamp is usually the earliest reliable one for a story. Comparing it against when each outlet picked the story up gives you a pickup curve: who ran it within the hour, who ran it the next morning, who never ran it at all. For a communications team that is a much sharper measure of reach than a count of mentions.
The third is rewriting. Outlets do not republish verbatim. They cut, they re-headline, they add a local angle. Holding the wire original next to the published copies is what makes those edits visible, and occasionally that difference is the story.
Learn how to use web scraping to solve data problems for your organization
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
If you specialize in machine learning, you need to feed large amounts of data to the algorithms. Web scraping is the easiest and the most efficient method of collecting the data from all over the Internet.
ScrapeIt is a managed extraction company, not a tool you have to learn. Our team builds the pipeline, watches it as the site changes, and repairs it before your dashboard goes quiet.
You see a sample first, in your format, on the hubs you actually track, with the syndication matching already applied so you can judge it on real data.
If you need the wire itself, in full, with the right to publish it, then yes, and we will tell you so. AP runs a commercial content programme for exactly that. Collection from the public site answers a different question: it gives you the reference copy of each story so the republished versions across your other sources can be matched to it and counted once.
The AP original goes into the dataset with a stable identifier. Across the rest of your feed we run near duplicate detection over the text, compare headlines and check the dateline, then attach each match to the original. Because there is a known reference text rather than an undifferentiated pile, the grouping is far more reliable than blind clustering.
Yes, and it is the most requested output on this source. It needs the wire and the outlets collected together: we report first seen time per outlet against the AP publication timestamp, which gives you a pickup curve rather than a total. Who ran it within the hour, who ran it the next day, who did not run it at all.
Agency copy names the place the reporting came from at the head of the story. It gives the item a geography that the section path does not, which makes filtering by where something happened possible rather than by which desk handled it. We keep it as written and parsed into place and country where it resolves cleanly.
Republication is not what this dataset is for. AP journalism is copyrighted and licensed commercially, and their terms restrict automated collection and reuse. We work only on public pages, never behind a login, we pace requests, and we build for analysis: monitoring, matching, measurement. If you need redistribution rights the route is a licence from AP. Have your own counsel approve the use case before the project starts.
Step 1 - Make a Request
You share your needs, expectations, and desired timeframe. We’ll suggest the best solution based on your request and budget.
Step 2 - Configuring Custom Web Crawlers
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Step 3 - Collect and Deliver
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Step 4 - Maintain and Support
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Scrapeit Sp. z o.o.
10/208 Legionowa str., 15-099, Bialystok, Poland
NIP: 5423457175
REGON: 523384582
