A large share of what CNN publishes has almost no body text, because it is video. Count by word length and the biggest broadcaster in the dataset looks like the thinnest.
ScrapeIt runs the collector as a managed service. You name the sections, topics and editions; we build the crawl, run it on your cadence and hand back CSV, JSON, Excel or a push into your warehouse, with item types separated, editions on every row and live pages collected entry by entry.
Cadence is per front. Main fronts during an active story justify frequent passes; topic pages and archive work run slower. Given the volume, a defined section list is what keeps the cost proportionate.
We collect what the site renders publicly. CNN journalism and video are copyrighted and the terms restrict automated collection and reuse, so the dataset is metadata and publicly rendered text for analysis rather than material for republication. We do not download video files. Bring the use case to your own counsel before the project starts.
The item record covers canonical URL, headline, summary line, section and subsection, edition, byline where one is published, publication timestamp, last updated timestamp and the item identifier used in CNN URLs.
Item type is a first class field and it is the one that saves the analysis. Article, video package, gallery, live page and interactive are recorded explicitly, so a video item with forty words of caption is never mistaken for a thin article. Where a video is the item, its duration is captured instead of a word count.
Front observations run alongside: which items sat on the front or a section front, in what order and at what time. On a broadcaster this matters more than usual, because promotion turns over faster than on a newspaper.
Live pages are collected as a parent record plus their stream of timestamped entries, each with its own text and author where credited. Collapsing a live page into one article throws away the reporting inside it.
Then the context fields: topic tags where published, lead image URL and caption, outbound links, publicly rendered body text and its word count for text items, and the collection timestamp on every row.
Edition handling is set up at the start rather than bolted on. Each front is collected as its own scheduled view, every row carries the edition it came from, and cross edition duplicates are grouped under one story identifier so coverage can be counted once globally or once per market.
Live coverage is the highest value and most often mishandled part of this source. Entries carry their own text, timestamp and author where credited, keyed to a parent that records the entry count and the time of the latest entry. That is what lets you reconstruct what was being reported at a given moment rather than only how the page ended up.
Historical backfill is practical because article URLs are stable and topic pages paginate a long way back. It runs as a one time job separate from the ongoing feed. Front position, as always, exists only from the point where somebody started observing.
Video items are kept with their metadata and duration and marked by type, never dropped. For any share of voice measure on a broadcaster, dropping video removes the part of the output with the largest audience.
CNN is a broadcaster that publishes on the web, not a newspaper with a video section, and the difference shows up in the data immediately. Alongside conventional articles the site carries video packages, photo galleries, live updates pages and interactive features, and several of those contain very little written text.
It also runs more than one edition. The domestic and international fronts promote different stories in a different order, and the same piece can appear on both. A collector that treats the editions as one source double counts; one that picks a single front misses what the other half of the audience was shown.
Structurally the site is arranged into sections with ordered fronts, plus topic pages that gather coverage on a subject. A news sitemap covers recently published items and responds normally, which makes discovery cheap. Article pages carry structured markup with headline, publisher and both timestamps.
Volume is very large: the front alone renders tens of thousands of words at any moment. That makes a considered schedule part of the design rather than an afterthought, and it is the reason most projects here define a section list up front instead of crawling everything.
Get a Quote
Developers
Customers worldwide
Pages extracted
Hours saved for our clients
€199 / one-time
setup fee - included
€169 / mo
setup fee €499
€229 / mo
setup fee €499
€349 / mo
setup fee €499
€549 / mo
setup fee €799
The standard media monitoring pipeline assumes an item is an article with a body. On this source that assumption quietly breaks a large share of the dataset. A six minute television package that led the front is stored as forty words of caption, and any measure built on text length ranks it below a wire brief nobody read.
Typing items fixes it. Once article, video, gallery and live page are separate types, a client can weight them deliberately, count them separately, or exclude the ones that do not fit the question. What they cannot do is discover the problem six weeks later inside a sentiment model.
The second reason is editions. Domestic and international fronts differ in selection and in order, and for an international brand that difference is often the finding: a story leading in one market and absent in the other. Both are collected, each row carries its edition, and the same story appearing in both is grouped rather than counted twice.
The third is pace. Promotion on a broadcaster front turns over through the day far faster than on a newspaper, so the interval between observations changes what you see. We set it against how fast that particular front actually moves rather than applying one number across the site.
The fourth is the live page. On any major breaking story most of CNN's reporting appears inside one, and a collector that stores it as a single article with a changing headline captures almost none of it.
Learn how to use web scraping to solve data problems for your organization
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
If you specialize in machine learning, you need to feed large amounts of data to the algorithms. Web scraping is the easiest and the most efficient method of collecting the data from all over the Internet.
ScrapeIt is a managed extraction company, not a tool you have to learn. Our team builds the pipeline, watches it as fronts and templates change, and repairs it before your dashboard goes quiet.
You see a sample first, in your format, across the sections and editions you actually track, with item types already separated so you can see how much of the output is video before committing.
Because a video package carries almost no body text, and any pipeline that measures items by word length ranks the most watched output at the bottom. We record item type explicitly - article, video, gallery, live page, interactive - and capture duration instead of a word count for video, so you can weight or exclude them deliberately rather than discovering the distortion inside a model later.
Yes, and for an international brand the difference between them is usually the finding. The fronts differ in selection and in order. Every row carries its edition, and a story appearing in both is grouped under one identifier so you can count coverage once globally or once per market.
Yes, as a parent record plus a stream of timestamped entries with author where credited. On a breaking story most of the reporting lives inside a live page, and storing it as one article with a changing headline captures almost none of it.
Faster than on a newspaper. Promotion on a broadcaster front turns over through the day, so the interval between observations changes what you see. We set the cadence per front against how fast it actually moves, rather than applying one number across the whole site.
Republication is not what this dataset is for. The journalism and video are copyrighted and the terms restrict automated collection and reuse. We work only on public pages, never behind a login, we pace requests, we do not download video files, and we build metadata and publicly rendered text for analysis. Have your own counsel approve the use case before the project starts.
Step 1 - Make a Request
You share your needs, expectations, and desired timeframe. We’ll suggest the best solution based on your request and budget.
Step 2 - Configuring Custom Web Crawlers
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Step 3 - Collect and Deliver
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Step 4 - Maintain and Support
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Scrapeit Sp. z o.o.
10/208 Legionowa str., 15-099, Bialystok, Poland
NIP: 5423457175
REGON: 523384582
