Deutsche Welle publishes 31 language editions written by separate desks, and it rewrites headlines after publication. We collect every edition and keep every version of the story.
Plans from €169 a month · Free project assessment · Reply within 1 business day
CSV · Excel · JSON · XML · FTP · SFTP · Amazon S3 · Google Cloud
API, JSON feed or CSV, XLSX and JSON files
ScheduleOne-time, monthly, weekly, daily or 3 times a day
Deutsche Welle pages, filters, fields and a schedule
Built, monitored and repaired by our team
API, JSON feed or CSV, XLSX and JSON files
We scrape Deutsche Welle as a managed service. Discovery runs on the cheap surfaces - the news sitemap and each edition's headlines page - and the fields come from the story pages, which carry the full record. Backfill walks the topic pages. Rows are deduplicated on the numeric ID, links lose their maca tracking parameter, and every headline version is stored with its timestamp. You get CSV, JSON, XLSX or an endpoint we host: a Deutsche Welle API of your own, with field names agreed up front. Anti-bot handling, proxy rotation and CAPTCHA solving are part of the service, and nothing needs a login. The price of a Deutsche Welle feed depends on editions, cadence and backfill depth.
One DW story is several records in one, and to extract Deutsche Welle data cleanly the export keeps them apart. These are the fields we collect for each article:
Deutsche Welle live blogs add a list of posts, each with its own ID, title, timestamp and text. Stories from the 2000s carry a free-text credit line instead of a structured byline, no category or country code and a single date; those fields stay empty rather than guessed.
The same ID system covers DW's media, and each type brings fields of its own.
A Deutsche Welle scraper has to start from what DW is: Germany's international broadcaster, a public broadcaster funded by federal taxes under the Deutsche Welle Act, on air since 1953 and based in Bonn and Berlin. It writes for audiences outside Germany, so dw.com is not a German site with an English corner. It is 31 language editions, written by separate language desks, each under its own path code. /en/, /de/, /es/, /ru/, /uk/ and /ar/ explain themselves, while /sw/ is Kiswahili, /ha/ Hausa, /am/ Amharic, /fa-ir/ Persian, /fa-af/ Dari, /ps/ Pashto, and /pt-br/ and /pt-002/ split Portuguese between Brazil and Africa. There is no paywall and no subscriber tier, so every story renders in full.
Every address ends in a numeric ID with a type prefix: a- for articles, live- for live blogs, video-, audio-, g- for galleries, t- for topic pages, s- for sections and program- for shows. The slug in front of the ID is decoration, since an address with the ID alone lands on the story, but the language path is part of the key: an English ID under /de/ leads nowhere. IDs are unique across the editions with one exception. /zh-hant/ is the Traditional Chinese twin of /zh/ and reuses its IDs, so Chinese stories are deduplicated, not counted twice.
Each edition also has a headlines page with its last seven days of output, an A to Z topic index and a sitemap set of its own. Together they turn the site into a Deutsche Welle database keyed on one number.
Delivery: CSV, JSON or XLSX files, our API or a JSON feed. MCP server on request.
Full plan detailsDeutsche Welle data scraping pays off because DW is Germany talking to the world, and each desk decides what its own audience hears: one broadcaster, one legal mandate, some thirty agendas. The German edition is far from the largest. Over the three months to late September 2026 the article sitemaps listed about 2,950 articles and live blogs in Spanish, 2,770 in Russian, 2,230 in Ukrainian, 1,880 in English and 1,040 in German.
Nothing is cut short by a paywall, so word counts and models compare fairly across editions.
Learn how to use web scraping to solve data problems for your organization
Data scraping is the wide term that includes documents, scans and old screens; web scraping is the part that works on websites. We compare sources, daily work, output and cost drivers, give a briefing checklist and walk through two of our projects.
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
ScrapeIt is a managed web scraping agency: we build, run and repair the collectors, and you receive the data. A Deutsche Welle project usually opens with a sample - one week of three editions you choose, with headline versions, category IDs and bylines - so you can test the schema against your use before a schedule is agreed. Files land in S3, GCS, SFTP, a webhook or your warehouse, and we keep the pipeline working as DW changes its pages.
Still unsure? Write to info@scrapeit.io
Yes. The ScrapeIt Deutsche Welle API returns the numeric ID, edition, all four headline fields, first publication date, category IDs and byline as JSON from an endpoint we host, refreshed on the schedule you set. The same data also comes as CSV or XLSX files or straight into your database.
Well beyond the sitemaps, which cover about three months. Topic pages go back fifty stories per page through each topic's full history, and the English Bundesliga topic reaches November 2001. A backfill is built from topic pages, the A to Z index and the links between stories, and addresses built on the numeric ID stay stable. Older records are thinner: stories from the 2000s carry a free-text credit instead of a structured byline, no category or country code and a single date, and we deliver those fields empty rather than inferred.
From each edition's headlines page and the news sitemap, which list new IDs as stories go up. The fields come from the story pages, which carry the full record: all four headline fields, byline, tags, body text and the three dates. Every headline version is stored with its timestamp, so a story reframed hours later keeps both versions, and first publication stays separate from the last republish.
Both, as metadata and links. For video we collect title, teaser, duration, programme, dates, thumbnail and stream address, and note whether subtitles exist; for audio the episode, host, programme, duration and MP3 link, plus keywords per episode. The English, Spanish and Arabic live channel pages add the TV schedule, with start, end, programme, episode title and teaser for each slot.
Yes. We collect only publicly available data - everything a visitor can see on Deutsche Welle - and we collect it legally. That covers every headline version, teasers, dates, categories, country codes, bylines, full article text and media metadata, with no paywall and no login involved. The data suits media monitoring, agenda comparison across language editions, tracking headline changes and research corpora.
You share your needs, expectations, and desired timeframe. We'll suggest the best solution based on your request and budget.
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Tell us which sites, which fields and how often. A couple of lines is enough to start.
Deutsche Welle data from €169/month. Free project assessment, reply within 1 business day.