data.gov.uk Scraper for Dataset Records and Linked Data

data.gov.uk is a directory: each record points to data a department or council publishes on its own site. Our data.gov.uk scraper collects both the record and the data.

Plans from €169 a month · Free project assessment · Reply within 1 business day

CSV · Excel · JSON · XML · FTP · SFTP · Amazon S3 · Google Cloud

data.gov.uk
data.gov.uk Scraper for Dataset Records and Linked Data
data.gov.ukwhat you get
Typical fields for public registers
record_identity_namestatusregistered_ataddressdocument_url
Delivery

API, JSON feed or CSV, XLSX and JSON files

Schedule

One-time, monthly, weekly, daily or 3 times a day

Scrape data.gov.uk With a Managed Service

  1. You send

    data.gov.uk pages, filters, fields and a schedule

  2. We run the scrapers

    Built, monitored and repaired by our team

  3. You receive the data

    API, JSON feed or CSV, XLSX and JSON files

You name the scope - a group of councils or one department, topics, formats, keywords, or the records behind one of the collections - and we build and run the crawlers and repair them when the directory changes its pages. Anti-bot handling, from pacing to proxy rotation, comes with the service.

Because data.gov.uk points to data rather than holding it, a project usually runs in two stages: first the dataset records, then, for the records you choose, the tables behind their data links on GOV.UK or the publisher's own site. Both carry the record id from its URL, so a row in a council spreadsheet always leads back to the catalogue entry that described it. Plans start from EUR 169 a month, or EUR 199 for a one-time extract.

data.gov.uk Datasets: Fields on Each Record

Each dataset record lives at /dataset/ followed by a unique id and a slug, and every record is laid out in the same blocks, whether a ministry or a borough council published it. We collect:

  • Record identitythe id and slug from the URL, the title, the alternative title where present, and the record URL.
  • Publisherthe publishing body as written, for example HM Land Registry, Greater London Authority, Environment Agency or a borough council.
  • ClassificationTopic, from Business and economy and Transport to Crime and justice and Government spending, kept as Not set where the publisher left it empty.
  • Datesthe Updated date of the record and of each data link, Date added, and the dataset reference date where shown.
  • Summarythe publisher's own description of the dataset.
  • Data linksone row per link with its label, URL, Format (for example CSV, XLS, JSON, PDF, ZIP or WEBPAGE), Preview and Updated, including the links hidden behind Show more.
  • Additional informationHarvest GUID, Frequency of update, Responsible party, extent in latitude and longitude, spatial reference system and metadata language, where the record carries them.

Search result cards add the publisher, the Updated date and a snippet, and each row keeps the filter values that brought it into the extract.

data.gov.uk Datasets: Fields on Each Record
data.gov.uk Data Beyond the Record

data.gov.uk Data Beyond the Record

The record describes the data; most buyers want the data itself, kept current:

  • Linked tablesfor the records in scope we follow each data link to the publisher's page and parse the table it leads to into one schema, so a series published as a new spreadsheet every quarter becomes one long table with a period column.
  • Collectionsthe curated pages under Business and economy, Environment, Government and Parliament, Land and property, People and Transport, plus the Early years spotlight. Land and property alone has pages for UK house prices, property price paid, planning data, addresses, dwelling stock, rents and the English Housing Survey.
  • Publisher coverageevery record of one publisher, or of all London boroughs, to see what each body publishes and what it leaves out.
  • Change trackingrepeated runs compare Updated dates, data links and formats, flagging new records, new links, links that stopped resolving and records that disappeared.

Merged with records from other national catalogues, the output becomes one inventory of public sector data with the same columns for every country.

About the data.gov.uk Catalogue

Source
data.gov.uk
Category
public registers
Delivery
API, JSON feed, CSV, XLSX
Plans
From €169 a month
Get a quote

data.gov.uk is the UK government's directory of public sector data, run by the Government Digital Service (GDS) in the Department for Science, Innovation and Technology, and its pages carry the National Data Library name in the header. It does not hold the data itself: central government departments, local authorities and other public bodies add links to data they publish on GOV.UK or on their own websites, and the directory describes each one in a standard record.

Visitors reach records in two ways. The Directory is a search with filters for Publisher, Topic and Format, an option to include datasets with no links, and sorting by Most relevant or Most recent. Collections are curated entry points by theme; Land and property, for instance, opens on the UK house price index, built from sales recorded by HM Land Registry, Registers of Scotland and Land and Property Services Northern Ireland and calculated by the Office for National Statistics.

Quality varies by publisher. Some records come with a topic, a frequency of update and a spatial extent; others show Topic as Not set and a single link to a web page. A dataset built from the directory has to carry those gaps as they are rather than fill them.

25Developers
500+Customers
1.5BPages extracted
15,000+Hours saved for our clients

data.gov.uk scraping plans and pricing

Airplane

€199one-time

Setup fee included

100,000
One-time
Up to 5 days
7 days

Helicopter

€169/ mo

Setup fee €499

250,000
Monthly
Up to 5 days
14 days

Glasses

€229/ mo

Setup fee €499

1,000,000
Weekly
Up to 5 days
30 days

Microscope popular

€349/ mo

Setup fee €499

2,000,000
Daily
Same day
60 days

DNA

€549/ mo

Setup fee €799

3,000,000
3 times daily
Same day
90 days

Delivery: CSV, JSON or XLSX files, our API or a JSON feed. MCP server on request.

Full plan details

Why Collect data.gov.uk Records and Data

The directory answers what the public sector publishes, who publishes it and whether it is kept current, and the linked tables hold the figures themselves:

  • Property and planning analyticsPrice Paid Data, which records property sales in England and Wales as submitted to HM Land Registry, together with the house price index and planning data, joined on area.
  • Local government benchmarkingthe same kind of dataset from many councils side by side, with publisher, format and Updated date showing which local authorities lag behind.
  • Spending and energy seriesGovernment spending records and the quarterly energy prices series from the Department for Energy Security and Net Zero, turned into continuous time series.
  • Source inventoriesevery record in a topic with its formats and frequency, for data vendors deciding which sources to integrate first.
  • Monitoringan alert when a publisher adds a dataset, adds a new period or lets a link break.

Each delivered row carries the record URL and the publisher name, so it can be cited.

About ScrapeIt

ScrapeIt is a managed web scraping service that collects and delivers data from public registers, catalogues and agency websites as well as commercial sites. A directory like this one is a two-level job - the records first, then many publisher sites behind them - and that second level is where most of the maintenance lives. We agree the scope in a free project assessment, send a sample, and deliver through our API, a JSON feed, SFTP or Google Cloud Storage.

FAQ

Still unsure? Write to info@scrapeit.io

Do you offer a data.gov.uk API for dataset records?

Yes. The ScrapeIt data.gov.uk API returns title, publisher, topic, Updated date, summary and every data link with its format as JSON from an endpoint we host, refreshed on the schedule you set. The same records also come as CSV or Excel files, a JSON feed or a load into your database.

Is it legal to scrape data.gov.uk?

Yes. The directory, its filters and every dataset record are open to anyone without signing in. We collect only publicly available data - everything a visitor can see on data.gov.uk - and we collect it legally. Personal names inside linked files are left out by default and handled under GDPR.

How often are data.gov.uk datasets updated, and how often do you check them?

It varies by record: HM Land Registry's Price Paid Data shows a monthly Frequency of update, while the quarterly energy prices record lists a separate link for each quarter. The Updated date on each search card shows which records moved, so a weekly pass over your publishers catches new links without re-reading everything.

Can you collect the data a record links to, not only the record?

Yes. For the records you pick we follow each data link, parse the CSV, XLS or JSON table it leads to and deliver it next to the record, keyed on the record id. Links whose Format reads WEBPAGE or PDF are handled case by case, since the figures may sit in a page table or a report.

Can you limit a run to councils or to one topic?

Yes. Publisher and Topic are filters of the directory itself, and Format narrows it further. We can scope a run to every London borough or to the Land and property collection, and we keep the publisher name exactly as the record writes it, so councils from several runs merge on one key.

How it works

  1. 1

    Make a request

    You share your needs, expectations, and desired timeframe. We'll suggest the best solution based on your request and budget.

  2. 2

    We configure the crawlers

    Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.

  3. 3

    Collect and deliver

    Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.

  4. 4

    Maintain and support

    Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.

Request a quote

Tell us which sites, which fields and how often. A couple of lines is enough to start.

  • We reply within 1 business day
  • A sample dataset before the full run
  • We sign an NDA on request
  • Delivery by API, JSON feed or files
Which sites, which fields, how often. A couple of lines is enough.

This form is protected by Google reCAPTCHA, which loads when you start filling it in; the Google Privacy Policy and Terms of Service apply. If you prefer not to use it, email us at info@scrapeit.io. Privacy & Cookie Policy

We reply within 1 business day. No obligation.

data.gov.uk data from €169/month. Free project assessment, reply within 1 business day.