Data.gov stores no data itself: it harvests metadata from federal, state and city publishers. Our Data.gov scraper collects those records and the tables they point to.
Plans from €169 a month · Free project assessment · Reply within 1 business day
CSV · Excel · JSON · XML · FTP · SFTP · Amazon S3 · Google Cloud
API, JSON feed or CSV, XLSX and JSON files
ScheduleOne-time, monthly, weekly, daily or 3 times a day
Data.gov pages, filters, fields and a schedule
Built, monitored and repaired by our team
API, JSON feed or CSV, XLSX and JSON files
You name the slice of the catalog you need - one agency such as the Department of Transportation, every City Government publisher, a keyword such as usdot number, or a box drawn on the map - and we build the crawler, run it and repair it when catalog.data.gov changes its pages. To scrape Data.gov properly the run opens every record behind a search card, because the card shows an excerpt and the formats while tags, resources and the Complete Metadata block sit on the record page. Pacing, proxy rotation and page rendering are part of the service.
Runs are one-off or scheduled daily, weekly or on demand. Each row keeps the catalog slug and the publisher's own identifier, so a record stays one row from run to run, and field documentation comes with the first delivery. A free project assessment shows what the scope will return first.
A catalog.data.gov record has two layers: the summary at the top of the page and the Complete Metadata block below it, which holds everything the publishing agency filled in. We read both and return one row per dataset, with its resources nested in the row or delivered as a linked table.
Some records name an individual employee as the contact. Those names and personal mailboxes are personal data, handled under GDPR and left out by default. The field documentation names the metadata key behind every column, so the dataset maps back to the catalog without guesswork.
A catalog entry is an index card. Its resource list points to the publisher's own pages - the State of Washington's for Electric Vehicle Population Data, the Department of Transportation's for the FMCSA Company Census File - and the rows live there. When the project needs them, we follow those resources, collect the tables, map each publisher's columns to one schema and stamp every row with the Data.gov record it came from. In the Company Census File, FMCSA gives each registered carrier a USDOT number, and that number becomes the key across runs.
Coverage of the catalog itself is built from its own filters: Geographic Area with Within or Intersect matching, Keywords, Organization, Organization Type, Publisher, and the geospatial-only or non-geospatial-only switch. Sorting by Last Published Date brings new datasets to the top, Popularity ranks by last month's page views.
Change monitoring comes from repeated runs. Each pass compares Dataset Last Updated and Catalog Last Checked with the previous one and reports new datasets, edited records and removed datasets per publisher. Because the catalog reflects an agency's additions, edits and deletions at the next harvest, a weekly change report on one organization reads like its release log.
Data.gov is the United States government's catalog of datasets. The General Services Administration launched it with the Office of Management and Budget on May 21, 2009, and GSA's Technology Transformation Services runs it today. Under the OPEN Government Data Act, federal agencies have to keep metadata inventories for a central catalog to harvest; catalog.data.gov is that catalog, and it lists more than half a million records.
Data.gov calls itself a metadata catalog and does not hold the data. Each agency keeps a harvest source, the harvester checks it on a set schedule - daily, weekly or monthly - and whatever the agency added, edited or deleted shows up after the next pass. States, counties and cities join when they publish their metadata the same way, which is why New York State lottery results and City of Chicago crime records sit beside NOAA and NASA entries. A record that looks wrong is corrected by the agency, never by Data.gov.
The weight is uneven. The U.S. Census Bureau publishes the largest share of records by far, much of it county-level geography, which is why tags such as county or equivalent entity and state fips code head the keyword filter. The Organizations page lists every publisher with its type and dataset count, a quick map of who actually feeds the catalog.
Delivery: CSV, JSON or XLSX files, our API or a JSON feed. MCP server on request.
Full plan detailsAnyone who sets out to scrape data from a government website first has to find which website holds it. Data.gov answers that for federal agencies and for the states, counties and cities that join, and its records make the answer something you can monitor.
Learn how to use web scraping to solve data problems for your organization
Eight US sources of federal records and court data, from SAM.gov contract notices to EPA ECHO violations and USPTO trademarks: what each site shows, the identifiers that join them, and how to build one dataset from several.
Data scraping is the wide term that includes documents, scans and old screens; web scraping is the part that works on websites. We compare sources, daily work, output and cost drivers, give a briefing checklist and walk through two of our projects.
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
ScrapeIt is a managed web scraping service: we build, run and repair the crawlers and deliver the dataset, one-off or on your schedule. Government catalogs are a source of their own kind - many publishers, one metadata standard applied unevenly, and the actual tables spread across agency sites. We take over that mapping work and deliver one consistent table with field documentation instead of a folder of agency files. Plans start from EUR 169 a month, or EUR 199 for a one-time extract.
Still unsure? Write to info@scrapeit.io
Yes. The ScrapeIt Data.gov API returns title, organization type, publisher, resource formats, Dataset Last Updated and access level as JSON from an endpoint we host, refreshed on the schedule you set. The same records also come as CSV or XLSX files, a JSON feed or a load straight into your database.
Yes. Catalog records, resource lists, tags and dates are open to any visitor of catalog.data.gov without an account. We collect only publicly available data - everything a visitor can see on Data.gov - and we collect it legally. Individual contact names are personal data and stay out of the dataset by default.
Yes. The scope uses the filters the catalog itself offers: Organization, Organization Type, Publisher, Keywords and Geographic Area. A brief can be, for example, every Federal Government record from the Department of Transportation plus all City Government publishers, refreshed weekly, with the jurisdiction badge kept as a column.
The harvester checks each agency's source on its own schedule, daily, weekly or monthly, and Accrual Periodicity, where an agency fills it, states the publisher's cadence, such as R/P1W for weekly. Weekly runs suit catalog monitoring; the tables behind busy records often need daily runs. Every pass records Catalog Last Checked, so you can see how current each entry was.
Yes. Each record lists its resources with format and address, usually on the publishing agency's or state's own pages. We collect those tables, map their columns to one schema and keep the Data.gov identifier on every row, so a merged table can always be traced back to its catalog entry.
You share your needs, expectations, and desired timeframe. We'll suggest the best solution based on your request and budget.
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Tell us which sites, which fields and how often. A couple of lines is enough to start.
Data.gov data from €169/month. Free project assessment, reply within 1 business day.