Data.gov.au gathers datasets from Commonwealth agencies, the states and local councils, many harvested from portals such as DataNSW. Our Data.gov.au scraper keeps the source on every row.
Plans from €169 a month · Free project assessment · Reply within 1 business day
CSV · Excel · JSON · XML · FTP · SFTP · Amazon S3 · Google Cloud
API, JSON feed or CSV, XLSX and JSON files
ScheduleOne-time, monthly, weekly, daily or 3 times a day
Data.gov.au pages, filters, fields and a schedule
Built, monitored and repaired by our team
API, JSON feed or CSV, XLSX and JSON files
Tell us which part of the catalogue matters - an organisation such as Geoscience Australia or the City of Gold Coast, a group like Civic Infrastructure, a list of LGAs, a format such as GeoJSON - and we build the crawler, schedule it and maintain it as the site changes. The question of how to scrape Data.gov.au well comes down to provenance: a harvested record has to stay linked to the state portal it came from, so our rows carry the Data Portal and Remote Last Updated values next to the local dates. Proxy rotation, pacing and page rendering are included.
Deliveries run once or on your schedule - daily, weekly or on demand - as CSV, Excel, JSON or JSONLines, or straight into Amazon S3, Google Cloud Storage or your database. Field documentation and a free project assessment come with it.
A Data.gov.au dataset page is built from three blocks: a header with Created and Updated dates, a list of resources, and an Additional Info table whose rows depend on where the record came from. We return one row per dataset and one per resource, joined on the landing page identifier.
Descriptions often carry field lists of their own. The Energy Rating records describe brand name, model number, star rating and country of origin for registered appliances, and when the rows matter we collect those resource tables as well, typed and documented.
The catalogue's own filters define coverage. Location takes an LGA from a list covering every state and territory, from Albury and Brisbane to Hobart and Darwin, typed bounding coordinates, or a box drawn on a map; beside it sit Organisation, Groups, Tags, Formats and a Date range by month. Results order by Relevance, Name Ascending, Name Descending or Last Modified, and each card shows the organisation, a description excerpt, the Dataset updated date and the formats. We slice by organisation and format so that large publishers such as Geoscience Australia Data and the Australian Ocean Data Network come out complete.
History is where Data.gov.au shows more than most catalogues. A dataset's Activity Stream lists New Package, Changed Package and Deleted Package events with the publisher account behind each change; City of Gold Coast Road Closures, which the council says is updated daily, logs a run of updates every day. Repeated runs add your own record of which datasets appeared, which resources were replaced and which were marked [INACTIVE].
Where the rows behind a record matter more than the record, such as council planning applications, we collect the resource tables and map them to one schema with the catalogue identifier on each row.
Data.gov.au is the central catalogue of Australian government data, managed by the Commonwealth Government and administered by the Australian Bureau of Statistics. It is meant as one place to reach the public data of federal, state and local government agencies, and it also takes publicly-funded research data and datasets from private institutions where they serve the public interest: CSIRO's Marlin catalogue, the Terrestrial Ecosystem Research Network and a national transport noise map from a company called Ambient share the same index.
The agencies stay the custodians. Data.gov.au does not create or maintain the datasets; each organisation does, and more than a thousand organisations are listed, from Albury City Council to the Department of Climate Change, Energy, the Environment and Water. Part of the catalogue arrives by harvest: organisations described as harvesters bring in records from Victorian Government and ACT Government catalogues, and NSW records carry a From DataNSW label. One dataset can therefore exist on a state portal and on Data.gov.au, with two update dates that do not always agree.
Spelling follows Australian usage - Organisations, catalogue - and dates are written day first, which matters the moment the fields are parsed.
Delivery: CSV, JSON or XLSX files, our API or a JSON feed. MCP server on request.
Full plan detailsThe value of Data.gov.au is the cross-jurisdiction view: a Commonwealth department, a state agency and a regional council publishing side by side in one catalogue.
Learn how to use web scraping to solve data problems for your organization
Data scraping is the wide term that includes documents, scans and old screens; web scraping is the part that works on websites. We compare sources, daily work, output and cost drivers, give a briefing checklist and walk through two of our projects.
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
ScrapeIt runs data collection as a managed service: our team builds the crawlers, keeps them running and hands over the dataset with field documentation. For a federated catalogue like Data.gov.au most of the work is reconciliation - harvested copies against state originals, day-first dates, map layers next to spreadsheets - so that the table you receive treats every record the same way. Order a sample on the organisations you track to see the columns first. Plans start from EUR 169 a month, or EUR 199 for a one-time extract.
Still unsure? Write to info@scrapeit.io
Yes. The ScrapeIt Data.gov.au API returns the title, organisation, groups, resource formats, Reference Period and Updated date of each record as JSON from an endpoint we host, on the refresh schedule you choose. You can take the same data as CSV or XLSX files or a JSON feed, or have it delivered to SFTP or Google Drive.
Yes. Dataset pages, organisation pages and Activity Streams on Data.gov.au open without an account. We collect only publicly available data - everything a visitor can see on Data.gov.au - and we collect it legally. Personal names in contact fields are handled under GDPR and left out by default.
Records come from Commonwealth agencies, state and territory governments, local councils, research bodies and some private institutions. A single state is a scope we build from the filters: the LGAs of that state in the Location box, and its publishers, such as NSW Government, in the Organisation filter. Harvested records keep their Data Portal value, so a DataNSW copy is told apart from a record published directly.
We keep both sides. The row carries the Data.gov.au Updated date, the Remote Last Updated date from the source, the Data Portal name and the address of the original record. When the two dates disagree the row is flagged, so you know which copy is current before you rely on it.
Yes. Each run reads the Activity Stream, where Data.gov.au logs New Package, Changed Package and Deleted Package events with the publisher account behind each one, and compares the resource list with the previous pass. A change report travels with the dataset: added records, replaced files, and datasets newly marked [INACTIVE].
You share your needs, expectations, and desired timeframe. We'll suggest the best solution based on your request and budget.
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Tell us which sites, which fields and how often. A couple of lines is enough to start.
Data.gov.au data from €169/month. Free project assessment, reply within 1 business day.