A Semantic Scholar record is two identifiers, a labelled citation edge and a summary the platform wrote itself. A scraper that reads titles only throws away the part you came for.
ScrapeIt writes the collector, runs it on our machines and repairs it when the platform moves. Anti-bot handling, proxy rotation and CAPTCHA solving are part of the managed service, and this project uses them: paper pages are assembled in the browser rather than served as finished markup, and the anonymous request pool throttles under load.
We plan collection the way the platform invites it, taking sitemap indexes as seeds, graph endpoints for records and the bulk releases when the answer is the whole corpus rather than a slice. Papers, citation edges and full-text snippets run on separate schedules, because a citation count moves weekly and a title never moves. A sample extract comes first, so field names, coverage and null rates can be checked before the full run. Delivery is CSV, JSON, JSONL, XLSX or an endpoint you call, pushed to S3, SFTP or your warehouse.
These are the fields we extract Semantic Scholar data into, under the platform's own names.
There is a public Semantic Scholar API, and it is generous enough to change what a scraper is for. Three services sit behind api.semanticscholar.org: Academic Graph, Recommendations and Datasets. Academic Graph covers paper and author lookup, relevance search that returns a hundred hits per call and stops at a thousand ranked results, a bulk search that walks a thousand records at a time on a continuation token, title matching, batch lookups of up to 500 paper ids, paged citations and references, and a snippet search returning roughly 500-word excerpts tagged with the section they came from.
Quotas are the catch. Anonymous callers share one pool and get throttled under load as a matter of course; an issued key starts at one request per second on every endpoint, and a single response is capped at 10 MB. Bulk search stops after ten million papers per query, and past that the Datasets API is the documented route, which will not show a download link at all without a key.
Those releases are heavy: a fresh one lands weekly, with papers at about 200 million records in thirty files of 1.5 GB, abstracts at 100 million, citations at 2.4 billion in thirty files of 8.5 GB, authors at 75 million, tldrs at 58 million, an ID map of 450 million rows, venues, parsed full text from open-access PDFs, and SPECTER vectors in thirty files of 28 GB. Diffs list updated and deleted files so you upsert by corpusid instead of pulling everything again. Lookups accept DOI, ARXIV, MAG, ACL, PMID, PMCID, CorpusId and URL prefixes, which is how a Semantic Scholar extract joins onto PubMed, arXiv or an internal DOI list.
Semantic Scholar is the academic search service run by Ai2, the Allen Institute for AI, and it is a graph before it is a website. Five entity types carry everything: papers, authors, publication venues, topics, and the citation edges between papers. The search box counts more than 237 million papers from all fields of science, while the developer overview quotes roughly 214 million papers, 2.49 billion citations and 79 million authors for the same graph in September 2026. The two counters answer different questions, and whoever sizes a Semantic Scholar scraper has to decide which one the job runs against.
Addresses follow the entity. A paper sits at /paper/ plus a title-and-author slug plus a 40 character hexadecimal ID, and the bare ID resolves on its own, so the slug is decoration. Authors answer at /author/ plus a display name plus a numeric ID. Topics answer at /topic/ plus a slug plus a numeric ID, and a topic page carries a definition with the ontology that supplied it. Venues break the pattern: they live at /venue with the venue name as a query parameter, and a venue page holds the full publication list with filters for fields of study, date range and PDF availability.
robots.txt names three sitemap indexes, one for papers, one for authors, one for topics. The paper index currently lists 2,877 files, the author index 3,261 and the topic index 19, with about thirty thousand addresses per file and a lastmod date on every entry. That is a seed list rather than an inventory, because the sitemaps reach a fraction of the corpus. The same file closes the search path, the reader, account pages and every query string on paper and author URLs, so faceted browsing is not the way in. The graph is.
Get a Quote
Developers
Customers worldwide
Pages extracted
Hours saved for our clients
€199 / one-time
setup fee - included
€169 / mo
setup fee €499
€229 / mo
setup fee €499
€349 / mo
setup fee €499
€549 / mo
setup fee €799
The reason to scrape Semantic Scholar is the citation layer, because the platform does not merely count references, it labels them. A citation edge can carry contexts, the sentences around the reference in the citing paper's body, intents drawn from Background, Method and Result, a contextsWithIntent pairing of the two, and an isInfluential flag. The bulk citations collection holds about 2.4 billion of those edges, and the platform defines citation velocity and acceleration on top of the raw count.
That converts questions which used to need reading into queries. Which groups build on a method instead of nodding at it in an introduction. Which journal gets cited for results and which for background. When a technique stopped being novel and turned into a citation of convenience. A bare count answers none of it; a labelled edge does, and the influential subset is a different ranking from the raw total.
Two limits belong in the same breath, because they shape coverage. Influence and intent labels need the full text of the citing paper, so work locked behind a publisher is labelled thinly or not at all. Field-of-study classification runs on English titles and abstracts. A Semantic Scholar data feed should state where the graph is dense and where it thins out instead of pretending it is uniform.
Collection stays tractable because the same filters exist end to end: year and publicationDateOrYear ranges, venue, fieldsOfStudy, minCitationCount, publicationTypes and an open-access-only switch, with sorting by citation count, publication date or paper ID.
Learn how to use web scraping to solve data problems for your organization
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
If you specialize in machine learning, you need to feed large amounts of data to the algorithms. Web scraping is the easiest and the most efficient method of collecting the data from all over the Internet.
ScrapeIt is a managed data collection agency. We write the crawlers, run them on our machines, fix them when a site changes and hand over clean files. You do not maintain code and you do not manage proxies.
We also say no. If a bulk release answers your question more cheaply than a crawl, you hear that on the first call. We do not sign in to reach subscription full text, we do not touch account pages or claimed profiles, and we do not assemble named researcher dossiers or contact lists out of author records.
Yes, and it is a real one. The Semantic Scholar API splits into Academic Graph for papers, authors, venues, citations and references, Recommendations for similar papers, and Datasets for full-corpus downloads. It returns metadata, abstracts where licensing allows, tldr summaries, open-access PDF links, embeddings and labelled citation edges. What it does not return is subscription full text, and abstracts are missing on some records for legal reasons. Anonymous use shares a throttled pool, an issued key starts at one request per second, and dataset download links require that key.
paperId and corpusId, title, abstract, venue and publicationVenue, year and publicationDate, journal name with volume and pages, publicationTypes, referenceCount, citationCount and influentialCitationCount, isOpenAccess with an openAccessPdf object holding url, status, licence and disclaimer, fieldsOfStudy and s2FieldsOfStudy with the source of each label, tldr, textAvailability, BibTeX, and the author list with affiliations, ROR-normalised affiliations, paperCount, citationCount, hIndex, ORCID and DBLP. Citation edges add contexts, intents and an influence flag.
By identifier, not by title. A lookup accepts DOI, ARXIV, MAG, ACL, PMID, PMCID and CorpusId prefixes, plus a URL form recognised for arxiv.org, aclweb.org, acm.org, biorxiv.org and semanticscholar.org, so an existing DOI or PMID list resolves straight into the graph. Inside the bulk files the join key is corpusid rather than the sha, and a separate mapping table links the two and marks the primary sha. Coverage is uneven: some records carry DOI, PubMed and DBLP at once, others only a legacy MAG number.
Cadence follows the field, not the calendar. New papers and new citation edges appear constantly, and the full-corpus release cycle is weekly, so citation counts, influential counts and open-access status deserve a weekly pass. Titles, venues and publication dates barely move and can run monthly. Full text and figures change only when a source PDF is reprocessed. Pricing for a Semantic Scholar project follows that split, since re-collecting a stable field costs the same as collecting a volatile one. Output ships as CSV, JSON, JSONL or XLSX, to S3, SFTP, a warehouse or an endpoint you call.
The bulk collections are published under an open data licence with attribution, and the platform asks that work built on them cite its data platform paper, so redistribution rules are clearer here than on most sites. The limits are equally clear: the API licence forbids working around rate limits, robots.txt closes search, reader and account paths, and Semantic Scholar cannot grant reuse rights to a publisher's article, which stays with the author or publisher. We work with what is published to anonymous visitors, we do not sign in, and named individuals are collected only as the bibliographic author string, with no dossiers and no contact details.
Step 1 - Make a Request
You share your needs, expectations, and desired timeframe. We’ll suggest the best solution based on your request and budget.
Step 2 - Configuring Custom Web Crawlers
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Step 3 - Collect and Deliver
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Step 4 - Maintain and Support
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Scrapeit Sp. z o.o.
10/208 Legionowa str., 15-099, Bialystok, Poland
NIP: 5423457175
REGON: 523384582
