A Semantic Scholar record is two identifiers, a labelled citation edge and a summary the platform wrote itself. A scraper that reads titles only throws away the part you came for.
Plans from €169/month · Free project assessment · Reply within 1 business day
ScrapeIt writes the collector, runs it on our machines and repairs it when the platform moves. Anti-bot handling, proxy rotation and CAPTCHA solving are part of the managed service, and this project uses them: paper pages are assembled in the browser rather than served as finished markup.
We plan collection the way the platform is built, taking sitemap indexes as seeds and following the graph from each paper to its authors, venue and citation edges. Papers, citation edges and full-text snippets run on separate schedules, because a citation count moves weekly and a title never moves. A sample extract comes first, so field names, coverage and null rates can be checked before the full run. Delivery is CSV, JSON, JSONL, XLSX or an endpoint you call, pushed to S3, SFTP or your warehouse.
These are the fields we extract Semantic Scholar data into, under the platform's own names.
Citation edges come as their own records rather than as a count. Each edge links a citing and a cited paper and carries the contexts - the sentences around the reference in the citing paper's body - the intents drawn from Background, Method and Result, and the isInfluential flag, so a citation graph arrives ready to query instead of needing to be read.
Topic and venue pages add two layers the paper record lacks. A topic page carries a definition with the ontology that supplied it, and a venue page holds the full publication list with filters for fields of study, date range and PDF availability, which turns a journal or conference into a complete list of its papers.
Author records carry authorId, name, affiliations with ROR identifiers, paperCount, citationCount, hIndex and ORCID or DBLP ids, and we build no named researcher profiles or contact lists from them.
Matching runs on identifiers, not titles. Every paper record carries its externalIds - DOI, ArXiv, PubMed, PubMedCentral, Medline, MAG, ACL, DBLP and CorpusId - which is how a Semantic Scholar extract joins onto PubMed, arXiv or an internal DOI list.
Semantic Scholar is the academic search service run by Ai2, the Allen Institute for AI, and it is a graph before it is a website. Five entity types carry everything: papers, authors, publication venues, topics, and the citation edges between papers. The search box counts more than 237 million papers from all fields of science, while the platform's own overview quotes roughly 214 million papers, 2.49 billion citations and 79 million authors for the same graph in September 2026. The two counters answer different questions, and whoever sizes a Semantic Scholar scraper has to decide which one the job runs against.
Addresses follow the entity. A paper sits at /paper/ plus a title-and-author slug plus a 40 character hexadecimal ID, and the bare ID resolves on its own, so the slug is decoration. Authors answer at /author/ plus a display name plus a numeric ID. Topics answer at /topic/ plus a slug plus a numeric ID, and a topic page carries a definition with the ontology that supplied it. Venues break the pattern: they live at /venue with the venue name as a query parameter, and a venue page holds the full publication list with filters for fields of study, date range and PDF availability.
Three sitemap indexes are published, one for papers, one for authors, one for topics. The paper index currently lists 2,877 files, the author index 3,261 and the topic index 19, with about thirty thousand addresses per file and a lastmod date on every entry. That is a seed list rather than an inventory, because the sitemaps reach a fraction of the corpus. The rest is reached through the graph: each paper leads to its authors, its venue, its references and the papers citing it.
Get a Quote
Developers
Customers worldwide
Pages extracted
Hours saved for our clients
€199 / one-time
setup fee - included
€169 / mo
setup fee €499
€229 / mo
setup fee €499
€349 / mo
setup fee €499
€549 / mo
setup fee €799
Delivery: CSV, JSON or XLSX files, our API or a JSON feed. MCP server on request.
Full plan details →The reason to scrape Semantic Scholar is the citation layer, because the platform does not merely count references, it labels them. A citation edge can carry contexts, the sentences around the reference in the citing paper's body, intents drawn from Background, Method and Result, a contextsWithIntent pairing of the two, and an isInfluential flag. The graph holds about 2.4 billion of those edges, and the platform defines citation velocity and acceleration on top of the raw count.
That converts questions which used to need reading into queries. Which groups build on a method instead of nodding at it in an introduction. Which journal gets cited for results and which for background. When a technique stopped being novel and turned into a citation of convenience. A bare count answers none of it; a labelled edge does, and the influential subset is a different ranking from the raw total.
Two limits belong in the same breath, because they shape coverage. Influence and intent labels need the full text of the citing paper, so work locked behind a publisher is labelled thinly or not at all. Field-of-study classification runs on English titles and abstracts. A Semantic Scholar data feed should state where the graph is dense and where it thins out instead of pretending it is uniform.
Collection stays tractable because the same filters exist end to end: year and publicationDateOrYear ranges, venue, fieldsOfStudy, minCitationCount, publicationTypes and an open-access-only switch, with sorting by citation count, publication date or paper ID.
Learn how to use web scraping to solve data problems for your organization
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
If you specialize in machine learning, you need to feed large amounts of data to the algorithms. Web scraping is the easiest and the most efficient method of collecting the data from all over the Internet.
ScrapeIt is a managed data collection agency. We write the crawlers, run them on our machines, fix them when a site changes and hand over clean files. You do not maintain code and you do not manage proxies.
We also say no. We do not sign in to reach subscription full text, we do not touch account pages or claimed profiles, and we do not assemble named researcher dossiers or contact lists out of author records.
Yes. The ScrapeIt Semantic Scholar API returns paperId and externalIds, title and abstract, citationCount and influentialCitationCount, tldr summaries and labelled citation edges as JSON from an endpoint we host, refreshed on the schedule you set. The same data also comes as CSV or XLSX files or straight into your database.
paperId and corpusId, title, abstract, venue and publicationVenue, year and publicationDate, journal name with volume and pages, publicationTypes, referenceCount, citationCount and influentialCitationCount, isOpenAccess with an openAccessPdf object holding url, status, licence and disclaimer, fieldsOfStudy and s2FieldsOfStudy with the source of each label, tldr, textAvailability, BibTeX, and the author list with affiliations, ROR-normalised affiliations, paperCount, citationCount, hIndex, ORCID and DBLP. Citation edges add contexts, intents and an influence flag.
By identifier, not by title. Every paper record we deliver carries externalIds - DOI, ArXiv, PubMed, PubMedCentral, Medline, MAG, ACL, DBLP and CorpusId - so an existing DOI or PMID list joins straight onto the extract. A paper can answer to several shas, and the primary one stays flagged. Coverage is uneven: some records carry DOI, PubMed and DBLP at once, others only a legacy MAG number.
Cadence follows the field, not the calendar. New papers and new citation edges appear constantly, so citation counts, influential counts and open-access status deserve a weekly pass. Titles, venues and publication dates barely move and can run monthly. Full text and figures change only when a source PDF is reprocessed. Pricing for a Semantic Scholar project follows that split, since re-collecting a stable field costs the same as collecting a volatile one. Output ships as CSV, JSON, JSONL or XLSX, to S3, SFTP, a warehouse or an endpoint you call.
Yes. Titles, abstracts, venues, dates, citation counts, citation contexts and the author strings printed on each paper are shown to any visitor without logging in. We collect only publicly available data - everything a visitor can see on Semantic Scholar - and we collect it legally. We do not sign in, and named individuals are collected only as the bibliographic author string, with no dossiers and no contact details.
Step 1 - Make a Request
You share your needs, expectations, and desired timeframe. We’ll suggest the best solution based on your request and budget.
Step 2 - Configuring Custom Web Crawlers
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Step 3 - Collect and Deliver
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Step 4 - Maintain and Support
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Semantic Scholar data from €169/month. Free project assessment, reply within 1 business day.
Scrapeit Sp. z o.o.
10/208 Legionowa str., 15-099, Bialystok, Poland
NIP: 5423457175
REGON: 523384582
