Mayo Clinic licenses its health library by API to partners. The same library is public on mayoclinic.org, and we turn it into records keyed by concept ID, language and publish date.
A Mayo Clinic scraping project starts with a choice of libraries - conditions, symptoms, tests and procedures, supplements, departments, trials, price files - and of languages. Every record keeps its concept ID, page ID, tab, language and publish date, so a re-crawl shows exactly which pages Mayo republished. Micromedex drug text and the triage engine stay on the licensed side: we can list which monographs exist and join open FDA labels instead. Both mayoclinic.org and the mayo.edu trial registry sit behind bot protection that shuts out non-browser clients, so runs go through full browser sessions backed by anti-bot handling, proxy rotation and CAPTCHA solving; we never sign in or touch the patient portal. Output is JSON Lines, CSV, Parquet, a database load or an endpoint we host.
Mayo Clinic data is deepest on the condition record. The Symptoms and causes tab runs Overview, Symptoms, When to see a doctor, Causes, Risk factors, Complications and Prevention. Diagnosis and treatment adds tests, treatment options, a pointer to Mayo studies, lifestyle and home remedies, alternative medicine, coping and support, and advice on preparing for the appointment. Doctors and departments names the departments and specialty groups that treat the condition, and Care at Mayo Clinic covers the care team, expertise and rankings, campuses and insurance.
Machine-readable layer. The page head carries the concept ID, Subject tags that list synonyms, Person Group tags for audience age bands such as 13 to 18 years teen, Content Package tags naming the library and syndication set, and a publish date. Embedded schema.org markup describes a MedicalCondition with alternate names and associated anatomy, plus signs and symptoms on one tab and typical tests on the other. Each page closes with a staff byline, references with access dates and links to associated procedures.
Tests, symptoms and the A to Z. Test and procedure guides run Overview, Why it's done, Risks, How you prepare, What you can expect and Results, marked up as MedicalTest or MedicalProcedure, with synonyms such as HbA1c test where they exist. A symptom page gives a definition, possible causes linked to condition pages - abdominal pain splits them into acute, chronic and progressive - and when to see a doctor. The A to Z indexes double as a synonym table: A fib points to atrial fibrillation.
Drugs and supplements. A drug monograph lists US and Canadian brand names, a description, dosage forms, before-using notes on allergies, children, older adults, breastfeeding and interacting drugs, proper use with dosing, missed dose and storage, precautions, and side effects by frequency. That text comes from Merative Micromedex, with its own last-updated date and a copyright line barring commercial use. Herbs and supplements such as melatonin, fish oil and vitamin D are Mayo staff pages instead: overview, what the research says, our take, safety and interactions.
The public Symptom Checker is a three-step form: pick a symptom from an adult or a child list, tick related factors under headings such as Pain is, Onset is, Triggered or worsened by, Relieved by and Accompanied by, then view possible causes. Each checker symptom opens with red-flag advice on when to seek emergency or prompt care. Causes are computed server-side from the ticked boxes, no public endpoint sits behind the form, and Mayo's reprint rules exclude the Symptom Checker with other interactive tools.
A Mayo Clinic symptom checker API is a licensed product, not a public one. The content brochure linked from Mayo's licensing site describes evidence-based symptom triage algorithms offered as an embedded app, Ask Mayo Clinic Online, or delivered by API, returning one of seven care levels: ambulance, emergency care, urgent visit within four hours, acute appointment within 24 hours, routine appointment, provider advice and manage symptoms at home.
The library is licensed the same way. Mayo Clinic Global Business Solutions offers it as Content Connection, by Realtime API or Bulk API, with keyword, fielded and faceted search, HL7 Infobutton support and ICD-10 lookup, in English, Spanish, Arabic and Simplified Chinese, as a full library or topic sets. Site pages even carry a syndication tag: the feed and the website share one editorial base.
Mayo Clinic is a not-for-profit academic medical center with campuses in Rochester, Minnesota, Phoenix and Scottsdale, Arizona, and Jacksonville, Florida, plus the Mayo Clinic Health System and a clinic in London. Its website is both the hospital's front door and a consumer health library written by editorial staff, checked by Mayo medical editors and copyrighted by the Mayo Foundation for Medical Education and Research. Most Mayo Clinic website information sits in five libraries: Diseases and Conditions, Symptoms, Tests and Procedures, Drugs and Supplements, and Healthy Lifestyle.
Addresses share one pattern. Every library page ends in a short type code and an eight-digit number, and pages about the same subject share a concept ID in the page head. A condition is split into tabs: diabetes currently runs /diseases-conditions/diabetes/symptoms-causes/syc-20371444, then diagnosis-treatment (drc), doctors-departments (ddc) and care-at-mayo-clinic (mac), all tied to concept CON-20371420. Tests and procedures sit under /tests-procedures/ with PRC concepts, symptoms under /symptoms/ with SYM IDs, drug monographs under /drugs-supplements/ as ingredient plus route, and departments under /departments-centers/ with ORG concepts. Spanish, Arabic and Simplified Chinese copies live under /es/, /ar/ and /zh-hans/ and keep the same IDs.
robots.txt admits every crawler and closes only /api/, JSON content models, script chunks and /edu/. Its sitemaps are split by content type, from conditions, symptoms and procedures to drugs, departments, expert answers, recipes, videos and London, most with Spanish, Arabic and Chinese twins. In September 2026 they held about 1,230 symptoms-and-causes pages, about 410 test and procedure guides and more than 2,800 drug monographs.
Get a Quote
Developers
Customers worldwide
Pages extracted
Hours saved for our clients
€199 / one-time
setup fee - included
€169 / mo
setup fee €499
€229 / mo
setup fee €499
€349 / mo
setup fee €499
€549 / mo
setup fee €799
Triage and search builders read the library as a graph. Symptom pages link to conditions, and conditions list their signs, typical tests and associated procedures, so one crawl yields symptom-to-condition-to-test edges written in plain language for patients. The public pages carry no ICD-10 codes; where you need them, we map names to ICD-10-CM as a separate step you can review.
Language teams use the translations. A Spanish, Arabic or Simplified Chinese copy keeps the English concept ID and page ID, so sections align into parallel medical text for translation memory or model evaluation.
Research and recruitment teams follow Mayo trials. Each department lists its open studies by campus, and each study on mayo.edu has a record with site IRB numbers and, where registered, an NCT ID that joins to ClinicalTrials.gov.
Pricing analysts want the Mayo Clinic price transparency files. The Arizona, Florida and Rochester hospital groups each publish a CMS standard-charges CSV with gross charge, discounted cash price and negotiated rates by payer, plan and billing code, and each file runs past ten gigabytes. We split, load and version them, so one payer or one code can be queried without opening the whole file.
Learn how to use web scraping to solve data problems for your organization
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
If you specialize in machine learning, you need to feed large amounts of data to the algorithms. Web scraping is the easiest and the most efficient method of collecting the data from all over the Internet.
ScrapeIt builds and operates data collection for clients who would rather not run crawlers in-house. For Mayo Clinic that means a collector per library, a schedule you set, an alert when a template or an ID pattern shifts, and a fix before the next delivery. We are not affiliated with Mayo Clinic or the Mayo Foundation for Medical Education and Research. We never collect patient data, and clinicians appear in our deliveries only as published facts about a department, never as profiles of people.
Not a self-serve one: there is no developer portal, no published endpoint reference and no price list, and robots.txt simply closes the site's own /api/ path to crawlers. The Mayo Clinic API that exists is a content license. Mayo Clinic Global Business Solutions offers Content Connection by Realtime or Bulk API - diseases and conditions, symptoms, tests and procedures, articles, definitions, questions and answers, recipes and videos in four languages, HL7 Infobutton-ready and searchable by ICD-10 - starting from a demo request. Doctor listings, trial records and price files are not among its listed content types.
Mayo publishes its own editorial rules. Pages are written by editorial staff, reviewed by medical editors drawn from its physicians and scientists, and closed with a reference list. Faster-moving topics - diseases, conditions, tests and procedures - are reviewed at least every two years, the whole library is checked each year, factual corrections stay listed for 30 days and stale pages can be archived. Every page shows a date, and drug monographs carry Merative's own update date. We re-crawl monthly by default and report which pages changed date or text since the last run.
For a condition: concept ID, page ID and tab, name and alternate names, each section as its own field, signs and symptoms, typical tests, associated anatomy and procedures, departments that treat it, audience age bands, publish date and references. Tests add synonyms and results; symptoms add linked causes; departments add campus contacts and open studies. English is the base, and the Spanish, Arabic and Simplified Chinese copies under /es/, /ar/ and /zh-hans/ reuse the same IDs, so one record can hold all four languages side by side.
Several exist, but they come from Mayo research rather than the website, and they are not something we collect. The PBC data record the Mayo Clinic trial in primary biliary cholangitis run from 1974 to 1984 and ship with the R survival package. The Low Dose CT Image and Projection Data collection, tied to the 2016 Low Dose CT Grand Challenge run with AAPM and NIBIB, sits on The Cancer Imaging Archive under CC BY 4.0, with scans that could reconstruct a face kept under controlled access. De-identified clinical data for companies goes through Mayo Clinic Platform agreements.
Mayo's terms of use forbid scrapers, crawlers and other automated access, allow one personal, noncommercial copy and fall under Minnesota law; drug pages add Merative's own bar on commercial use, and free reprints are print-only. Internal analysis and republishing are therefore separate questions: we agree the use case with your legal team before the first run, stay on pages open to the public, and send anyone who wants to republish text to Mayo's licensing team or to Merative. Files arrive as JSON Lines, CSV or Parquet, or load straight into your database, each row keyed by concept ID and publish date.
Step 1 - Make a Request
You share your needs, expectations, and desired timeframe. We’ll suggest the best solution based on your request and budget.
Step 2 - Configuring Custom Web Crawlers
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Step 3 - Collect and Deliver
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Step 4 - Maintain and Support
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Scrapeit Sp. z o.o.
10/208 Legionowa str., 15-099, Bialystok, Poland
NIP: 5423457175
REGON: 523384582
