There are no articles here, only links and argument. The rank decays by an algorithm, so a single snapshot tells you nothing about what the front page actually looked like.
Plans from €169 a month · Free project assessment · Reply within 1 business day
CSV · Excel · JSON · XML · FTP · SFTP · Amazon S3 · Google Cloud
API, JSON feed or CSV, XLSX and JSON files
ScheduleOne-time, monthly, weekly, daily or 3 times a day
Hacker News pages, filters, fields and a schedule
Built, monitored and repaired by our team
API, JSON feed or CSV, XLSX and JSON files
ScrapeIt builds and runs the pipeline as a managed service. You name the keywords, domains or products you care about; we collect on a schedule tight enough to capture the ranking, preserve thread structure, and hand back CSV, JSON, Excel or a push into your warehouse.
Cadence here is the product. Front page observations need to be frequent enough that an item cannot appear and vanish between passes, while older items need almost no attention at all. We set those separately and say what each costs.
We collect only publicly available data - everything a visitor can see on Hacker News - and we collect it legally, at a civil rate. We treat handles as identifiers rather than as people.
The submission record covers item identifier, title, submitted URL and its resolved domain, submitter, submission timestamp, item type, points, comment count and the discussion URL.
Rank is recorded as observations rather than as a value. Each pass stores the position the item held on the front page and the time of that pass, which is the only way to reconstruct how long something stayed visible. Points and comment count are captured at each observation too, so their growth curves come out of the same data.
The resolved domain of the submitted link is captured deliberately. It is what lets a client ask which publications, vendors or repositories this audience actually links to, which is a very different question from what they comment about.
Comment threads are collected with their structure intact: comment identifier, parent, author, timestamp, depth and text. Flattening a thread destroys the argument, and the argument is the point.
Every row carries the observation timestamp, and derived measures such as time spent on the front page are computed from stored observations rather than supplied as an unexplained number.
Thread structure is preserved because the shape carries meaning. A top level comment with forty replies is a different object from forty separate top level comments, and any summarisation that ignores depth will misread which objection actually mattered.
Brand and product mentions inside threads are extracted and normalised, which is what most commercial clients are here for. The mention that matters is rarely in the title; it is three levels deep in a reply comparing your product with a competitor.
Personal data needs a word. Submitters and commenters are pseudonymous accounts, and we treat the handle as an identifier rather than a person: no attempt to link handles to real identities, no profile building across sites. If a brief asks for that, we will explain why we are not doing it.
Pacing is what keeps this source stable: we space requests, cache aggressively and avoid re-fetching items that cannot have changed. Historical items are immutable once old, so a backfill is collected once and never revisited.
Hacker News is a community submission board. Users post links, other users vote and argue underneath, and an algorithm orders the front page. It publishes no journalism of its own, so anyone expecting article text from this source is looking in the wrong place.
Every item shows the same public record: title, submitted link and its domain, submitter, age, points and comment count, with the full discussion one click away. That uniform shape is what makes the board straightforward to collect in full.
The ranking is the thing that makes this source interesting and the thing most people fail to capture. Position on the front page is a function of points, age and other factors, and it decays continuously. A story can sit near the top for two hours and be gone by lunch. Read once a day, the front page looks almost arbitrary; read on a schedule, it becomes a record of what a large technical audience was paying attention to and for how long.
Comments are the other half. Threads on this board are often longer and more informed than the article they hang under, and for product teams they are a direct channel of unsolicited technical feedback.
Delivery: CSV, JSON or XLSX files, our API or a JSON feed. MCP server on request.
Full plan detailsThe front page is a ranking, and a ranking without time is not data. An item at position three is at position three at that instant; two hours later it may be at thirty, and whether it spent those two hours climbing or falling is the entire story.
So the only useful design observes repeatedly and stores every observation. Out of that fall the measures people actually want: how long an item stayed on the front page, how fast it climbed, where its points curve flattened, how much discussion it generated relative to its score. None of those exist in a single pull, and none can be reconstructed afterwards.
The second reason is the audience. This board is a concentrated population of engineers and founders, and what it surfaces about a developer tool, a launch or an outage tends to lead the wider technical conversation. For a company selling to that audience, appearing here is a measurable event with a measurable duration.
The third is the comment thread. Criticism here is unusually specific and unusually early, and it arrives without a survey. Collected with structure intact, a thread is a usable document; flattened into a blob, it is noise.
The fourth is link destinations. Tracking which domains get submitted and upvoted over time is a clean read on what this audience trusts, and it needs nothing more than the resolved URL.
Learn how to use web scraping to solve data problems for your organization
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
If you specialize in machine learning, you need to feed large amounts of data to the algorithms. Web scraping is the easiest and the most efficient method of collecting the data from all over the Internet.
ScrapeIt is a managed extraction company, not a tool you have to learn. Our team builds the pipeline, watches it as the board changes, and repairs it before a gap opens in a ranking series whose whole value is continuity.
You see a sample first, in your format, over your keywords and domains, with rank observations already populated from a real window so you can judge the cadence rather than take it on trust.
Still unsure? Write to info@scrapeit.io
No, because it is not there. This is a submission board: links, points, and discussion. If you need the articles themselves they have to come from the publishers, which is a separate collection we often run alongside this one and join on the resolved URL.
Because the front page is a ranking that decays continuously. An item can climb, sit near the top for two hours and be gone by lunch. Read once a day, the board looks arbitrary. Read on a tight schedule, you get how long something stayed visible and how fast it climbed, and neither can be reconstructed afterwards.
Yes, with structure intact: identifier, parent, author, timestamp, depth and text. The shape carries meaning - a top level comment with forty replies is not the same as forty separate comments - and flattening a thread destroys the argument, which is usually what a client is actually here for.
Yes. The ScrapeIt Hacker News API returns item identifier, title, resolved domain, points, comment count and rank observations as JSON from an endpoint we host, refreshed on the schedule you set. The same data also comes as CSV or XLSX files or straight into your database.
We treat handles as identifiers, not as people. No attempt is made to link accounts to real identities and we do not build cross site profiles. If a brief asks for that, we explain why we are not doing it - the analysis a client actually needs never requires it.
You share your needs, expectations, and desired timeframe. We'll suggest the best solution based on your request and budget.
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Tell us which sites, which fields and how often. A couple of lines is enough to start.
Hacker News data from €169/month. Free project assessment, reply within 1 business day.