Spot and contract are two prices for the same lane on the same day. Average them together and you have invented a number nobody pays.
ScrapeIt runs the collection as a managed service. You name the lanes and equipment types; we build the pipeline with rate basis and fuel convention on every row, period labels preserved, and hand back CSV, JSON, Excel or a push into your warehouse.
Published trends update on a defined cycle rather than continuously, so cadence follows the publication rather than a daily crawl that mostly re-reads the same figure.
We collect published market data only and do not touch the subscription load board. Crawl rules are honoured and requests paced. The benchmarks are the publisher's product and terms restrict reuse, so the dataset is for internal analysis rather than republication, and your counsel should see the use case before the project starts.
Rate records carry the lane as origin and destination markets, equipment type - van, reefer or flatbed - the rate, its basis as spot or contract, the currency and the period the figure describes.
Rate basis is mandatory on every row. Without it spot and contract collapse into one series and the resulting chart is a blend of two different markets moving for different reasons.
Fuel treatment is recorded explicitly: whether a published figure includes fuel surcharge or is linewhaul only. Diesel moves independently of freight demand, and a rate series that silently switches between the two conventions produces movements that look like market shifts and are accounting.
Equipment type is a first class field because van, reefer and flatbed are separate markets with separate capacity and seasonality, and an all-equipment average describes none of them.
Every row carries the collection timestamp and the period label as published, so a figure can always be tied back to the window it covers.
Lane level series are the main deliverable and they need the period labels handled correctly, since published trends cover defined windows rather than instants and joining them to daily operational data requires knowing which window each figure describes.
Seasonality analysis is one of the better uses: produce seasons, holiday capacity crunches and weather disruption all show in these series, and a client planning a year of freight buying needs the seasonal shape more than the current level.
Spread tracking between spot and contract is the analysis most procurement teams arrive for, and it falls out of the data once the basis field exists.
The boundary we hold is the load board. Live posted loads and trucks are a subscription product carrying commercially sensitive information from brokers and carriers, and we do not collect it. Published market trends are what this source offers openly and what these projects use.
DAT operates the largest load board in North American trucking and publishes rate benchmarks derived from the transactions that flow through it. For anyone pricing or buying truckload freight in the United States, its numbers are a common reference.
Two things need separating before any collection starts. The load board itself - live loads and trucks posted by brokers and carriers - is a subscription product and is not what these projects touch. The published rate trends are open, substantial and directly useful, and they are what a collection here is actually about.
Within those trends the important distinction is spot versus contract. Spot is what the load moves for today; contract is what was agreed for a period. They diverge, sometimes dramatically, and the gap between them is one of the more informative numbers in the market. A dataset that averages them produces a price nobody pays.
The trend pages respond directly with substantial content and crawl rules are published, disallowing search query addresses and nothing that matters to this work. A sitemap index is published.
Get a Quote
Developers
Customers worldwide
Pages extracted
Hours saved for our clients
€199 / one-time
setup fee - included
€169 / mo
setup fee €499
€229 / mo
setup fee €499
€349 / mo
setup fee €499
€549 / mo
setup fee €799
Freight pricing briefs usually ask for a national rate. It exists, it is published, and it is close to useless for anything operational.
Truckload pricing is a lane market. A lane out of a tight origin prices nothing like a lane into one, and the imbalance between directions is what determines whether a carrier will take the load at all. National averages hide that completely, and the lanes a shipper actually runs are the only lanes that matter to them.
The second trap is the spot and contract blend. These are different commitments with different risk, and the spread between them over time is a read on where the market is heading - narrowing when capacity tightens, widening when it loosens. Merged, the spread disappears and so does the signal.
The third is fuel. A rate that includes fuel surcharge moves when diesel moves, which has nothing to do with freight demand. Recording the convention lets an analyst strip it out; not recording it guarantees somebody eventually explains a diesel spike as a demand surge.
The fourth is that published benchmarks are aggregates of transactions, not quotes for your freight. They describe a market and they are not a price you can hold anybody to - we say that plainly rather than letting a benchmark be read as an offer.
Learn how to use web scraping to solve data problems for your organization
Leveraging advances in technology, the AI-powered web scraper has skyrocketed in demand and is helping to expand capabilities by automating tedious daily tasks and speeding up data collection from thousands of websites several times over.
Web scraping is a method of obtaining web data by extracting it from pages of web resources with the help of a program, that is, in automatic mode. It is used to syntactically convert web pages into more usable forms.
If you specialize in machine learning, you need to feed large amounts of data to the algorithms. Web scraping is the easiest and the most efficient method of collecting the data from all over the Internet.
ScrapeIt is a managed extraction company, not a tool you have to learn. Our team builds the pipeline, watches it as trend pages and market definitions change, and flags a definition change rather than letting it arrive as an unexplained step in your series.
You see a sample first, in your format, over the lanes you actually run, with basis and fuel convention visible so you can check the numbers against what you are already paying.
You can, and it will not help you much. Truckload is a lane market - a lane out of a tight origin prices nothing like a lane into one, and directional imbalance decides whether a carrier takes the load at all. National averages hide exactly that.
Because they are different commitments with different risk, and the spread between them over time is a read on where the market is heading. Averaged together the spread disappears, and with it the signal most procurement teams came for.
The convention is recorded on the row - whether a figure includes fuel or is linehaul only. Diesel moves independently of freight demand, and a series that silently switches convention will eventually have somebody explain a diesel spike as a demand surge.
No. Live posted loads and trucks are a subscription product carrying commercially sensitive information from brokers and carriers. Published market trends are what this source offers openly and what these projects use.
No, they are aggregates of transactions describing a market. They are not a price you can hold a carrier to. We say that plainly rather than letting a benchmark be read as an offer, because that misreading is expensive.
Step 1 - Make a Request
You share your needs, expectations, and desired timeframe. We’ll suggest the best solution based on your request and budget.
Step 2 - Configuring Custom Web Crawlers
Our specialists configure the crawlers and extract a sample dataset for your review before proceeding with the full-scale extraction.
Step 3 - Collect and Deliver
Once you approve the sample, we launch the project and start full data collection. We gather, filter, and structure the data for easy use, delivering it on time in your preferred format.
Step 4 - Maintain and Support
Our team manages ongoing processes, monitors website changes, and supports all data extraction cycles. We can also help integrate data into your systems or create dashboards to simplify analysis.
Scrapeit Sp. z o.o.
10/208 Legionowa str., 15-099, Bialystok, Poland
NIP: 5423457175
REGON: 523384582
