Flight Pricing at Scale: Opodo Scraped with Full Filter Logic

A client came to us for flight ticket prices from the booking site Opodo.com. The data was not meant for a person to read - it fed an analytics product, so the rows had to arrive already narrowed down to exactly the flights the client cared about. The brief was a complete two-month window of fares, needed within days for a product demo.

Sources: opodo.com
Coverage: round trips on 18 long-haul routes from one hub, both directions, every departure date from 1 November to 31 December 2024, return a week later
Frequency: one-off, the full window re-collected several times within a week as the rules were tightened
Volume: 469,366 flight options collected, 13,317 rows in the final file
Delivery: a file per run, with the discounted and the full fare in EUR
Setup and launch: first full file 2 days after the brief, final file on day 7

The filtering had to happen on the site, not in our code

The straightforward way to build a job like this is to pull every result the search returns and sort it out locally. The client asked for the opposite. The filters had to be applied on the page itself, so that the site's own web server did the selection and sent back only the flights matching the requested parameters: direct or one-stop, with no layover longer than five hours.

Instead of a fetch-and-parse task, the scraper has to operate the search interface the way a person would: origin, destination, date, then the filter controls, until the result list is already the answer. Every condition applied on the site is a batch of results we never have to request, download and throw away.

Three conditions decided which row survived

First, coverage: every requested route, every departure date in the window. Second, the return leg had to be operated by the same airline as the outbound leg. Third, for each airline on each departure date, only the best-priced option was kept: one row per carrier per departure date, not a dump of everything that carrier offers.

The stop rules were set inside the interface. Where the airline rules belonged turned out to be the hardest question of the project.

Slider control, which is rare in production automation

Automating that sequence turned into the most involved flow of user interactions we had built at that point. It required driving a slider - moving the maximum stop duration handle to five hours and having the page register the new value. There is nothing to type in; the automated browser has to produce the movement itself.

Making that work once is a demo. Making it hold for an entire run, where the full interaction flow is repeated for every route and departure date, was the real engineering problem on this project.

Opodo results: 469,366 options in, 13,317 rows out

The first full file went out two days after the brief, with about 11,000 rows. The client checked it against the site and found airlines missing. One cause was a filter of our own: airline names were matched against the client's list of carriers, and one carrier was spelled slightly differently on the site, so its flights never reached the file.

From then on the scraper stored everything the site showed for each search and applied the airline rules only when the file was built. Over the two months that came to 469,366 flight options, reduced to 13,317 unique rows. A second pass went back only to the slots where a requested airline had no fare and searched them again with the site's airline filter. It added 2,148 rows, 19% on top of the 11,169 from the first pass.

What the client got in the final Opodo file

The final file arrived seven days after the brief: one row per airline, route and departure day, each already a same-airline round trip at the best available price, with both fares side by side. That was about 70% of the theoretical maximum of 19,154 rows. The rest were combinations Opodo did not show at the time; the client's own checks found fares that appeared a day later.

The client confirmed the file was a clear improvement that missed almost nothing the site offered, enough to build the demo on.

See our Opodo scraper and more travel and hospitality data projects; data can also come through our API or as a JSON feed. To scope fare collection for your own routes, contact our team.