The stats/data group FiveThirtyEight has made a big deal recently about expanding their analysis and coverage into more and different types of research. Earlier this week that took the form of trying to validate the fare predictor service offered up by Kayak.com. And wow did they do it badly.
There are pretty charts and even some data included in the report, but it is far from a coherent or even reasonably structured. I’ll forgive that the search started only two weeks out from travel, even though that limits the value of the analysis. Even with that limit there should still be some tracking on what changes are happening in the fares. And that’s where the research really falls apart. He didn’t follow through on tracking the data.
I started my test on March 29. I searched for non-stop economy fares on Kayak for 32 of the most popular domestic routes, including New York to Chicago, San Francisco to Seattle and Los Angeles to Miami, specifying a departure of April 12 — two weeks out — and a return of April 18. …
Kayak issued immediate “buy” recommendations for 17 of the 32 queries. Given that I would have accepted the 14-day-out prices anyway, using Kayak made no difference for these routes, and so I stopped analyzing them.
So we don’t really know if Kayak was correct on those 17 recommendations. That the fare was acceptable to the customer is fine, but that would have been the case without the recommendation engine. And we have no idea if the fares ended up lower at some point the in following two weeks. Just having a fare that is acceptable doesn’t mean that it is the best fare offered in the timeframe.
There’s also the part where purchasing a ticket skews future fares and without actually completing the transactions we cannot know if the fare prediction was true or not. Then again, the researcher stopped tracking the fares after deciding it was okay so this wasn’t really in play for a variety of reasons.
I’d love to see some better research and validation on this topic. I think there is probably some decent data to be gathered and mined. If only I had the time…
The post Testing the fare predictors. Badly. appeared first on Wandering Aramean.
No comments:
Post a Comment