Increasing revenue
through rapid experimentation
Background
Flight Delay Guarantee is a paid service that offers you the option to rebook a disrupted flight on any airline or receive a full refund. As one of Hopper's Fintech products, it was one of the company's biggest revenue generators.
As a Senior Product Designer in the Disruption team, I designed multiple rapid experiments with a focus on re-merchandising Flight Delay Guarantee to increase revenue within a tight timeline. This involved tracking results by creating prototypes with low-code tools such as Webflow, and quickly collecting user feedback. These experiments led to an increase of more than $1 in Revenue per Transaction (RPT).
I have omitted confidential information in this case study to comply with my non-disclosure agreement.
Context
The Flight Delay Guarantee offer
The current design of the Flight Delay Guarantee offer was a bottom sheet presented on top of the Confirm Itinerary screen, and there the users had the option to “Add to my trip” or decline it with “No, thanks”.
Problem
Users wanted flexibility to choose their coverage
Previous research had shown that customers had different needs for different types of travel, so we wanted to experiment with combining the ability to choose your coverage and the price for it.
We had the potential to reach even more users if we gave them the flexibility to choose their coverage
Research
What flight disruptions do users care about the most?
Hypothesis
Users would pay for cancellations coverage
We assumed flight cancellations, being more disruptive than delays, would make users eager for coverage. Research proved the opposite.
Reality
Airlines should compensate users for cancellations
Users feel strongly that when an airline cancels a flight, the airline owns it. Asking them to pay extra for that protection felt like adding insult to injury.
...What about flight delays?
A flight delay coverage looks too good to be true ✨
Users were surprised when they found out that they could purchase protection against 1-hour flight delays, allowing them to rebook their disrupted flight or get a refund for it. A common quote was that this protection “Looks too good to be true”.
Experimentation
Smoke Tests: Testing multiple coverage options
Round 1: 7 different customized coverages and pricing
To run rapid and trackable experiments without the need for engineers to build the interface, I built 7 different variants we wanted to experiment with using Webflow (a low-code tool), and this allowed us to track the success of each variant once we launched it.
What if we split the current coverage into 2 options?
And let users choose what best fits their needs?
This concept offered a single coverage for a smaller price and an option to add delays + cancellations, the upgrade option, for just $5 more. We had two variants: one where the single coverage was Delays, and one where it was Cancellations.
Could the current coverages be more successful if offered alone?
We had two variants where only one of the two coverages was offered — Cancellations or Delays.
If we offered better coverage for a slightly higher price, would users add it?
From time to time, users wanted coverage against cancellations happening anytime between purchase and departure. In this variant, they could choose between same-day coverage (cancellations on the day of travel) or full coverage (cancellations anytime from booking to departure).
Would users be interested in a less comprehensive, but cheaper coverage?
This variant had the goal to understand if users would be willing to pay less for a 2-hour coverage instead of the current 1-hour coverage.
What if coverage went beyond delays and cancellations?
What if users could be covered against ANY type of disruptions during their trip? What if you miss it because you arrived late, or the security lines were too long, really anything? This is the idea behind the most comprehensive variant we tested.
Experiment Results
First-round results: Upgrade wins
By tracking the attach rate of each variant we found a winner: The upgrade concept.
The “upgrade concept” was the most purchased option 🎉
On the research we learned that the small difference in price between the single and the upgraded option made the “combo” option look better.
Cancellation coverages had a good performance 👍
Even though users expected airlines to handle cancellations, the Cancellation-only options were slightly less popular than the Upgrade options.
The Delays offers did not perform well 🕐
This finding was a total surprise — the two options that explored offering only a Flight Delay coverage had low attach rates, and this is unexpected since users mentioned in the interviews that coverage against delays looked like a good deal.
Disruption for any reason had the worst performance 👎
Even more surprising was seeing that the most comprehensive coverage had the lowest attach rate. It may have seemed too good to be true, lacked details, or felt too generic… So for the short-term, it meant that we wouldn't explore this coverage option.
Experimentation
Round 2: Upgrade to cover cancellations or delays?
It’s time for a visual update ✨
Now that the first experiment has ended, I took the time to update the designs to make it more modern and aligned with our brand while getting it ready for the 2nd round of tests.
Delays with an upgrade to cover Cancellations wins
Since Delays coverage alone wasn’t appealing, it likely pushed more users toward the Upgrade option—leading to higher revenue.
Experimentation
Round 3: Preselect to remove friction
Previous experiments ran by other teams at Hopper showed that replacing the radio buttons with a preselected option instead led to a positive impact on the attach rate and overall booking conversion. So we decided to test it.
The pros and cons
Preselect what’s best
By preselecting we made it easier to add the best option to your booking.
Too easy to reject
Users can easily decline the offer by tapping on the “No, thanks” button without even reading the offer.
Pricing psychology: Sort from high to low
Initial prices establish a reference price and people will use them to evaluate subsequent prices. While scanning products that get higher in price, users feel pressured to pounce on a cheaper item before they get too expensive.
The opposite is also true: Each new product feels like a loss in quality. Customers feel pressured to pounce on an option before they lose too much quality. That’s why we chose to change the order of the options from the highest to the lowest price.
Final Results
Business Impact
The preselect variant won, and all 3 rounds of experiments combined led to an increase on both Attach Rate and Revenue per Attach.
