B5894

Date: July 1, 2017

ZSOLT KATONA
BRIAN BELL

Rocket Fuel: Measuring the Effectiveness of Online

Advertising1
“Experiment and learn —the secret of successful
advertising for humans and computers alike.”
—C LARA D EMIDENKO , R OCKET F UEL

It was a typical foggy morning in the San Francisco Bay Area in February 2016 as the low clouds
that had been pushed in by the sea breeze the previous night were slowly starting to burn off.
Tensions were running high in Rocket Fuel’s Redwood City headquarters as Clara Demidenko, a
technical account manager, and her team were putting the final touches on a presentation for an
all-important client meeting. The client’s representatives from TaskaBella Inc., a manufacturer of
women’s luxury accessories, were already late as their morning flight into nearby SFO was
delayed due to the low visibility at the airport. The extra two hours came in handy for Demidenko
to go over the presentation with her team one last time. Their objective was to convince
TaskaBella that the pilot advertising campaign that Rocket Fuel had recently completed was
effective and to present strong evidence that the return on investment (ROI) was positive. The
stakes were high as TaskaBella was in the process of reallocating a significant portion of its
advertising budget spent on handbags and related accessories and was concluding a series of pilot
studies to decide how to move forward.

Rocket Fuel Inc. (NASDAQ: FUEL) was founded in March 2008 with a vision of transforming
the digital advertising industry through big data and artificial intelligence. At a time when real-
time bidding (RTB) and ad exchanges were still nascent, Rocket Fuel quickly grew into the
industry leader by bringing the precision and conclusiveness of science to the world of marketing
to drive higher ROI. By 2016, Rocket Fuel has grown into a business with nearly half a billion

1
Certain names and figures have been altered in the case study to protect the identity of the entities involved.

Associate Professor Zsolt Katona and Brian Bell (MBA’18) prepared this case as the basis for class discussion rather
than to illustrate either effective or ineffective handling of an administrative situation.

Copyright © 2017 by University of California at Berkeley Haas School of Business. All rights reserved. No part of
this publication may be reproduced, stored in a retrieval system, used in a spreadsheet, or transmitted in any form or by
any means without the express written permission of the Berkeley-Haas Case Series.
This document is authorized for use only in Dr. Arunima Shah's Digital Marketing-PGP 2019-21-IV at Indian Institute of Management - Raipur from Jun 2020 to Oct 2020.
 ROCKET FUEL 2

dollars in annual revenue. Rocket Fuel’s full Programmatic Marketing Platform is designed to
leverage artificial intelligence (AI) to determine what marketing actions to take with a particular
person in a particular moment of time. Using Machine Learning (ML) techniques the platform
continuously updates the features of its underlying model to improve campaign effectiveness in
real time.

Just six months before the February 2016 meeting, TaskaBella approached Rocket Fuel to run a
trial campaign for a newly released handbag model. The two teams discussed the overall scope of
the campaign with an objective of targeting around half-million consumers online with
TaskaBella’s handbag ads. Over half of the discussion was centered on measuring the success of
the campaign. TaskaBella was mostly concerned about reaching sufficiently high levels of
conversion rates so that advertising dollars led to enough purchases to justify the spend.
Demidenko explained that CPMs (cost per thousand) vary and were affected by many factors, but
with the target sites and population TaskaBella had in mind, they could expect to pay an average
of around $9 for a thousand impressions. TaskaBella estimated that given the price tag of over
$100 on the handbag and given the variable costs associated with a unit, a converting user is
worth about $40.2

At first, the perspective numbers looked very promising to TaskaBella; based on data from
previous campaigns run with other providers, they estimated that a CPM of around $10 would
translate to a cost of roughly $10 to $15 per conversion. However, Adam Burrup, a TaskaBella
digital marketing manager, pointed out that since the manufacturer already had a strong social
media presence, many consumers may buy the new handbag as a result of word-of-mouth, after
hearing about it from friends. Therefore, it would be important to know the extent to which
conversions can be attributed to the advertising campaign. Demidenko agreed and pointed out
that one of Rocket Fuel’s primary objective in every campaign is to provide evidence that the ads
do make a difference. They do so by carving out a small portion of users reached during the
campaign to form a control group. Users in the control group will be shown a public service
announcement (PSA), such as wildfire prevention or Mothers Against Drunk Driving (MADD),
instead of the ad in the exact same size and position on the page (See examples of PSA’s in
Exhibit 1). By randomly selecting which user is in the control group and which user is exposed,
they can then measure how much difference advertising makes.

After some brief calculations, TaskaBella and Rocket Fuel settled on a control group size of four
percent. While Rocket Fuel’s platform automatically calculates the size of the control group
based on the campaign’s parameters, TaskaBella’s representatives wanted to understand the
factors that determine the resulting size. Keeping in mind that serving PSAs to the control group
costs money and represent a lost opportunity to convert potential customers, TaskaBella wanted
to avoid having too large of a control group. Demidenko had often experienced clients who
wanted to reduce the control group size—sometimes even in the middle of the campaign—and
she was prepared to explain the reasons to avoid a too small control group. First, conversion rates
in display advertising are typically low compared to other experimental settings requiring a
sizable control group to detect lift. Second, even if advertising is effective, the lift might be small.
If the control group is too small and the lift is not very high, the experiment will not be able to
detect a significant lift. Finally, a larger control group allows for a deeper analysis with
potentially significant differences in different segments of the population. For example, Rocket
Fuel’s internal algorithms that decide on ad placement and bids leverage the control group at the

2
Worth of $40 per converted user is calculated by taking the incremental or marginal revenue ($100+) of each handbag
sold less the incremental or marginal costs associated with each handbag sold (material, labor, sales commission, etc.).
The incremental or variable costs do not include fixed costs such as advertising expense.

This document is authorized for use only in Dr. Arunima Shah's Digital Marketing-PGP 2019-21-IV at Indian Institute of Management - Raipur from Jun 2020 to Oct 2020.
 ROCKET FUEL 3

individual level to identify factors that make an impression more or less likely to lead to a
conversion.

The trial campaign ran from November 2015 to February 2016. The ads were first loaded into
Rocket Fuel’s content delivery network by an operations associate, which enables the quick
displaying of the ads anywhere in the world within milliseconds. When the user visits a
publisher’s page (like CNN.com or NYtimes.com), the publisher partners with a supply-side
vendor to put the impression opportunity out to auction. This all happens millions of times per
second across the dozens of such exchanges that Rocket Fuel is integrated with. Rocket Fuel
receives the bid request and needs to respond within 100 milliseconds in order for its bid to
qualify. Rocket Fuel receives a hashed cookie ID in the bid request, which allows it to
anonymously identify the user in its system, along with many other parameters about the ad size
and content. It then applies ML models to predict the probability that this user will take the
required action or actions for each of its campaigns running at the time of the bid. For each
campaign, users that have been assigned to the control group are not actually served a campaign
ad, but a PSA.

By the end of the campaign, roughly 14.5 million impressions were served at an average CPM of
$9. Out of the approximately 590,000 users identified, about 15,000 bought the new handbag.
Rocket Fuel assembled a user-by-user summary of the campaign results by exporting the
appropriate fields from their database. A detailed description of the CSV file containing the data
is given in Exhibit 2. At the end of the campaign, the Rocket Fuel team quickly started working
on analyzing the results in detail. In preparation for the meeting, the team planned to use simple
charts and basic statistical analysis to demonstrate the effectiveness of the campaign. Demidenko
insisted that the Rocket Fuel team should present the results in a format that is digestible for not
only engineers, but for all of TaskaBella’s managers.

While waiting for TaskaBella’s representatives to arrive after the late landing at SFO, Demidenko
discussed the main points of the presentation. The first part of the presentation covered the basic
statistics about the campaign. Most importantly, the total cost of the advertising was compared to
the value of conversions to get an idea about the profitability and the ROI. The next section of the
presentation covered an analysis of the number of impressions users were exposed to and how
these figures impacted the effectiveness of the ad. Finally, the last part of the presentation
explored timing issues by analyzing the days and hours in a week and the changes in response to
advertising.

This document is authorized for use only in Dr. Arunima Shah's Digital Marketing-PGP 2019-21-IV at Indian Institute of Management - Raipur from Jun 2020 to Oct 2020.
 ROCKET FUEL 4

Case Discussion Questions

1. Was the advertising campaign effective? Did additional consumers convert as a result of
the ad campaign?
2. Was the campaign profitable?
a. How much more money did TaskaBella make by running the campaign
(excluding advertising costs)?
b. What was the cost of the campaign?
c. Calculate the ROI of the campaign. Was the campaign profitable?
d. What was the opportunity cost of including a control group; how much more
could have TaskaBella made with a smaller control group or not having a control
group at all?
3. How did the number of impressions seen by each user influence the effectiveness of
advertising?
a. Create a chart of conversion rates as a function of the number of ads displayed to
users. Plot conversion rates for those who were in the control group and for those
who were exposed to the ad. Group together number of impressions as necessary
to obtain a meaningful plot. (Conversion rate means the percentage of unique
users who made a purchase.)
b. What can you infer from the charts? In what region is advertising most effective?
c. What do the above figures imply for the design of the next campaign assuming
that consumer response would be similar?
4. How does consumer response to advertising vary on different days of the week and at
different times of the day?
a. Create a chart with the conversion rates for the control group and the exposed
group as a function of the day of week when they were shown the most
impressions.

b. Create the same chart for hours within a day (excluding the period between
midnight and 8 a.m.).
c. What days/hours is advertising most/least effective?

This document is authorized for use only in Dr. Arunima Shah's Digital Marketing-PGP 2019-21-IV at Indian Institute of Management - Raipur from Jun 2020 to Oct 2020.
 ROCKET FUEL 5

Exhibit 1 Public Service Announcement Examples from the Ad Council

The Ad Council is a non-profit organization that produces, distributes and promotes campaigns
about issues in the interest of the public, focusing on roughly 50 national campaigns at a time.
The campaigns are initiated by government agencies, non-government organizations (NGOs), or
other non-profits. The ad copies are created by volunteer talent from leading agencies and
disseminated through donated ad space and airtime. Below are two examples of the Ad Council’s
national campaigns.

Wildfire Prevention

Texting and Driving Prevention

Source: Ad Council (public domain usage)

This document is authorized for use only in Dr. Arunima Shah's Digital Marketing-PGP 2019-21-IV at Indian Institute of Management - Raipur from Jun 2020 to Oct 2020.
 ROCKET FUEL 6

Exhibit 2 Description of CSV file data set

Each row in the CSV file data set (rocketfuel_data.csv) represents a uniquely identified user in
the ad campaign. For each user, the following six columns are provided.

user id: Unique identifier of the user

test: Whether the user was exposed to advertising or was in the control group. 1 if the user was
exposed to the real ad, 0 if the user was in the control group and was shown a PSA.

converted: Whether the user converted. 1 if the user bought the handbag during the campaign, 0 if
not.

tot_impr: The total number of ad impressions the user encountered. For users in the control group
this counts the number of times they encountered the PSA.For exposed users it counts the number
of times they were shown the ad.

mode_impr_day: Shows the day of the week on which the user encountered the most number of
impressions. 1 means Monday, 7 means Sunday. For example if a given user encountered 2
impressions on Mondays, 3 on Tuesdays, 7 on Wednesdays, 0 on Thursdays and, Fridays, 9 on
Saturdays and 2 on Sundays, this column takes the value of 6 (Saturday).

mode_impr_hour: Shows the hour of the day (0-23) in which the user encountered the most
number of impressions.

Source: Author’s dataset.

This document is authorized for use only in Dr. Arunima Shah's Digital Marketing-PGP 2019-21-IV at Indian Institute of Management - Raipur from Jun 2020 to Oct 2020.