QuantifiedNation #5 The Incremental Path to Experimentation
Food for thought
We have discussed about KPIs for the last editions of Quantified Nation. On QN#3 KPIs and Attribution or QN#4 Brand Measurement.
Tempting as it was to dive straight into digital metrics, it's crucial to highlight other vital metrics in marketing decision-making: experimentation results, such as A/B testing or incrementality studies. We often lean heavily on digital metrics for quick decisions. But why not apply the same diligence to experimentation outcomes? These tests offer the gold standard for understanding the causal link between a marketing "intervention" and its business impact.
This brings us to a pivotal milestone in marketing measurement: understanding the dynamics between all marketing efforts and business results. Experimentation allows us to untangle these relationships one at a time, or a few at a time. If you're not leveraging today insights from past experiments for current initiatives, you're likely missing out on significant gains.
Are you gearing up for your next experiment? Find unanswered questions in your team. Define hypothesis, design the test and run it. Learn, make decisions, start again.
Requesting the rollout of a new marketing technology solution for enhanced reporting or an improved attribution model is common. Meanwhile, marketing activities continue unabated. With years of media spending, you can build a strong case for a Marketing Mix Model, allowing for ongoing campaign activities.
Experimentation, however, demands a specific campaign or experience setup, where insights are tied to particular times and contexts. By isolating test and control groups, we ensure the difference is due to the marketing campaign. This is the opposite to attribution or marketing mix modeling results, which while also reflect past performances, they do not provide the direct causal insights experimentation provides.
In today's intricate marketing arena, where a plethora of channels and tactics vie for our attention and budgets, pinpointing success has become more pivotal—and more daunting—than ever. Traditional metrics like impressions, clicks, or even data-driven attributed conversions can sometimes mislead, offering a fragmented view of our marketing impact.
As we progress in this newsletter, I hope you share our concerns about relying solely on ROI metrics from platform data or even cross-channel data-driven attribution. Do they truly serve as your source of truth? How well do they capture the user journey?
Identifying users across their journey poses significant technical and regulatory challenges. We need more than digital tracking to accurately measure the real impact of marketing activities on business outcomes. This means setting up scenarios that help us determine which marketing tactic is superior (A/B testing) or how a campaign directly influenced a business result (incrementality testing).
Let's zero in on incrementality testing.
What is Incrementality, and Why Does it Matter
Imagine marketing as a series of ripples on water. Each campaign or initiative is a pebble, its impact spreading far and wide. However, not all ripples reach the shore. Incrementality helps us measure the true distance traveled by the ripples for each pebble, distinguishing the direct contributions of our efforts from the background waves of market movement and external factors.
At its core, incrementality is about cutting through the noise to identify the true outcomes resulted by our campaigns, sidelining all else. Consider this: a slice of your sales would probably happen with or without your marketing magic. Some consumers are already in tune with your brand, primed to convert, or influenced by other factors. Incrementality is the tool that allow us to isolate the real uplift—whether it's in sales, consideration, or brand awareness—that directly results from your marketing magic. In the complex arena of marketing effectiveness, incrementality shines a spotlight on the causal dynamics often hidden in the shadows of correlation.
Understanding incrementality is not just about measuring outcomes; it's about recognizing the real value that our marketing initiatives bring to the table, allowing us to allocate resources more effectively and drive genuine growth. In essence, incrementality testing is a critical solution in our marketing toolbox, offering a clear, undistorted view of our campaigns' effectiveness and guiding us toward more strategic, impactful marketing decisions.
Thinking incrementality first is a powerful critical lens for our marketing strategy
The Path to Measuring Incrementality
Mastering incrementality comes with its challenges, but the benefits are significant. The focus of different incrementality approaches is to establish how to set up test and control groups. We expose the campaign to the test group but not to the control group, and measure the outcomes after a cooldown period of 2-3 weeks. Here are the primary methods:
User-Based experiments: Advances in privacy-focused technology enable tracking individual user journeys across platforms—while safeguarding privacy—to determine the incremental impact of each interaction. Many advertising platforms (such as Google Ads or Meta Ads) and internal channels like email or web personalization offer user-based incrementality solutions and insights.
Geo-Experimentation: This involves running campaigns in specific regions while leaving others unexposed to measure outcome differences and provide strong evidence of incremental lift.
This method is particularly valuable for complicated scenarios, for example assessing upper-funnel marketing activities, allowing enough time to gauge the impact on brand awareness or consideration. However, the risk of inconclusive results increases if the experiment extends beyond three months. Incrementality testing is most effective for impacts occurring within weeks after a campaign. It requires a clear, measurable objective, with daily and regional detail to compare test and control areas.
You can use established methodologies like Google’s Trimmed Match (documentation available in Python) or Meta’s GeoLift (documentation available in R) to define your test and control groups based on historical data and calculate incrementality post-campaign.
Synthetic Approach: Advanced modeling techniques tailored for incrementality allow you to examine historical data and external factors to identify influencing relationships on your results. This approach doesn't require a test and control group, making it an indirect solution for incrementality. Yet, it's essential for any marketing analyst’s toolkit to discover relationships suggesting causality and to hypothesize for future incrementality tests.
These methodologies involve time series analysis to understand how an intervention affects the data over time. The Google package CausalImpact is a reliable option for adopting this strategy.
There are many considerations when looking into how to incorporate incrementality in your decision making, all of them require a change in the mindset to embrace test and learn while avoiding settling with recurrent KPIs. Here are some key steps:
Embrace Experimentation: Aim to design A/B tests, even on a small scale. Seek support from your technology providers for conducting user-based incrementality experiments.
Dive into Geo-Testing: When possible, launch pilot campaigns in targeted markets, keeping control regions aside for accurate comparisons.
Invest in expertise: Mastering incrementality demands data science and analytical prowess. Either cultivate these skills internally or collaborate with specialized partners.
Start Simple, Then Expand: Initially, concentrate on a single channel or campaign. Gradually, broaden your incrementality practices to encompass your entire marketing mix.
Measuring incrementality is an ongoing journey rather than a singular goal. By moving your focus from superficial metrics to genuine impact, you'll open the door to a new realm of marketing excellence—powered by data-driven insights and a dedication to perpetual enhancement.
Hot takes - Industry updates
Deloitte on open-source MMMs
Deloitte just published their POV regarding open-source MMM (link) and how to succeed on the insourcing-outsourcing MMM dichotomy. This is a great piece of professional advice on approaching a MMM project within your company or even providing such service as a consultant. It’s important to remember that MMM is so much about the process as it is about the methodology. As one could expect, a hybrid approach has the most potential, but it’s the hardest to manage. Either way, each organization should reflect on to best complement their own operational capabilities/limitations and not limit the decision to methodological aspects.
Marketing is an investment - Paid report
The Institute for Practitioners in Advertising has just released their report “Marketing is an investment” that you can download for a very reasonable fee on this link. We’ve read it and strongly recommend. It has a unique point of view on the very-much-researched CMO-CFO topic, including interviews to financial analysts. In searching the Holy Grail of positioning marketing as an investment, one can’t avoid the fundamental fact that accounting practices for CAPEX and COGS are very formal. It was also revealing to see that ROI is a debated metric among CFO.
CMOs are well alive
Are we observing the demise of the CMO, their transformation into a new scope, rising into stars of the organization (forelated reading)? That’s a usual subject of debates, whitepapers, research pieces. Mark Ritson advocates in a recent column (link) for a no-drama position regarding CMOs, after all things very seldom change so drastically.
The hype cycle for online advertising
What about marketing effectiveness? A recent report by Dr. Grace Kite points that digital marketing performance is getting better (link).
This is happening mainly for three reasons:
Test and learn: brands are adopting experimentation more broadly,
Learning by measuring: brands are moving away from last click attribution and integrating econometrics/MMM and lift testing
Learning by learning: professionals are today much more prepared thanks to the availability of learning resources
Industry updates and events
Register to the Think Measurement with Google event
Google is hosting a measurement themed event at the 27th of March. You can register clicking here. This is the third session of the event called “Think Measurement” and you can also watch on demand sessions 1 and 2, covering topics such as consented and durable measurement or Google Analytics.
This third session will also be focused on how to mitigate the impact of changes in regulations and technology on measurement. That is, how to capture consent signals and have modeled conversions and what to implement now that third party cookies are going away soon. We hope you enjoy it!
Chart of the week
Lots of very different datapoints are pointing into the direction of weaker social networks, more loneliness… Another one for the debate, this one on card games.
Want more insights? Get them on your own clicking here
Learning corner
If you want to give first steps on measuring incrementality within your digital campaigns, you can check this Google course on Incrementality and these Meta courses on Measurement and Incrementality. These are essentials for anyone embarking in the world of incrementality.
Gran idea hacer podcast para complementar!!! Se entiende mejor y se agradece que sea cortito!!!