Quantified Nation #7: Modern Measurement
¿Hablas español?
Check our the first chapter of our podcast “Comentando Quantified Nation”
Google’s Modern Measurement Playbook
Hot of the press paper from Google measurement experts, the "Modern Measurement Playbook".
This 2024 update led by Ana Carreira, builds on the 2019 works on the "3 Grand Challenges" by Matthew Taylor, Ludwig Bruetting, Jonas Bruus Christensen and Baptiste Tougeron, where the framework and foundations were laid off. It also builds on the “Media effectiveness guide for CMOs (and CFOs)” published in 2023. If you are not familiar with these two, set some time to go over them as well.
This playbook is a must-read for anyone working in media measurement. It offers valuable insights into how companies are combining MMMs, experiments, and attribution.
While the playbook has garnered significant attention—which we'll discuss later—let's first deep dive into its content:
The stated goal is to use media effectiveness measurement to make better business decisions. It provides fundamentals, guidelines, best practices, and a framework for implementation. Although comprehensive, there are areas ripe for expansion: a deeper dive into reach and frequency metrics, a reimagining of attribution models, a broader integration of brand measurement, and addressing the challenges of working with calibration multipliers over time. Nevertheless, this publication marks a significant step forward in establishing best practices, and it's exciting to envision what future iterations might bring.
Let's begin with the last chapter. Why? Because it focuses on the end goal of the playbook: building your own measurement framework.
It presents a comprehensive yet theoretical view of an ideal media effectiveness measurement system. This involves a nested MMM incorporating brand metrics, calibrated with incrementality at a channel level. Additionally, it calls for a cross-channel, data-driven attribution model—also calibrated with incrementality—and inclusive of brand lift studies. The framework emphasizes the need for a learning agenda to ensure continuous calibration and a process for synthesizing this wealth of information into actionable business decisions. It's undeniably a challenge, but the potential payoff is immense. Now, let's turn to the first chapter to unravel the fundamentals.
The playbook establishes two foundational goals for media effectiveness measurement: (1) understanding the full impact of media investments and (2) understanding how to optimize these investments.
I see these two goals as interconnected points on the path to better marketing decisions within an organization and within the technology that empowers much of today’s marketing. Regardless of the specific system you use, it must align both long-term budget allocation and short-term optimizations with key business KPIs. These are two sides of the same coin, and a robust measurement framework is the solution.
Essential building blocks for this setup include defining the right business KPIs, embracing a test-and-learn mentality, and ensuring proper data collection, governance, and visualization. While the playbook doesn't delve deeply into these areas, mastering them is crucial before embarking on a comprehensive media effectiveness measurement (MEM) framework.
This framework which combines MMMs, experiments, and attribution, is deeply explained looking into each method's benefits, challenges, and best practices. A crucial understanding is why these methodologies often yield discrepancies.
For example, the playbook provides a shortcut to grasp the attributed vs. incremental concept when comparing attribution to incrementality results:
View each tool’s output to set the upper and lower bounds of performance. For digital click-based channels, in-house attribution typically represents a generous view of that strategy’s contribution (i.e. the upper-bound), and incrementality represents the most conservative view (i.e. the lower-bound). Your MMM’s assessment should fall within the range of these bounds.
Integrating these solutions is a hot topic in marketing measurement, and the playbook outlines how this can evolve through three stages. It begins with (1) comparing different sources, then (2) introducing calibration when possible, and finally (3) reaching a sophisticated stage with a continuous feedback loop between tools, inclusive of a learning agenda.
While the playbook discourages seeking a single source of truth, the reality is that such a source never truly existed. Creating a new media effectiveness framework that integrates diverse data sources appears to be the most promising approach. This is summarized in the following image.
The playbook's next section looks into specific guidelines and best practices across key areas:
Transform your knowledge of media effectiveness into actions
Clear planning, KPI setting, and target definition are emphasized. While establishing incrementality-based targets is a sound idea, it can be challenging to execute consistently unless you have a standardized experimentation approach across various conditions. While I haven't personally witnessed this level of rigor yet, I believe it's achievable in the near future.
Attribution plays a crucial role in this framework by providing baselines for campaign and channel optimization. Although attribution is often criticized for its "incidental" rather than incremental approach to measurement, it still provides essential data for the algorithms and AI that drive much of today's digital marketing. Aligning your attribution model with your business goals is key to maximizing the effectiveness of your digital marketing efforts.
Combine incrementality experiments results with data-driven attribution
The playbook offers a straightforward, albeit somewhat naive, example of taking iROAS results from an incrementality experiment, comparing them to attributed ROAS, and using this comparison to create calibration multipliers for future attribution results. While this approach has potential, it's difficult to scale due to the need for incrementality tests across most, if not all, relevant channels.
Combine attribution results with MMM
Three key areas are highlighted:
Utilizing attribution for speed and granularity unavailable in MMMs.
Calibrating attribution outputs with MMM results.
Using attribution results to eliminate flawed MMM model versions.
Option 1 is the most common approach, while 2 and 3 offer valuable checks to ensure your MMM exercise stays on track.
Use results from incrementality experiments to enrich MMM
This is a personal favorite! MMMs should guide where to conduct incrementality experiments. Discrepancies between MMM ROAS and incrementality experiment results should be investigated. Calibration of MMMs using incrementality experiment results can be done with both frequentist and Bayesian approaches, each with unique considerations.
We explored this topic in our previous newsletter on the launch of Meridian, highlighting its ability to support this particular approach.
Design incrementality experiments following best practices
A clear learning agenda grounded in existing knowledge is vital for implementing this vision. Robust experiment design methodology based on business questions and hypotheses, alongside consistent KPIs and conclusions, are crucial for adherence to best practices.
Knowing when not to use incrementality is equally important. Page 28 of the playbook dedicates itself to this topic, covering key considerations. Appendices 1 and 2 offer a deeper dive into this complex issue.
Integrate brand measurement within your framework
The playbook touches on the importance of brand measurement but doesn't fully explore its integration. However, this is an area ripe for future development. Nested brand-equity MMMs are essential for understanding how brand building generates long-term business value and guiding execution.
Modern measurement heavily emphasizes business outcomes. Integrating brand measurement requires a way to normalize it against these outcomes. A closer look at "Modern Brand Measurement" was recently published on Think With Google here.
Planning and optimizing for brand outcomes shouldn't be done in isolation or opposition to planning and optimizing for sales or profit. This is easier said than done, but achieving this balance is crucial for long-term success.
Want to know where to start in your measurement journey and where to go next? It's in the paper.
Want to know best practices on how to calibrate MMM and experiments? You'll find them on the paper.
Want to avoid the most common mistakes? Read the doc!
Remember, this is a Must read!
Link 👇 , find the attached PDF for the full content!
Reactions to the playbook launch
As expected, the industry has reacted to the playbook. Professor Koen Pauwels has published an article (link) sharing his views. His first point is on how the calibration of MMM with A/B tests should we exercised with caution, which we can’t agree more. His second point is on his views and also other industry thought leaders on the limitations of attribution as a 3rd pilar of a measure. As mentioned above, attribution results are essential for the optimization of many digital marketing campaigns so it is very important to have it aligned with business outcomes. Just because of this fact, it deserves a place in the framework.
Also, for a deeper review we recommend you to check the live podcast by János Moldvay, Co-Founder & CEO at Adtriba covering the playbook.
Worth mentioning as well some comments on the review for companies or SaaS offering this framework as a service. Jim Kingsbury mentions names such as Measured, LiftLab, Recast or Haus. While these options do not offer a comprehensive approach to this framework, it is exciting to see the future of Modern Marketing Measurement and the opportunities ahead to improve how media effectiveness can help business results.
Industry updates and upcoming events
Flat101 has just released their 2024 conversion rate benchmark study. You can download here. Over the years, Flat101 has managed to collaborate with hundreds of real businesses to create this yearly study. This time over 1000 companies have shared their data, mostly on the Spanish market. Extremely valuable document with results by vertical, format… so companies know where they stand.
Open Budget Allocator - Macarena Estévez
Spanish’s analytics legend Macarena Estévez is hosting an event to launch is latest initiative OBA (Open Budget Allocator). You can register for the May 22nd event on this link. While the details will be announced on the event, the tool is described as a free media allocation tool using official data.
Measurecamp Madrid is almost here!
Measurecamp Spain will happen on May 25th. Last ticket release already passed so if you were so lucky to get your ticket, let’s meet there!
Blueprint for Brand Measurement - Kantar
Kantar seems to be up for a big thing in the brand building arena. Checkout their webinars on this link.
Chart of the week
The Peppa Pig era has come to an end, behold the new era of Bluey!
Bluey is the latest and hottest Australian export, seen it?
Oldies but goodies
Lookout by System1’s Orlando Wood is an ode to advertising creativity throught the scientific exploration of advertising attention. It’s published by the IPA (link) and you can purchase on their own site or in Amazon.
