QN#15: Meridian is now available for everyone
Google’s Meridian Launch
After months of invitation-only, Google has fully opened Meridian for public use.
Meridian is a framework for building Marketing Mix Models (MMMs). Provided as an open-source Python library, it offers highly customizable options to meet the needs of most companies with various marketing channels. Meridian helps measure the impact of marketing channels, while also accounting for non-marketing variables. Similar to other MMM frameworks, it uses aggregated time-series data on daily/weekly media investment and other media metrics, along with marketing and non-marketing data in the same time-series format, to explain their contribution to sales or other business KPIs.
If you have experience building MMMs and Python knowledge, diving deeper into Meridian is a must for you. The README in the GitHub repository and the “Getting started guide” which allows you to run Meridian end-to-end in a Colab notebook with dummy data, are excellent starting points.
Before starting to get your hands into it or partnering with a data scientist to use Meridian within your organization, we recommend reading further this episode of Quantified Nation to gain background on the research behind it and the ecosystem's response to its launch.
MMM Research at Google: the road to Meridian
Meridian is the culmination of extensive research at Google, focusing on the application of Bayesian statistics to model the impact of advertising on consumer behavior. This involves accounting for prior knowledge, lagged effects, diminishing returns and saturation, and selection bias from ad targeting, which affects search ad evaluation, among other considerations. This is an ongoing research with many interesting additions in the roadmap which will be included in future releases of Meridian. There's a bright future ahead!
In recent years, numerous experts from academia and industry have contributed to the advancement of MMM models in various ways. The rapid evolution of this field is remarkable. Google has contributed to this progress with research that forms the foundation of Meridian. Key research publications include:
Challenges and Opportunities in Media Mix Modeling (2017 - 3803.pdf)
Geo-level Bayesian Hierarchical Media Mix Modeling (2017 - 3804.pdf)
A Hierarchical Bayesian Approach to Improve Media Mix Models Using Category Data (2017 - 3805.pdf)
Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects (2017 - 3806.pdf)
Bias Correction For Paid Search In Media Mix Modeling (2018 - 4388.pdf)
Bayesian Hierarchical Media Mix Model Incorporating Reach and Frequency Data (2023 - 7327.pdf)
Media Mix Model Calibration With Bayesian Priors (2024 - 7494.pdf)
The four publications published in 2017 (note the consecutive numbering in the Google Research archive) lay the groundwork for the overall methodology. The three most recent publications cover each of the key innovations introduced by Meridian, as highlighted in Quantified Nation #6:
Control for organic demand by incorporating search query volume data to mitigate paid search bias.
Enhance video measurement actionability by modeling reach and frequency.
Improve MMM accuracy by integrating incrementality experiments as prior knowledge.
Recognizing the volume of material, we've created a 27-minutes podcast using NotebookLM, summarizing this research (available also in our YouTube channel). Whether you're new to MMM or an experienced practitioner, we recommend you to listen to this audio. It summarizes the most relevant research Google has contributed to MMM and distributed as a product through Meridian. Enjoy it!
Here’s a quote from the AI generated podcast:
[…] -Speaker 1: The accuracy of your MMM results depends on the quality of the data you feed it. Garbage in, garbage out, as they say
-Speaker 2: Speaking of garbage, one of the biggest takeaways for me is the importance of understanding and mitigating bias […]
Meridian also incorporates insights from other recent research, such as Uber’s Bayesian Time Varying Coefficient Model with Applications to Marketing Mix Modeling (2021) as well as foundational work on causal inference and applied Bayesian inference from Gelman and Rubin’s Iterative Simulation Using Multiple Sequences (1992).
TensorFlow Probability is the responsible for probabilistic reasoning and statistical analysis behind Meridian and it recommends using a minimum of one GPU for optimal performance .
The response of the ecosystem
This project is very close to our hearts, and we recognize that we may have some personal feelings about it. To ensure objectivity, we conducted social listening across various LinkedIn posts, Medium articles, and other sources discussing this Google project and analyzed these discussions using Google AI Studio. While many of the reviews are based solely on the official documentation, it's valuable to summarize the ecosystem's reaction. Here's a summary of the pros and cons:
Pros:
Bayesian Framework: Enables incorporating prior knowledge, improves stability with limited data, and provides uncertainty quantification.
Geo-Level Hierarchical Modeling: Captures regional nuances and variations, improves statistical power for regions with sparse data.
Reach and Frequency Integration: Models ad saturation effects more realistically, potentially leading to more precise measurement of incremental reach.
Google Search Query Volume (GQV) Incorporation: Provides a control variable to account for external confounding factors impacting organic demand.
Open Source: Meridian boasts a user-friendly interface that enhances the efficiency and productivity of analytical teams.
Cons:
Prior Specification Challenges: Subjectivity in prior selection, risk of "same prior, same posterior" if priors are poorly informed.
Potential for Multicollinearity: Bayesian MMMs can be more susceptible to inflated posteriors under multicollinearity, hindering accurate attribution.
GQV as a Control Variable: Accuracy depends on GQV reflecting true organic demand. There is also a risk of under-attributing impact of channels that drive search volume.
Lack of Time-Varying Coefficients: Assumes constant media effects, limiting ability to capture dynamic responses and seasonal fluctuations.
Let’s review some individual examples of comments left, mostly by MMM providers:
Andy Kozak from Forvio, highlights that current MMMs often treat Reach and Frequency as secondary considerations, and that Meridian will facilitate the measurement of true incremental reach. He also notes key differences between Meridian and Robyn in how incrementality knowledge is incorporated. Furthermore, he points out that if a marketing campaign influences Google queries for the brand—as many do—using query volumes as an estimator of demand risks underestimating the campaign's contribution within the model.
Aryma Labs published an extensive review in their Substack newsletter, primarily criticizing Meridian's Bayesian approach from an efficiency and almos philosophical standpoint. They argue that prior settings significantly influence the posterior in MMM, which is inherently a "small data" problem. They also contend that Bayesian MMMs are more susceptible to manipulation than Frequentist MMMs, the approach they advocate. This represents a strong contrarian view on Bayesian modeling.
Kishalaya Mukhopadhyay reports on one of Meridian's key parameters: knots. To capture the effect of different time periods on the measured outcome, Meridian uses "knots" to represent time effects with fewer parameters, rather than assigning a separate parameter to each individual time period. More knots result in a smoother line, effectively capturing seasonality by determining each time period's effect as a weighted average of the nearby knots.
This eMarketer report on Meridian’s launch provides additional insights into MMM usage within US companies, as well as other measurement solutions. It reinforces the need to combine various measurement methodologies for the most accurate assessment of marketing performance, a much appreciated approach here in Quantified Nation. It appears that multi-channel measurement is a key focus this year, even more than GenAI:
In case you are looking for a step by step video, Gabriele Franco, the founder of Cassandra (an MMM SaaS solution) has published a tutorial with almost 5K views already, that’s a lot.
Henry Innis is the founder of another MMM provider called Mutinex. He published an interesting view on the open source vs closed model, where vendors have to protect their IP to innovate and differentiate while at the same time it is needed to enhance trust and transparency.
It is great to see so much enthusiasm about MMMs. We expect it to keep growing as more companies adopt it and there is much more scrutiny on how it is used. There seems to be a great future ahead for Meridian!
Industry Updates
Felipe Thomaz’s article on multimedia planning
We read Felipe Thomaz' latest article, which was referenced earlier in QN12 and several forums such as this Mi3 Interview. This is supposed to set a new paradigm in the bothism era, after all the Byron-Sharp school's most aggressive claims are facing increasing resistance (for example the renewed interest in differentiation which was anathema until recently). As usual on these academic papers, they are lengthy and rather hard to read, but there are nuggets that are worth following.
This paper is not just “another one”. Because of the gravitas of the authors, the size/complexity of the research and its confrontational nature. This could be part of the corpus of a new era. On the limitation side, the insights are not really "elevator pitch ready" and that will hinder its popularity. In fact, trying to summarize the paper could result in it sounding "old news" where in reality it's not. There's enough argumentation in the first pages on why this piece is relevant vs. the previous methodologies.
The paper analyzes a very sizable database of Kantar's x-media campaigns database which includes 1) Media types and 2) Brand results. What does the paper argue?
First, the paper looks at all the campaigns in the database and compares their results vs. a perfect scenario where ideal results would be obtained. They call it a "stochastic frontier analysis". The results show that 80% of campaigns are able to reach >80% of their "reach frontier", meaning that most campaigns are doing pretty well in terms of reach optimization. However, only 1% of the campaigns are able to achieve >80% of their "brand lift frontier". The author argues this is the fault of mindlessly pursuing reach-optimization as a strategy. That's the starting point and the rest of the paper focuses on 4 brand-lift metrics.
Second, the authors analyze the 1083 campaigns in the database and build 7 clusters of "archetypes", similar to when one does a consumer segmentation study. This is an interesting and useful approach to analyzing campaign types.
Third, there is a long list of results on how the different archetypes and individual media deliver on brand-lift results. Since there are a lot of results, this allows for several ways of interpreting the data.
As an archetype, the highest results belong to #4 (which is a mix of TV+Outdoor+Facebook+Display). Another interesting one is #3 has both the highest "low confidence interval" and 2nd highest mean result. This means that it's the best proposition in terms of risk/reward.
Because all archetypes are a mix of lots of media, one can isolate single media beyond the archetypes. In that scenario, YouTube is consistently #2 behind Cinema (which is a really niche media).
One of the authors has been pretty harsh on Byron Sharp publicly several times and this paper would insist in that direction, in particular it criticizes the reach-maximization media planning tactics. In a way, that's the main agenda of the paper. Interestingly, Byron Sharp or his colleagues are not mentioned by name.
Meta Measurement virtual session
Meta will be hosting “Project Robyn & Meta Open Source Measurement Global Community Connect” on February 25th. Registration link
Latest WARC report “The multiplier effect”
A 100 page report with case studies and meta-analysis data that can be downloaded here. Given its ambition, it’s very difficult to summarize. Every marketer and measurement expert will find useful data points, insights and good visuals.
The core message is that Brand and Performance make sense in an integrated way: there should be no “brand campaigns” or “performance teams”, because they both work together and in fact reinforce each other. A strong brand will make the performance marketing results even better and also help to command a price premium. This goes very much in line with the growing consensus on how advertising works, beautifully summarized by Les Binet. What readers of the report will find is even more data points on why it’s the case.
In what regards to measurement, the paper takes a hierarchical approach on the measurement triangle (MMM, experiments, attribution) and puts MMM in the most important position, with experiments and attribution as complementary.
Chart of the week
Spring is coming soon! It’s remarkable that there is some consistency across years and across countries with such a different climate!
Oldies but goldies
Today we want to bring an old book which ideas remain in marketers minds for a strong reason. "Scientific Advertising”, written in 1923 by Claude C. Hopkins. Available for download here
This book advocated for a shift from creative expression to data-driven salesmanship (we know both are needed), pushing the idea that advertising should be treated as a quantifiable investment, demanding measurable results (all in!). He started techniques like couponing and testing in direct mail to track campaign effectiveness, establishing the basis for modern direct marketing and testing.
Furthermore, Hopkins stressed the importance of understanding consumer psychology, urging marketers to present their messages from the customer's viewpoint. He advocated for specific, factual claims over vague assertions, believing consumers respond better to concrete information. The power of compelling headlines, designed to draw readers into the full advertisement, was also a key principle. Look at who wrote the introduction to the 1960 edition:
Nobody, at any level, should be allowed to have anything to do with advertising until he has read this book seven times. It changed the course of my life.
David Ogilvy
This book is a great read for every marketer who wants to make data driven decisions and enjoys understanding the history of it, promoting an analytical approach, measurable results and deep consumer understanding. No doubt these are principles that are still highly relevant in today's marketing landscape.
Thank you for reaching the bottom of the newsletter, share and see you soon!