_edited.png)
Determinants of behavior and their efficacy as targets of behavioral change interventions: A meta-meta-analysis
Alternate title: "Throwing spaghetti at the wall to see what sticks: A scattershot approach to behaviour change"
In their ambitious paper, Albarracín et al. aim to synthesize multidisciplinary meta-analyses to identify the most effective determinants for behaviour change interventions. It’s a paper that attempts to boldly go where many have gone before - trying to make sense of the vast and tangled literature on behaviour change interventions. Indeed, the authors should be commended for their bravery, as taking on this task is like trying to untangle a giant ball of yarn after a dozen kittens have had their way with it!
The key points of the article:
The authors conducted a meta-meta-analysis to identify the most effective individual and social-structural determinants of behaviour change across various domains.
Habits, access, and social support were found to be the most effective intervention targets, while knowledge, general skills, general attitudes, beliefs, and trustworthiness showed negligible effects.
The paper argues that policymakers should prioritize interventions that enable individuals to overcome obstacles and facilitate behaviour change, rather than focusing on less effective determinants like knowledge and beliefs.
The findings challenge the conventional wisdom that increasing knowledge and changing attitudes are the primary keys to behaviour change, suggesting a shift towards targeting contextual factors.
The basic idea is straightforward enough: gather up all the meta-analyses you can find on behavior change interventions, classify the determinants and intervention targets, and see which ones come out on top. Simple, right? Well, not so fast.
Selection bias
The reliance on existing meta-analyses should raise concerns about selection bias and the potential for publication bias to skew the results. Meta-analyses with significant findings are more likely to be published, which can lead to an overrepresentation of positive results in the literature.
The authors acknowledge this limitation but do not adequately address how it might impact their conclusions. It's like trying to bake a cake using only the ingredients you find in your pantry - you might end up with something edible, but it's hardly a comprehensive recipe for success.
Comparability of effect sizes
The authors do not provide sufficient information about the specific contexts represented in the included meta-analyses, making it difficult to interpret the comparative effect sizes. Comparing effect sizes across diverse domains, contexts, and populations is problematic, as the included meta-analyses likely span a wide range of behaviours, populations, settings, and intervention modalities. The meaning and implications of a "small" or "large" effect can vary greatly depending on the nature of the behaviour and intervention.
It's like comparing apples to orangutans or the speed of a cheetah to that of a snail - while you can calculate an average, it tells you little about the unique characteristics of each species or allows you to make broad generalizations about the speed of all animals.
The same principle applies to behaviour change interventions. What works for promoting physical activity among elderly adults in rural communities might not translate to reducing substance abuse among urban youth. By lumping all these different interventions together, the authors have created a muddled picture that tells us little about what actually works in specific contexts.
For example, the impact of "knowledge" on behaviour change may be stronger for simple, one-time behaviours than for complex, habitual behaviours, but this distinction is lost in the overall average effect size. Similarly, it’s also odd to combine studies on behavioural skills training for smoking cessation with those for diet changes without considering their distinct mechanisms and contexts.
Limitations of the meta-meta-analysis approach
The paper's conclusions and recommendations are limited by several factors inherent to the meta-meta-analysis approach:
Confounding factors: The analysis doesn’t adequately control for potential confounders, such as intervention quality, target population characteristics, and specific behaviours addressed. The interaction between different determinants and intervention components is also not fully considered, and unmeasured confounders may influence the observed effect sizes.
Interpretation of effect sizes: The authors use arbitrary cutoffs for classifying effect sizes as negligible, small, medium, or large, which may not reflect practical significance in specific contexts – nor do they adequately consider the variability of effect sizes within each determinant or intervention category.
Overgeneralization: The authors make broad recommendations for policymakers based on a synthesis with significant limitations but they don’t sufficiently acknowledge the context-specific nature of behaviour change interventions or provide guidance on adapting the findings to different settings and populations. This one-size-fits-all approach fails to recognize that effective behaviour change interventions must be tailored to the unique characteristics and challenges of each context.
Lack of nuance and methodological rigor
The paper presents a simplistic picture of behaviour change determinants as universally more or less effective, without adequately considering the complex interplay between individual, social, and structural factors. They also fail to discuss potential unintended consequences or trade-offs of focusing on certain determinants or interventions.
Behaviour change interventions are as unique as snowflakes, each one shaped by the specific context, population, and challenges it addresses - what works in one context might melt faster than an ice cream cone in the Sahara in another.
Methodologically, the lack of a pre-registered analysis plan increases the risk of researcher degrees of freedom and bias – e.g. the authors don’t provide enough detail on their search strategy, inclusion/exclusion criteria, and data extraction process, nor do they conduct a formal quality assessment of the included meta-analyses.
This approach assumes that all meta-analyses are created equal, which is far from the truth. Differences in study design, population, and intervention fidelity can significantly impact effect sizes, yet these nuances are glossed over in the pursuit of broad generalizations.
Disconnected from established frameworks: a missed opportunity
Theoretical frameworks provide a structured approach to understanding complex interactions between determinants and behaviours - without a unifying theory, the various findings feel scattered and disconnected. As such, one of the most significant shortcomings of Albarracín et al.'s paper is its failure to engage with well-established behaviour change frameworks like the Behaviour Change Wheel (BCW) and attempts to unify the field like the Human Behaviour-Change Project (HBCP).
These frameworks are like the Rosetta Stones of behaviour change research - they provide a common language and structure for understanding the complex factors that shape human behaviour.
By mapping their findings to the BCW and HBCP, the authors could have provided valuable insights into the underlying mechanisms of behaviour change and the most effective intervention strategies, but instead, they've left us with a jumble of disconnected findings and recommendations that are difficult to interpret and apply in practice.
The BCW, with its COM-B model provides a systematic approach to understanding and targeting the factors that shape behaviour and by not mapping their findings onto this framework, the authors miss an opportunity to provide actionable guidance and to contribute to a cumulative science of behaviour change – the lack of integration makes it difficult for practitioners to translate the paper's insights into concrete strategies and intervention functions.
Similarly, the HBCP's ongoing work to develop an ontology of behaviour change interventions could have provided a structured way to classify and compare the interventions included in the meta-analyses. The ontology aims to standardize descriptions of behaviour change techniques (BCTs) and interventions, much like a shared language that allows researchers and practitioners to communicate effectively. By not engaging with this ontology, the authors limit the precision and interpretability of their findings.
For example, the term "behavioural skills training" likely encompasses multiple BCTs, such as demonstration, instruction, and feedback, each of which may have different levels of effectiveness. Without coding the interventions using a standardized taxonomy, the authors miss the opportunity to conduct more fine-grained analyses and to identify specific techniques that are most effective for targeting each determinant.
The paradox of meta-meta-analysis and the overreach of policy recommendations
The authors' meta-meta-analysis approach rests on the shaky foundation of aggregating findings from diverse meta-analyses, which themselves are already a step removed from primary studies. By further abstracting and averaging effect sizes across contexts, the paper risks losing the nuanced understanding essential for effective behaviour change interventions. It's like trying to provide personalized dating advice based on a national survey of relationship satisfaction - the generic insights might be interesting, but they're unlikely to help any specific individual navigate their unique challenges.
Despite these limitations, the authors make sweeping policy recommendations, urging decision-makers to prioritize certain determinants over others. These recommendations are based on a high-level synthesis of meta-analyses with varying quality, specificity, and relevance to different policy contexts.
The authors fail to grapple with the political, economic, and ethical dimensions of their advice or to provide concrete guidance on translating their findings into practice. It's like handing policymakers a map with no legend, no scale, and no indication of where they are - colorful and impressive-looking, but ultimately not very useful for navigating the complex terrain of behaviour change.
The elusive holy grail of behaviour change
In conclusion, while the ambitious attempt to synthesize the vast literature on behaviour change interventions is admirable, the approach falls short due to methodological limitations, lack of nuance, and disconnection from established frameworks. The meta-meta-analysis, though well-intentioned, ultimately provides a blurry bird's-eye view of the field, sacrificing critical context and detail for the sake of broad generalizations.
The road to effective behaviour change is paved with good intentions but littered with oversimplified models and one-size-fits-all solutions – and while Albarracín et al.'s paper shines a light on some important determinants, the sweeping recommendations have allowed for yet more superficial and simplistic viewpoints to flood the public discourse in behaviour change.
If we really want to advance the science and practice of behaviour change, we need to embrace complexity and context-specificity. We need to base our understanding in established theoretical frameworks, engage with initiatives such as the ontology of behaviour change interventions, and prioritise nuanced, tailored approaches over broad generalizations.
Trying to identify universal levers for change is like searching for the Holy Grail - a noble quest, but ultimately a fool's errand.
You can read the paper and the supplementary materials here:
Comments