Based on the achievements of randomised controlled trials (RCTs) in medicine, and the need for effective government interventions in support of business, some have advocated for the use of RCTs in the evaluation of business support programmes (BSPs). Notwithstanding these recommendations, the use of RCTs in the evaluation of BSPs has been resisted by (almost) all.
Policy makers and managers are correct in their reluctance to undertake RCT-based evaluations for four reasons.
While RCTs require the random allocation of support, judicious programmes select firms on the basis of potential and amenability to support.
Excellent business support, like the coaching of Olympic athletes, is about creating champions, not about training or about remedial interventions. If you want to create champions, you can’t begin with random selection.
While RCTs require treatments that exhibit low variability, the most effective BSPs draw upon substantive knowledge to provide support that is customised
Treatments need not be uniform, but there must be sufficiently low-variability in treatments for it to be clear what is, and what is not, a treatment. This rules out interventions that are highly customised and treatments that build on past interventions the effects of which may still be unfolding, both of which are common in the case of business support.
BSPs aim to produce outliers — firms whose performance is exceptional. When outliers are present, very large samples will be required to produce reliable results
Internal validity can be compromised by outlier responses to treatment, that is, subjects that respond exceptionally well to treatment (Deaton and Cartwright, 2016). This is an important caveat when contemplating the use of RCTs for the evaluation of business support programmes, because factors that are difficult to measure, such as managerial capabilities, may explain the responsiveness of firms to treatment.
Outlier participants who perform exceptionally well may be the basis upon which a positive Average Treatment Effect (ATE) for the programme is claimed if they are part of the treatment group, or the basis upon which a negative or null ATE results if they are part of the control group
Deaton and Cartwright (2016) use a simulation to show that a sample size of 1000 is required to compensate for a single positive outlier sufficiently well to have a better than 94% chance estimating the correct sign of the ATE
An RCT may not yield a meaningful contribution to knowledge. The strength of an RCT is its ability to estimate the magnitude of the treatment effect under controlled conditions
While some authors feel that RCTs are indispensable to learning and rigorous evaluation, even in fields where they difficult to conduct (Oliver et al., 2002; Walwyn and Wessely, 2005), others lament the effect that RCTs have on the ability of programme designers and managers to learn.
The standardisation of treatments that is inherent to an RCT prevents more micro level experimentation and learning, constraining the administration of programmes, and subordinating programme managers to the needs of evaluators (Perrin, 2002)
Writing on the use of RCTs in population health, Sanson-Fisher et al. (2007) observe that given the need for flexible, broad, and complex interventions, a focus on those that can be tested by RCTs may threaten the development and evaluation of innovative interventions with potentially significant public health consequences
—
The primary explanation for why there are (almost) no RCT-based evaluations of government interventions in support of business, is that the requirement for the random allocation of support makes it impossible to select companies for support based on factors that are known to increase the likelihood of successful outcomes.
To learn more, read Dr Margaret Dalziel’s full paper: “Why there are (almost) no randomised controlled trial-based evaluations and support programs” on nature.com.