Climate change ‘awareness’ is no longer the problem. ‘Concern’ and ‘intent’ fix nothing. Evaluating the real-world impact of calls to action produces tiny numbers, but at least they’ll mean something.

This article outlines the design principles and taxonomy of See Through’s Action Evaluation Tool (STAET), a novel methodology which measures impact instead of intention.

The Good Intention Trap

To pave a road to Heaven, only a fool apes how the Devil builds a road to Hell.

• So why do those pursuing the nirvana of a sustainable future ignore the proverbial folly of the road to Hell being paved with good intentions?
• Why do people who claim to be serious about carbon drawdown still imagine intentions are an end in themselves?
• Why do ‘activists’ trumpet outcomes like ‘shifting public opinion’, ‘widening the Overton window’, or getting people in a focus group to use different words, when none of outcomes diminish a single molecule of greenhouse gas emissions?

These are not rhetorical questions. The answers range from wilful self-delusion to cynical corporate greenwashing. Interesting, maybe, but any fascination only illustrates the futility of focusing solely on motivation with no regard to outcome.

For an issue as urgent, existential and measurable as human-induced climate change, minutely measuring intent, rather than impact, is worse than futile. 

Intent is often the mother of action, but intent without action makes the problem worse. It fosters a culture of complacency. A world dealing in the currency of ‘awareness’, ‘concern’ and ‘intent to act’ is a world where nothing changes. Climate scientists have been telling us for half a century that Business As Usual will kill our children, if it doesn’t kill us first.

So what should anyone who lacks an appreciation of fiddle music while Rome burns, or who’s perturbed by Titanic deckchair re-arrangement, do?

The Call To Action Trap

Any activists’ answer take the form of a Call To Action (CTA).

Not all actions are the same.

This may sound obvious, but the ‘impact marketing industry’, has allowed itself to become complicit in climate complacency.

‘Impact finance’ veteran-turned pioneering impact marketing doyenne Julie Davitz should know. In Likely to Act Is Not Acting, Davitz points out her industry’s obsession with measuring intent, but not consequence.

When claiming their New Thing is an industry-busting disruptor, marketing disrupters are fond of citing Hans Christian Anderson’s The Emperor’s New Clothes. Their intent is usually to cast themselves as fearless truth tellers, but this misses two critical points:

1. The point of the fable is not just the crowd’s ignorance, but also the Emperor’s vanity and the tailors’ sycophancy.
2. The Emperor’s New Clothes is a cautionary fairy tale, not a peer-reviewed behavioural psychology research paper.

The Imperial Fashion/Corporate Greenwash Industry

Anderson was not documenting the real-world impact of the little boy laughing at the Emperor’s nakedness. His tale was not a journalistic exposé of the depth of the imaginary clothes industry’s cynicism, or the intractability of the crowd’s wilful self-delusion. The Emperor’s New Clothes is a warning against all these human follies.

The response of the impact marketing industry to Likely to Act Is Not Acting has been more George Orwell than Walt Disney.

In a bleak irony, rather than point and laugh at the naked Emperor, Davitz’s fellow impact marketing tailors didn’t respond to her indictment of their trade with cheers. They weren’t even outraged. They just ignored it.

In the real world, it turns out that the tailors are quite happy to keep on designing new non-existent creations and submitting their invoices, while the crowd keeps on applauding the naked Emperor.

This is the Call To Action trap. 

In the real world, so far, most climate CTAs are not gritty documentaries, but fairy tales. 

• Corporate greenwashers are happy to parade their nakedness.
• Tailors are happy to get paid for making more imaginary clothes.
• The rest of us prefer to go along with the farce, probably because we suspect we’re all naked too.

No wonder we’d rather measure intent than impact.

Where’s the real-world impact measuring stick?

Davitz’s article gives buttock-clenching examples of the kind of measuring sticks used by the imaginary-clothes tailors to measure what they, and their clients, claim as ‘impact’. 

She quotes in detail from research commissioned by CBS measuring the ‘impact’ of a climate-themed episode of a popular political drama, Madame Secretary.

Impact marketing professionals meticuloutly measured before-and-after responses to an episode in which a super typhoon threatens to destroy the Micronesian island nation of Nauru. The broadcaster summarises the ‘action’ as: ‘U.S. Secretary of State Elizabeth McCord and her team must figure out how to evacuate and permanently relocate the entire island’s population’. (To be clear, this is pure fiction with a climate twist. This podcast episode describes a real-world version of the US doing exactly this for Cold War reasons in the Da Chen islands in 1955).

Davitz summarizes the impact marketers’ key findings:

The study found that watching the episode moved climate threat certainty from 71 to 78. It moved worry from 65 to 73. Support for robust government action, 51 to 61. Support for compensating vulnerable nations (a genuinely hard-to-move attitude) jumped from 6 to 23. Accepting climate migrants, 29 to 43. Some of these shifts were still standing two weeks later. Participants were reportedly three times more likely to call themselves single-issue climate voters.

Davitz’s uncomfortable statement of the obvious is more courteous than Anderson’s Little Boy’s finger-pointing and hooting, but her derision of the Emperor’s nakedness is no less devastating:

The ridiculous part is a field that has now demonstrated, over and over, that content reliably moves attitudes, and then treats the next question – do moved attitudes become action? – as somebody else’s job. Or worse, as a rude question. It’s not a rude question. It’s the only question left. 

Having politely skewered the useless array of fancy tools and measuring sticks impact marketer tailors use to create their imaginary haute-couture fig leaves, Davitz asks what tools we should use instead: 

The “does content move attitudes” debate is basically settled. What’s unsettled, what’s barely been funded, barely been built, barely been attempted, is the layer that runs from a moved attitude to a tracked action.

Amazingly, there is no standard taxonomy to measure CTA impact. 

Stanford University’s Doer School of Sustainability, at the forefront of such taxonomies, metrics and tools, has not even published a prototype.

So here’s one.

See Through’s Proposed Action Evaluation Tool

The See Through network is well -placed to pioneer a tool that evaluates planetary impact rather than individual intent.

As a climate action ecosystem with no bank account that uses tonnes of greenhouse gases reduced or sequestered as its internal metric, the Network has exclusive access to some key components for this measuring tool.

Over years, the various arms of the See Through network have created a rich bin of projects, prototypes, and data to rummage through in order to construct a useful action evaluation tool prototype.

• Action/Inaction: the See Through Goal of ‘Speeding Up Carbon Drawdown by Helping the Inactive Become Active’ has action, not awareness, as its explicit target.

• Awareness-Action continuum: the four-step (Tempted, Intrigued, Engaged and Active) See Through Engagement Journey has action as its final destination. The See Through Games methodology arrives at the same destination in three steps – Think, Learn, Act. 

• Effective/Ineffective: the key diagnostic of See Through’s target audience taxonomy is effective vs. ineffective activism. A key ‘effective’ criterion is having empirical, measurable, real-world climate impact.

• Data: the diversity and scope of See Through’s dozens of projects have produced a diverse set of real-world datasets based on these methodologies. 

• Scale: 1.5M and rising Facebook group members and half a million and rising video views and podcast streams means See Through Together content has the numbers to measure impact at scale. (Spoiler, the numbers who actually end up being nudged into doing anything meaningful are pitifully small. It’s just as well the Network’s pro bono contributors aren’t trying to monetize the results. Clients would not want to pay for figures that deal in tenths of a percent – bad for business in a world which values the impressive-looking numbers Davitz quotes).

• Motivation Calibration: each strand of See Through Together’s wide range of social media content is assigned one of 4 of ‘C-Bomb’ ratings (where ‘C’ is for ‘climate’ or ‘carbon’). C-Bomb 1 means climate is explicitly mentioned ‘Never’, C-Bomb 2 is ‘Barely, C-Bomb 3 is ‘Sometimes, C-Bomb 4 is ‘Always’ (this article is a C-Bomb 4). The results test the conventional wisdom by measuring whether content drives a carbon-reducing action, not whether it shifts the audience’s awareness/intent/concern/motivation etc.

• Real numbers: See Through Together’s initial strict application of these methodologies reveals the real nature of the challenge. Considerably less than 1% of people can be persuaded to click away from the warm embrace of the social media environment to a See Through website, where they’ll find a chance to engage with a meaningful CTA. This is only depressing to those beguiled by the numbers Davitz points and laughs at. For pragmatists, it’s confronting reality. Instead of despairing, See Through’s contributors work to increase the pool of viewers,and A/B test tweaks to nudge real-world outcomes from 0.1% to 0.2%.

Proposed Solution

Who will develop this tool? 

See Through Network contributors include experts that cover the skillsets required to design, test and deliver it. If you’d like to help, you’re more than welcome to join in. 

How will it work? 

Here’s our best attempt at a technical explanation for non-technical people.

The See Through Action Evaluation Tool (STAET) is an open-source, standardised software tool to measure a CTA’s real-world impact if implemented.  

The tool is designed to balance:

• Simplicity: fill in your variables, and STAET generates a simple score that’s easy to understand at a glance.
• Complexity: the score reflects the problem’s complexity and nuance.

STAET uses a 0-100 scale where:

• 0 = inaction (regardless of any degree of ‘concern’, ‘intent’, ‘awareness’ etc. 
• 100 = the most effective possible action (the user’s settings determine what this means for any particular CTA).

This 0-100 scale is similar to Transparency International’s Corruption Perceptions Index (CPI), which ranks 181 countries by assigning a 0-100 score. 

At time of writing, the CPI ranges from #1 Denmark (89/100) to #181 South Sudan (9/100).

The benefits of a 0-100 range are:

1. Nuance: the range of scores reflects a broad spectrum of possible impacts, rather than an Effective/Ineffective binary.
2. Clarity: the absolute score gives an immediate sense of the action’s practical utility.
3. Progress/Retreat: comparing scores before/after new data indicates a trend.
4. Prioritisation: relative ranking helps allocate resources to the most effective actions.

Proposed Methodology

Transparency International’s CPI methodology is a poll-of-polls meta-survey. The final score is compiled from a dozen or so authoritative sources that themselves use a variety of methodologies to measure a commonly-understood concept (‘corruption’).

This is not a practical methodology for climate CTAs as:

a) no other action evaluation tools exist from which to derive a meta-survey.

b) ‘climate effectiveness’ is a much woollier, more ill-defined concept than ‘corruption’ (See Through Together’s Think Game data provides detailed evidence of this).

See Through is therefore considering a relatively simple statistical approach, most likely some form of weighted sum calculation. 

This is a type of multi-criteria decision analysis (MCDA). 

Users enter their preferences to match their field of endeavour, and their CTAs to meet further that endeavour.

STAET applies these settings to a range of standardised inputs or decision factors.

Users can either use See Through’s own default settings, or assign weightings they consider appropriate to generate a useful simple output.

STAET then generates a 0-100 score to evaluate that CTA.

This kind of model requires two key elements – factors and weightings.

Factors

a comprehensive, but not overwhelmingly complex, taxonomy of factors with clear, consistent metrics. Taxonomy is an inexact art, but good rules of thumb are: 

• 1-3 factors: too simple to be useful
• 10+ factors: too many variables to be practical
• 5-10 factors: a manageable range
• 7 factors: ideal

Weighting

Weighting is the relative importance of each factor for a particular purpose, as determined by the user for their particular goal. 

Initially, these can be assigned broad ranges or approximations like none, very low, low, mid, high, very high, perfect  etc..

Each range has a number value assigned. A simple example would be  low = 1, mid = 3, high = 5, but don’t have to be linear. Weightings are based on the user’s hunches/best guesses/calculations.

Broad ranges are useful in the absence of more granular data. Where appropriate and available, ranges can be replaced by real-world numbers (e.g. tonnes of greenhouse gas reduced or sequestered).

Effectiveness Factors

Factors should be a set of considerations that between them comprehensively embrace the entirety of what they describe. 

Factors are the answer to the question ‘What constitutes an effective CTA, for any cause?’

The STAET taxonomy has the following seven key factors. Each measured by a particular metric, requiring hard numbers to be entered by the user. For clarity, we’ve added an indication of what a ‘better’ number would be.

1. Target audience: number of potential respondents to a tailored CTA, e.g. number of people attending a screening, YouTube video viewers, podcast audience, or event audience. 

Metric: numbers of people.

More = better. 

2. Audience engagement: ratio of target audience to actual respondents to CTA. Similar to the standard ‘Awareness’ metric, this would be the number of people viewing a video, listening to a podcast, reading an article, attending an event etc. Metric: percentage.

Higher = better.

3. Time cost: how long the respondent invests to fulfill the CTA.

Metric: seconds/minutes/hours/days.

Longer = better. 

4. Frequency: CTA degree of commitment/replicability.

Metric: one-off/multiple-event/long-term regular commitment.

Bigger = better. 

5. Absolute Financial Cost: donation relative to recipient needs.

Metric: percentage of recipient’s fundraising target.

More = better.

6. Relative Financial Cost: donation relative to donor capacity.

Metric: percentage of donor’s annual income (for individual), profit (for business) or budget (for NGO/gov body).

More = better.

7. Emissions reduced: best estimate of the net real-world impact. For climate effectiveness.

Metric: tonnes of CO₂e reduced or sequestered.

Heavier = better.

This taxonomy is designed to apply to any CTA for any cause. 

In the case of See Through’s carbon drawdown goal, Factor 7 is the ultimate meaningful outcome. 

Once the See Through ecosystem is fully functional, tonnes of CO₂e will be the only metric used. Until then, estimating this number involves too much guesswork to be anything more than a vague aspiration, hence the need for factors 1-6. 

Weighting STAET’s seven factors should provide any activist with a more accurate, standardised, granular guide to assess a particular CTA’s impact/effectiveness before and after deployment. 

Imperfect, yes, but when available options are non-existent or guesswork, a more rigorous approach, however tentative, is better than nothing.

Weighting Factors

Unlike the hard numbers required by STAET’s seven factors, weightings are based on the user’s own judgements of what constitutes success or failure for their particular goal.

In designing evaluation tools, there are broadly two approaches to determining weighting, ‘top-down’ predicated weightings, or ‘bottom-up’ retro-engineered weightings.

Both ultimately reflect the opinions of whoever sets the weightings. 

Weightings could be based on one user’s off-the-cuff hunch, or the result of extensive deliberation between a panel of experts. 

The more care and attention the user puts into the weightings, the more ‘accurate’ and useful the outputs will be.

Predicated weightings

1. A user/expert panel hypothesises the weightings in principle.
2. They enter their weightings into STAET and apply them to specific CTAs.
3. The tool evaluates the real-world outcomes, delivers a 0-100 score.
4. The use/expert panel evaluate the scores.
5. If the scores ‘feel wrong’, users iteratively tweak the weightings until STAET generates results that match their calculations/hunches of the best way to reach the defined Goal.
6. Once STAET generates useful/convincing scores when applied to fresh inputs CTAs, these become their standard settings.

Retro-engineered weightings

1. An expert panel agrees on a set of sample CTAs and scores (the more the better).
2. This sample set of expert-agreed intuitive scores for different CTAs is used to tweak the weightings until they match their hunches.
3. Once the weightings generate useful/convincing results when applied to fresh inputs, these become their standard settings.

These two approaches are not mutually exclusive. 

Indeed, combining both may be the most efficient way to design a tool that delivers consistent, standardised, useful evaluations of new CTAs, tailored to the particular STAET user’s settings.

Example: predicated weightings

To illustrate how these approaches work, here are the initial approximate weightings See Through is using for its Goal (‘Speeding Up Carbon Drawdown by Helping the Inactive Become Active’). These will become STAET’s default settings.

These examples only quote a broad range. At time of publication, See Through is still working on the numbers it will apply to that range.

1. Target audience: neutral (i.e. self-defined by the particular CTA)
2. Audience engagement: neutral (i.e. whatever the numbers tell) 
3. Time cost: very low for seconds invested, low for minutes invested, mid for hours invested, high for days invested,  
4. Frequency: low for one-off, mid for multiple-event, high for long-term commitment.
5. Absolute Financial Cost: tiny for up to $100, low for $100-10,000, mid for $10K+ (as the See Through ecosystem has no bank account, any donations have to either be in kind with no strings attached, like this US$500K donation or delivered via a commercial Partner that has signed and published an MoU with a See Through legal entity)
6. Relative Financial Cost: very low for up to the cost of a cup of coffee, low for an hour at minimum wage, mid for a day’s wage, high for a week’s wage, very high for a month+. (a financial donation reduces no carbon directly, but the degree of commitment should be measured in relation to the donor’s resources. See Through deems a poor person donating £10 as being more meaningful to its Goal than a billionaire donating $1M on condition it delivers tax breaks).
7. Emissions reduced: best estimate of the CTA’s potential net real-world emissions impact. Low for grammes, mid for kgs, high-mid for tonnes, etc. up to very high for gigatonnes+.

Example: samples for retro-engineered weighting

These examples are based on the collective intuition of See Through’s current expert panel, as a basis for retro-engineering weighting values.

Remember:

0 = inaction (regardless of any degree of ‘concern’, ‘intent’, ‘awareness’ etc. 

• 100 = the most effective possible action (the user’s settings determine what this means for any particular CTA).

0/100 = failure to respond to any CTA.

1/100  =  respondent invests 1 second to ‘Like’ a piece of content without leaving the social media platform where they encounter it.

2/100 = respondent invests a few seconds to a few minutes to write a supportive or positive Comment without leaving the social media platform where they encounter it.

3/100 = respondent invests a few seconds to share CTA with others without leaving the social media platform where they encounter it.

4/100 = respondent invests a few seconds to share CTA with others while leaving the social media platform where they encounter it.

5/100 = respondent clicks any link that takes them away from the social media platform on which they encounter it to an activist website listing CTAs.

6/100 =  having landed on an activist website, respondent invests 10 seconds to click a non-binding online petition they’ve not looked at long enough to read properly.

7/100 = having landed on an activist website, respondent invests 3 seconds to subscribe to a newsletter.

8/100 = having landed on an activist website, respondent opens AND reads more than one edition of the newsletter they’re subscribed to. 

9/100 = having landed on an activist website, respondent invests a minute to donate a cup-of-coffee-level sum to a cause.

10/100 = having landed on an activist website, respondent invests a minute to donate a day’s earnings-level sum to a cause.

12/100 = an individual registers for See Through’s On The Record CTA

15/100 =  having landed on an activist website, respondent invests an hour for a CTA ( e.g. finding an old person, recording and sending a video for How To Live Without Plastic)

20/100 = an SME owner invests 1-3 hours to complete a CTA (e.g. filling in See Through Carbon data collection form once).

25/100 = a business or NGO entity individual registers for See Through’s On The Record CTA

30/100 = an individual responds to an ad-hoc ongoing CTA commitment (e.g. ST Counsellor)

30/100 = an individual or SME signs an MoU to be a See Through Ecosystem partner (e.g. ST Partner)

40/100 = an SME owner investing 1-2 hours to complete a See Through Carbon data collection form a year later, so we can measure any emissions reduction.

40/100 = an individual committing to one day a week developing/administering See Through projects

50/100 = an individual or SME acts on See Through Ecosystem partner MoU.

55/100 = a legislator registers for See Through’s On The Record CTA

57/100 = an individual or SME signs a Contract, becoming a See Through Ecosystem Patron 

60/100 = an individual commits to 2 days a week developing/administering See Through projects

70/100 = an individual commits to 3 days a week developing/administering See Through projects

89/100 = an individual commits to full time developing/administering See Through projects

90/100 = carbon reducing legislation is passed and enforced as a result of See Through’s On The Record CTA.

Who owns this tool’s IP?

The See Through ecosystem practices radical transparency. This means, like all its projects, STAET is open-source.

Anyone is welcome to use the IP in this article, and the software when published at www.seethroughtogether.org  for any purpose. 

STAET is designed to be adaptable to any CTA for any goal. 

The following apply to all See Through ecosystem IP, including STAET:

• Anyone can use or adapt See Through IP for their own purposes.
• Crediting or linking to us would be civil, but is not a condition.
• Using or adapting our ideas or data is not ‘stealing’. 
• Anyone intending to monetise their tool would have to convince investors/funders they could charge for a tool that the IP creators offer for free.
• Anyone intending to control their version of the IP would have to convince their clients/users they have more integrity and/or competence than the IP’s originators.
• See Through has a unique dataset and array of social media testing facilities unavailable to third parties.
• If you want to help develop this tool, whether you’re a business or an individual, here’s how to join the team