Soil Science or Fiction? #2 Measuring a change is the same as measuring an impact

By:  A.J. (Ashok) Kumar, VP, Sustainability Sciences at Indigo Ag 

 Joanie Abbott, Associate Manager, Sustainability Solutions at Indigo Ag 

I’m back! Sorry it took me a bit longer between my last post and this one as I’ve been on the road talking about soil health, regen ag, and all kinds of fun things. I want to take a minute to introduce my colleague and accountability buddy, Joan Abbott, who is working with me to keep this series buzzing. For simplicity, we’ll continue to write in the first person singular, but know that Joanie is due credit here too! In my first post, I discussed how we CAN measure the impacts of regenerative agriculture on soil organic carbon at scale. 

But what does it mean to measure impacts? It’s going to vary a bit by what kind of impact you care about. In the voluntary carbon market, we’re looking to measure the impact of what was done in the project compared to what would have happened without the project. 

This leads me to our second misconception…

Misconception #2: Measuring soil carbon at two points in time is enough to generate a valid carbon credit. 

 Why? Because the difference between two points in time is a measure of change, not necessarily impact. 

This approach—measure a field now, measure it again in five years, and take the difference as the farmer’s impact—has a fatal flaw: You cannot distinguish the farm management impact from the weather.

Practices (like cover cropping or reduced tillage) influence soil carbon over time. But so does weather.

If a field sees a huge carbon increase, is that due to the farmer's new practices, or a run of unusually wet, cool years? If it drops, is that bad farming, or a severe drought?

You can’t separate those impacts when you only have two data points. This temporal accounting focus has proven to be a recipe for disaster and over-crediting in the forestry world.

We don't have to guess if this works in the agricultural soil carbon world. It doesn't.

Take the Australian carbon market, which uses a temporal approach. Research shows that this methodology has led to over-crediting (1). Why? “Namely, carbon credits are being awarded for changes associated with seasonal conditions (changes that would have happened anyway) rather than human actions (2).” Farmers weren't penalized for the negative swings either. As a result, a lot of credited carbon wasn't due to management. This drags down the integrity of the whole market.

Measuring changes over time is incredibly important for inventory accounting, but it’s not what carbon credits are trying to measure. Buyers of carbon credits want to know that their financing has helped cause the change.

What we truly care about is the Difference in Differences.

A difference in differences compares the change over time in the treated field (the farmer's new practice) against the change over time in a counterfactual scenario: what would have happened in that same year, with that same weather, on the same soil type, without the new practice. It’s impossible to measure the thing that didn’t happen, but we can measure proxies.

There are at least three ways to do that today:

• The digital twin: a modeled counterfactual - Using calibrated and validated models (that I described in my last article), we can model out the alternative scenarios to what would have happened in a field compared to what the farmer did in the project (i.e., model their field without a cover crop) and take that difference over time and subtract that from the difference in time on what did happen in the project (the difference in two scenarios difference in time). This approach is allowed under both the Climate Action Reserve's Soil Enrichment Protocol and Verra’s VM0042 methodology. This approach has been used by Indigo Ag in our CAR14592 project to date.
• The physical twin: a controlled experiment - I can’t treat the exact same soil two different ways, but I can treat soil right next to each other differently in many cases. Agricultural research tracing back hundreds of years (from Incan research to the Rothamsted trials) has moved forward by creating plots side by side where each plot gets one difference in treatment and everything else is treated the same. Here the difference in differences is between time points and the treatment and control areas. Experiments like these make up the calibration and validation dataset for the models we deploy at scale. When testing new ideas or products we rely on these kinds of experiments. They are, however, expensive and hard to run at scale (and I know this from experience!). Farmers are in the business of farming, not running controlled plot trials.
• The close cousin: a comparative baseline - If you can’t have a field subdivided and managed like a twin, you can try to look for fields that are “similar enough” apart from your intervention and look at the difference in them over time. Verra’s VM0042 allows for project developers to use control fields that are measured compared to project fields provided they meet certain similarity criteria. Striking the balance between feasibility and validity is hard. How close should the fields be? If they have to be across the fence, again that becomes quite expensive. Verra’s current guidance allows control to cover a radius of 250 km. Weather can vary quite a bit within 250 km so that likely introduces a lot of noise.
  
  

Each of these approaches has strengths and weaknesses. Where there is strong existing literature, a modeled counterfactual can give high quality answers at scale. When a practice or product is new or a cropping system is understudied, a comparative baseline approach can provide a way forward while waiting for controlled experiments to complete and get published. In any case, it’s a good idea to look for ways to test your assumptions and the validity of the difference in difference approach you are using.

The soil sample is not the truth; it's a data point. In the soil space, folks often talk about a soil sample as “ground truth”, but it depends on what you think you are measuring. Just like a single glucose reading doesn't diagnose diabetes, a single point in time measurement at a single location doesn't prove management impact (after a decade working in the health space, I can’t resist the health analogies!). Even two measurements over time don’t prove much. The measure of a soil sample itself also raises many questions (measured in situ, core extracted, soil preparation, analysis method, etc.), but perhaps I’ll save that for another day.

The point is that no measure is perfect, but imperfect measures can still be useful if put together in the right way with accounting for their strengths and weaknesses (e.g. accounting for uncertainty). We need the full context: the practice history, the validated practice change, and the difference in differences to help us make a causal link between a change and an impact.

(1) Mitchell, E., Takeda, N., Grace, L., Grace, P., Day, K., Ahmadi, S., … Rowlings, D. (2024). Making soil carbon credits work for climate change mitigation. Carbon Management, 15(1). https://doi.org/10.1080/17583004.2024.2430780

(2) Here's how to fix Australia's Approach to Soil Carbon Credits... https://findanexpert.unimelb.edu.au/news/67192-here's-how-to-fix-australia's-approach-to-soil-carbon-credits-so-they-really-count-towards-our-climate-goals