When you’re looking at research data, make sure you know what you’re comparing, and which new products will make a real difference on your farm
After 36 years of writing columns you might expect that I’ve dealt with some topics more than once. This is one of them. It is all about interpreting research trials and the deluge of yield data that is used to convince a farmer to use a variety or product.
It is all about natural variation in soils and crops. When we compare A to B, how do we decide how large a yield difference is significant?
There is the first big problem. When research types compare varieties or products they use statistical analysis to determine if the difference is “significant.” But, that is a bad term.
The yield difference can be due to the products we are comparing or it might be due to natural variation. To deal with this problem, research plots replicate the comparison a number of times. Then statistical analysis is done to separate natural variation from that due to the product, variety, etc. If the difference can be proven to be due to the variety/product it is said to be significant.
Instead of saying the difference is “significant,” the term we should be using is “real.”
To my farm, “significant” means that I will feel the difference in my wallet. With canola at $14 a bushel, five bushels per acre would be significant to most of us. There are very few experiments that can measure a difference of less than five bu./ac.
Least significant difference
Another term used is LSD — not the stuff you used to get high as a teenager — it means “least significant difference.” In other words, if the difference between the lowest and highest yield in an experiment is less than the LSD it is most likely (usually 95 per cent probability) due to random chance.
The 2012 Canola Performance Trials report LSD values of 5.4 to 7.8 bu./ac. for the average small plot results. For the small plot results by location the LSD varies from three to 12.3 bu./ac.
Let’s do a simple — albeit extreme — example with a high LSD value:
Let’s first look at the gross revenue column of the table. For variety D, with 50 bu./ac. at $14 per bushel, the gross revenue is $700 per acre. For variety A, it’s $574 per acre. I make that difference to be $126/ac. On 2,000 acres, that is a quarter of a million dollars. Pretty significant to this old fossil.
However, the average yield is 45 bu./ac., and if the statistics are to be believed that is what should be reported for all the varieties. The LSD for this example, shown at the bottom of the table, is 10. The difference between the highest and lowest is only nine, so there is no statistical, or “real” difference among any of the varieties.
When sales folks present us with research data to convince us to use a product or variety we must look very carefully at the data. Be careful how you use research data. Four is not always bigger than three.
Please, please, please do not misinterpret what I am saying about the Canola Trials. That is a very good program that combines small plot and field scale trials and has rigorous inspection with rejection of field scale data that does not meet specs. And they report the LSD and give an example of how to use it. I only chose the Canola Variety data for this example because it is widely reported and widely used.
But, there are many times when data is presented without the rigorous protocol of the canola trials.
How many times have you heard a presentation that included: “We measured a difference of three bushels an acre between A and B, but the difference was not ‘significant.” I have done it many times myself. We should throw out the term “significant” and replace it with “real.”
A common research plot design is a experiment replicated four times with each treatment randomly placed in each of the four replications. Seldom does a experiment replicated four times find a “real” difference of anything less than five bu./ac.
So, be careful when interpreting the data to justify your expense for an input. †