What the Heck is Statistical Significance and Why Does It Matter?

As fundraising professionals, we know how important it is to use data to support our decisions and strategies. We work hard to create and execute meaningful tests that will positively impact our program’s success. But how can we be sure that the data we collect and analyze is valid and reliable? That’s where statistical significance comes in.

Statistical significance measures the likelihood that the results of a test are real and repeatable — not just due to chance. The higher the rate of significance the more confidence you can have that your test will yield the same results in the future.

Why does this matter?

Statistical significance helps us know that our test results truly represent our donor base. Many organizations declare a test variable a winner without reaching this important threshold. Unfortunately, without statistical significance, you can’t be sure you’d receive the same results in the future, so you may roll out a tactic that actually harms your fundraising results.  

Not taking advantage of this important metric? We’ve got you covered.

The MWD Statistical Calculator is a free tool you can use to test the statistical significance of the response rate between two test panels. Use it for email testing, direct mail testing, and even donation form optimization, just remember to split your test panels randomly!

Falling short of significant results?

If you aren’t able to reach a confidence level of 90% or higher, it may mean you need to extend or repeat your test. The larger your audience, the better the chances of achieving significant results.

For this reason, some small organizations may struggle with reaching statistical significance. If so, we recommend taking real-world considerations into account alongside the results you’ve gathered:

• Staff time
• Tech requirements
• Brand Alignment
• Accessibility

Testing is key for any direct response fundraising program, and understanding your results is a pivotal part of the testing process. You should always check your results for statistical significance before declaring a winner and rolling out a new strategy. MWD’s free statistical calculator makes it easy.

Happy testing!