Understanding Statistical Significance
Statistical significance tells you whether your results are real or just random chance. This guide explains the concepts and best practices.
What is Statistical Significance?
Statistical significance measures the probability that your results aren't due to random variation. A 95% confidence level means there's only a 5% chance your results are random.
Sample Size Matters
Larger sample sizes give more reliable results. CrowAI recommends at least 100 conversions per variant before declaring a winner.
P-Values Explained
A p-value less than 0.05 indicates statistical significance at the 95% confidence level. Lower p-values mean stronger evidence.
Avoiding False Positives
Don't stop experiments too early. Running multiple tests increases the chance of false positives. Use proper statistical methods.
Practical Significance
A result can be statistically significant but not practically meaningful. Consider the actual impact on your business goals.
CrowAI's Approach
CrowAI uses Bayesian statistics to give you real-time probability estimates while protecting against common statistical errors.