Author(s): Siddharth Mahato

Originally published on Towards AI.

Give Me a Few Minutes, and You’ll Finally Understand P-Values (No Math Required)

If you’ve ever read a scientific study, scrolled through a fitness article, or watched a news story about a “groundbreaking new drug,” you’ve certainly seen it.

A little, mysterious number at the bottom of the page: p = 0.03 or p < 0.05.

We’ve all heard that if this number is “low enough” the results are “statistically significant.” We’ve all seen headlines claiming something is “scientifically proven” or “backed by data.”

But what, exactly, does that mean?

Is it evidence? Are you sure? And why do scientists get so hung up on this one little decimal point?

I find this topic fascinating because p-values are used everywhere starting from medicine and economics to psychology and yet they’re rarely explained in a way that’s easy to understand.

So that’s exactly what we’re going to learn in this article.

The Mistake Almost Everyone Makes

Before we dive in, let me pop a bubble that 90% of people, including many college graduates, believe:

Most people think: p = 0.05 means that there is only a 5 % chance that this result is a coincidence.

That’s one of the most common misconceptions in statistics.
And don’t worry, you’re not alone. Even many scientists find it difficult to explain this correctly.

So what does it really mean?
To understand that, we have to forget statistics for a moment and leave out of the mathematical circle entirely.

The Courtroom Analogy (Your “Coin” Moment)

Forget about computers, figures and formulas. Picture yourself as a juror in a courtroom.

There is a defendant in the court. The law presumes them “Innocent Until Proven Guilty.”
This is where you start. Your starting point.

Now, here’s how this translates into statistics:
> Null Hypothesis (H₀): The defendant is NOT GUILTY (This is your default assumption, the “nothing is going on” position).
> The Alternative Hypothesis (H₁): The defendant is GUILTY. (That’s what you want to prove that something is happening).

Now, the prosecutor brings evidence with him. Let’s say a witness places the defendant near the crime scene at the exact time of the murder.

This is where the p-value enters the picture:

The p-value is the answer to this exact question:
“If the defendant is TRULY innocent (H₀), what are the chances we would still see evidence THIS extreme or even more extreme just by pure bad luck or random chance?”

Let’s break that down with two scenarios:

Scenario 1: High p-value (e.g., p = 0.4 or 40%)

Imagine the prosecutor says: “The defendant was seen walking his dog two blocks away from the scene of the crime.

If the defendant is in fact innocent there is a 40 % chance that a person who is innocent would simply be walking their dog nearby. “ That’s perfectly natural.” Coincidences do occur every day.

There is little evidence. YOU CAN’T CONVICT. You stick with “Innocent” (you fail to reject the null hypothesis).

Scenario 2: Low p-value (e.g., p = 0.01 or 1%)

Now imagine the prosecutor says: “We found the defendant’s fingerprints inside the victim’s house, his DNA on the weapon, and three eyewitnesses placing him there at the exact time of the murder.”

If the defendant is truly innocent, there is only a 1% chance that we would find ALL of this evidence entirely by random bad luck.

That seems highly unlikely, right?

The evidence is so strong that it’s safer to reject “Innocent” and say “Guilty” (you reject the null hypothesis).

So, What Does p = 0.05 Actually Mean?

In science, researchers agreed on a common threshold decades ago:
If the probability of observing these results under the “innocent” (null) assumption is less than 5% (p < 0.05), we reject the null hypothesis.

We say: “This result is statistically significant.

But here’s the BIG catch that 99% of people don’t see:

> A p-value does NOT tell you that your drug works
> A p-value does NOT mean your hypothesis is correct.
> A p-value does NOT tell you that the effect is important or meaningful.

A p-value ONLY tells you that your data is unusual if the “null” (no effect) were true.

It’s like the courtroom analogy: The p-value is the evidence against the accused. This is NOT a finding of guilt.”

It doesn’t tell us the size of the effect. It doesn’t measure importance. It is just measuring surprise.

Why Is This A Big Deal (And Why Do We Care)?

Let’s get real, with a real world example.
Imagine you are testing a new pill for weight loss.

> Your null hypothesis (H₀) is that the pill does nothing. People lose weight completely randomly.
> Your Test: You give the pill to 10,000 people. They lose an average of 0.1 pounds following 3 months.

Because your sample size is HUGE, the p-value might be coming out as p = 0.001 (extremely low!)
You publish a headline: “New pill produces statistically significant weight loss (p < 0.001)!”

But look at the reality: People only lost 0.1 pounds. That is “significant” scientifically(i.e. not some random fluke), but USELESS in practice.

A deep breath before stepping on the scale could be worth 0.1 pounds to you.

This is the reason why scientists are now warning against blind trust in p-values. It’s not the number. It’s what it means in the real world.

The Metal Detector Analogy

Think of the p-value as a metal detector in the airport.

> If the metal detector does not beep, you assume the person is safe (You fail to reject the null).
> If it beeps (p < 0.05), you take them aside for a deeper search.

But a beeping detector does not mean they’re a terrorist. Just means you have to do some more digging.

The p-value is the warning, not the verdict. It warns you to be careful – not to jump to conclusions.

Then, Why Are We Still Using It?

Because it’s the best “gatekeeper” we have.

Without a p-value, anyone could claim that their magic potion works based on pure luck. The p-value helps filter out the noise from the signal.

It’s not perfect. It’s flawed. It’s widely misunderstood. But for now, it’s the best tool we’ve got to separate random chance from something worth investigating.

One Thing To Remember

From this article, I want you to remember these points very well:

• A p-value is NOT the probability that your hypothesis is true.
• It is the probability that your data would look this way (or more extreme) if your hypothesis were FALSE and pure randomness was at play.
• p < 0.05 = “This is weird enough to pay attention to. Let’s investigate further.”
• p > 0.05 = “This looks like random noise. Move along.”

And here is the kicker:

Statistical significance (p-value) is not the same as practical significance (real-world importance).

Whenever anyone tells you that a study is “statistically proven,” you’ll never just agree without first asking the most intelligent question in the room:

That’s good, but is the effect really big enough to make a difference?

The field of statistics is still developing. Debates, controversies, and better alternatives are being worked on. However, behind all the complicated terms that are being used there is one simple idea:

At its core, a p-value is simply a way of measuring how surprising your data would be if nothing interesting were actually happening.

Continue Learning

If you found this article helpful and would like to continue learning SQL, Python, statistics and data analytics through beginner-friendly resources, you can explore more here:

🔗 https://topmate.io/siddharth_mahato

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI