A note on stats and probability

One thing led to another, and I ended up re-reading an answer I had given on Math Stackexchange a few years ago. The question asked “How to determine if coin comes up heads more often than tails?”

Others gave a bunch of standard answers, but I do like my answer:

Others have provided fine answers for the standard method of calculating the probability of getting k or more heads in n tosses.

However, I think you are approaching the question from the wrong direction. Few people have tossed a single coin enough times. Pearson did it around 1900: He tossed a coin 24,000 times and came up with 12,012 heads. Count Buffon tossed one 4040 times and got 2048 heads and Kerrich did it 10,000 times in a German POW camp and got 5067 heads. (Moore et al. “Introduction to the Practice of Statistics”, 6th ed, p.240).

If you want to make sure a given coin is fair, you should use physical measurements and analyze the surface and composition of the coin to make sure its weight is uniformly distributed, it’s not bent etc.

If someone offers you a bet where you pay 10 dollars if the coin comes up heads, and he pays 10,000 dollars if it comes up tails, you can be assured that the coin is not fair ;-)

Otherwise, we assume a generic coin picked without any special procedure is fair. Without that assumption, life as we know it would not be possible.

For example, casinos are not required to roll their dice thousands of times to ascertain their fairness. In fact, doing so may wear out the corners or edges. Rather, there are specific manufacturing process requirements that need to be met.

Similarly, no one wastes time trying to ensure that the coin used to determine who gets the ball at the start of a football game is fair. NFL’s rules on the coin flip are simple and they do not involve ascertaining the fairness of the coin.

Probability will tell you that if 1,000 people each toss their fair coins 30 times, most of the percentages will be very close to 50%. Determining whether an individual coin is fair is not a task for Statistics.

See also Dynamical Bias in Coin Tossing and Chance News 11.02 referenced in that article.

That’s a nice, intuitive explanation of the literal question “My question is this: How “off” should the result be for it to be probable that the coin is not fair?” Basically, if you want to determine if a given coin is fair, examine the coin rather than tossing it n times.

You can simulate taking M samples of N coin tosses using the following Perl script. Invoke it as:

$ ./coin-toss.pl sample_size number_of_samples

By default, each sample contains the results of 30 tosses and we take 1,000 samples. That approximates the sampling distribution for sample size 30:

#!/usr/bin/env perl

use strict;
use warnings;

use Const::Fast;
use Statistics::Histogram qw( get_histogram );

run(\@ARGV);

sub run {
    my $argv = shift;
    const my $sample_size => ($argv->[0] // 30);
    const my $number_of_samples => ($argv->[1] // 1_000);

    my @sample_heads_props = map sample_heads($sample_size),
                             1 .. $number_of_samples;
    print get_histogram(\@sample_heads_props, 10);
}

sub sample_heads {
    my $sample_size = shift;
    my $count_heads;

    $count_heads += (0.5 >= rand) for 1 .. $sample_size;
    return $count_heads/$sample_size;
}

Example (30 samples of size 1,000):

$ ./prob.pl 1000 30
Count: 30
Range:  0.454 -  0.534; Mean:  0.500; Median:  0.502; Stddev:  0.018
Percentiles:  90th:  0.523; 95th:  0.527; 99th:  0.534
   0.454 -    0.462:     1 #######
   0.462 -    0.478:     1 #######
   0.478 -    0.485:     6 ########################################
   0.485 -    0.493:     1 #######
   0.493 -    0.501:     4 ###########################
   0.501 -    0.510:     8 #####################################################
   0.510 -    0.518:     4 ###########################
   0.518 -    0.526:     3 ####################
   0.526 -    0.534:     2 #############

Example (1,000 samples of size 10)

$ ./prob.pl 10 1000
Count: 1000
Range:  0.000 -  1.000; Mean:  0.511; Median:  0.500; Stddev:  0.161
Percentiles:  90th:  0.700; 95th:  0.800; 99th:  0.900
   0.000 -    0.168:     6 #
   0.168 -    0.240:    43 ##########
   0.240 -    0.316:   111 ##########################
   0.316 -    0.483:   200 ###############################################
   0.483 -    0.575:   226 #####################################################
   0.575 -    0.672:   222 ####################################################
   0.672 -    0.775:   121 ############################
   0.775 -    0.884:    57 #############
   0.884 -    1.000:    13 ###