Hey smarty, use Dumbbench when benchmarking Perl code

In my post about a passage in perlperf, I built all benchmarking on the same snippet of code found in that document.

brian reminded me that I should have used Dumbbench so that I can (excerpting from the excellent documentation) “compare robust estimates of the run time of meaningless benchmarks.” True. I forgot about that excellent module that incorporates sound statistical reasoning into the business of meaningless microbenchmarks.

So, I re-wrote the benchmarking script to use Dumbbench. In the process, I fixed an annoying flaw in the original examples by passing a fresh copy of the data structure to each piece of code. Dumbbench allows me to easily measure the influence of such initialization code with the dry_run_code option.

#!/usr/bin/env perl

use strict;
use warnings;

use Dumbbench;
use Storable qw( dclone );

my $hash = {
    scores => {
        myscore => 101,
        yourscore => 99,
    },
};

my $bench = Dumbbench->new(
    target_rel_precision => 0.5 / 100,
    initial_runs => 1_000,
);

my $dry_run_code = sub { sub { my $ref = shift }->(dclone $hash) };

$bench->add_instances(
    Dumbbench::Instance::PerlSub->new(
        name => 'direct',
        code => sub {
            sub {
                my $ref = shift;
                my $x = $ref->{scores}{myscore} /
                    ($ref->{scores}{yourscore} + $ref->{scores}{myscore});
                my $y = $ref->{scores}{yourscore} /
                    ($ref->{scores}{yourscore} + $ref->{scores}{myscore});
                $ref->{scores}{myscore} = $x;
                $ref->{scores}{yourscore} = $y;
                return;
            }->(dclone $hash)
        },
        dry_run_code => $dry_run_code,
    ),
    Dumbbench::Instance::PerlSub->new(
        name => 'dereference',
        code => sub {
            sub {
                my $ref = shift->{scores};
                my $myscore = $ref->{myscore};
                my $yourscore = $ref->{yourscore};
                my $denominator = $myscore + $yourscore;
                my $x = $myscore / $denominator;
                my $y = $yourscore / $denominator;
                $ref->{myscore} = $x;
                $ref->{yourscore} = $y;
                return;
            }->(dclone $hash)
        },
        dry_run_code => $dry_run_code,
    ),
    Dumbbench::Instance::PerlSub->new(
        name => 'reasonable',
        code => sub {
            sub {
                my $ref = shift->{scores};
                my $denominator = $ref->{myscore} + $ref->{yourscore};
                $ref->{myscore} /= $denominator;
                $ref->{yourscore} /= $denominator;
                return;
            }->(dclone $hash)
        },
        dry_run_code => $dry_run_code,
    ),
);

$bench->run;
$bench->report;

And, here are the results:

direct:      Rounded run time per iteration: 2.8082e-06 +/- 1.9e-09 (0.1%)
dereference: Rounded run time per iteration: 2.0905e-06 +/- 1.9e-09 (0.1%)
reasonable:  Rounded run time per iteration: 1.4179e-06 +/- 2.0e-09 (0.1%)