Modernize the PRNG used for rand() by using a Configure option

[scottchiefbaker]

Per past discussion on p5p we know the drand48 we use as a PRNG has limitations. This PR attempts to cleanly upgrade the PRNG used for rand() calls, while leaving the existing drand48 code in place for internal Perl stuff (hash seed, etc). I found randfunc and seedfunc options in Configure that allow us to point at a new PRNG without affecting the existing code.

I created a new prng.h file which is largely self-contained except for a couple of additions to embed.fnc and Configure. I have included two new PRNG implementations: PCG64 (my preference) and xoroshiro128** as options.

Completed items

• Modern PRNG implementation (PCG64)
• Detailed instructions for future devs on how to extend and modify
• Updated unit tests
• Verify srand() functionality works as expected
• Verify the new rand() outputs the full 53 bit state capable from a double (drand48 could only do 48 bits)
  • ./perl -I lib -E 'for (1..5) { printf("%064b\n", rand() * 2**64-1); }'

TODO

• prng.h does not seem to be rebuilt consistently after changes. Do I need to add this new file to build system?
• Bikeshed on what the best PRNG is in 2026
• make regen puts the functions prototypes in a weird location "Used in locale.c and perl.c"
• Add an option to get a random integer? rand64()?

Alternate options

• We don't do anything. rand() is "good enough"
• Point users at CPAN. Random::Simple is a drop-in replacement for rand() and srand() already

[leonerd]

Are these fairly arbitrary numbers, or did they get created by some specific process? Either way, I'd like to see some sort of comment explaining why these numbers in particular.

[scottchiefbaker]

That algorithm was designed by people WAY smarter than me. Per a previous iteration of this PR I added notes in prng.h with links to various documents talking about PCG64 and how to choose a good PRNG:

// Notes:
// https://zephyrtronium.github.io/articles/randomness.html
// https://docs.oracle.com/en/java/javase/21/core/choosing-prng-algorithm.html
// https://github.com/alvoskov/SmokeRand

TLDR; PCG64 has been tested and vetted by people smarter than me.

[tonycoz]

I think it could use a reference, maybe https://en.wikipedia.org/wiki/Permuted_congruential_generator

I looked for more "original" references, but the original paper doesn't have the constants that I saw.

[Leont]

I suspect this should be U64 instead of uint64_t

[tonycoz]

I think it could be more consistent, some places use U64, some uint64_t

We do use uint64_t in a few other places.

[leonerd]

As we're now in the pre-release change freeze ahead of 5.44.0, this is no longer a suitable candidate for merge. But that means we can spend some time getting it nicely lined up ready for the 5.45.x development series.

[scottchiefbaker]

I am be interested in helping get this landed, but I lack the requisite core Perl knowledge to get it done.

I'm asking for someone more knowledgeable about the Perl internals to adopt this and get it committed. I will be happy to help test and offer implementation advice.

[tonycoz]

PR #24294 adds threading support, amongst other changes, but hasn't received any comments.

[scottchiefbaker]

I pulled in the changes from @tonycoz's branch and made a couple of code clean-ups. Overall it looks good and does what we need it to.

This branch should be mergeable whenever we open things up for v5.45.

[scottchiefbaker]

This script outputs the average number of bits of randomness seen in rand()

use strict;
use warnings;
use v5.16;
use List::Util qw( sum );

sub avg { sum(@_)/@_ }

###############################################################################

my @data;
for (1 .. 1000) {
    my $str = sprintf("%064b", int(rand() * 2**64 - 1));

    my ($zeros) = $str =~ m/(0*)$/;
    push(@data, 64 - length($zeros) + 1);
}

printf("~%0.2f bits\n", avg(@data));

Prior to this patch, I consistently get ~48 bits of randomness. After landing this we get the full 53 bits of entropy a double can hold. Getting more bits of entropy is nice, but we also get much better statistical properties from a new PRNG as well.

[scottchiefbaker]

Using Smokerand we can see that PCG64 is significantly faster than drand48.

# drand48
Nanoseconds per call:
  Raw result:                 2.44578
  For empty 'dummy' PRNG:     1.08109
  Corrected result:           1.36469
  Corrected result (GiB/sec): 2.72977

# PCG64  
Nanoseconds per call:
  Raw result:                 1.94734
  For empty 'dummy' PRNG:     1.08104
  Corrected result:           0.866292
  Corrected result (GiB/sec): 8.60054

This change should give us a boost in rand() generations per second.

[tonycoz]

Using Smokerand we can see that PCG64 is significantly faster than drand48.

I'm surprised by that - perl's implementation is just a multiplication and an add on 64-bit systems, while PCG64 is 2 multiplies and several other ops.

Did you test perl's drand48() or the system drand48()?

[scottchiefbaker]

The speed comparison results are direct from Smokerand comparing the two algorithms. I was surprised also.

smokerand speed bin/generators/drand48.so

vs

smokerand speed bin/generators/pcg64_64.so

[tonycoz]

perlhack disallows C++ (//) style comments (I don't know why)