Entropy analysis

We firstly check that entropy is actually being generated by Mata Hari’s entropy source. We are trying to confirm that irreducible information is being generated in a near homogeneous manner (ergodicity), rather than some initial chaotic start up event. The complexity test below shows a good uniformity of entropy production, with all ten normalised segments having a value of around 1.

Essentially the device works due to quantization error when sampling a random (uncorrelated) source at a 10 bit resolution. And the beauty is that we can form an additive noise model for the circuit whereby the Zener generated noise rides atop any ambient noise like radio, EMI or anything induced from the mains supply.

Kolmogorov complexity test for entropy creation within REALLYREALLYRANDOM's Type 3 (Mata Hari) raw entropy source.

Kolmogorov complexity test for entropy creation.

Raw signal and histogram of REALLYREALLYRANDOM's Type 3 (Mata Hari) entropy source.

Raw signal and histogram.

The above probability density function shows the range of quantized signal values via the internal ADC sampling at the default nominal rate of 10 kSa/s $(\epsilon = 1.1 \text{mV}, \tau = 112 \: \mu \text{s})$ or 8.93 kSa/s actual. We have fitted a curve to it, and it appears to be slightly skewed which we infer to be a log-normal distribution. This is proof that some Zener avalanche effect occurs within our 8.2 V diode with slightly over 0.8 V compliance (given a healthy battery).

But this signal can’t be used as is. ent gives a lag = 1 autocorrelation ($R$) value of 0.049733. Typically $R \le 10^{-3}$ if the data set is to be considered IID. IIDness dramatically increases our confidence in min.entropy ($H_{\infty}$) determination and bypasses all those dodgy NIST 90B non-IID shenanigans and hoo-ha detailed here. We will not even bother to perform an IID test on this raw data; it would fail.

So in keeping with our Three Golden rules, we will modify our $(\epsilon, \tau)$ sampling methodology and reduce $N_{\epsilon}$ to only one bit, equivalent to a one bit 1.1 mV digitiser. We will sample as (analogRead(portNo) & 0b1). Hopefully this will result in IID samples. Thus testing…

$ java -jar "iid-tester.jar"  /tmp/mata-hari-1mb-x1bit.bin   /tmp/results.json
Starting IID test...
Baseline: 587677
Starting a compressor thread...
Starting a compressor thread...
Starting a compressor thread...
Starting a compressor thread...
Starting a compressor thread...
Compressor Thread-0 started.
Compressor Thread-3 started.
Compressor Thread-1 started.
Compressor Thread-4 started.
Compressor Thread-2 started.
Compressor Thread-1 Permutation 1, Normalised statistic = 0.9969881414450454
Compressor Thread-4 Permutation 1, Normalised statistic = 0.9976670858311624
Compressor Thread-3 Permutation 1, Normalised statistic = 0.9961169145636123
Compressor Thread-0 Permutation 1, Normalised statistic = 0.9970749238101883
Compressor Thread-2 Permutation 1, Normalised statistic = 0.997983586221683
Compressor Thread-1 Permutation 2, Normalised statistic = 0.9959790837483856
Compressor Thread-4 Permutation 2, Normalised statistic = 0.9961917856237354
Compressor Thread-3 Permutation 2, Normalised statistic = 1.0002178067203582
Compressor Thread-0 Permutation 2, Normalised statistic = 0.9985417159426011
Compressor Thread-2 Permutation 2, Normalised statistic = 1.0003930730656465
Compressor Thread-1 Permutation 3, Normalised statistic = 0.9989194744732225
Compressor Thread-4 Permutation 3, Normalised statistic = 1.0013595903872365
Compressor Thread-2 Permutation 3, Normalised statistic = 0.9996205398543758
Compressor Thread-3 Permutation 3, Normalised statistic = 0.9974816097958572
Compressor Thread-0 Permutation 3, Normalised statistic = 0.9992853216988243
CURRENT RANK = 12
Remaining running time to full completion = 6672s
Estimated full completion date/time = 2020-12-17T22:45:25.671464
Compressor Thread-1 Permutation 4, Normalised statistic = 0.9990692165934688
Compressor Thread-4 Permutation 4, Normalised statistic = 1.0011111545968279
Compressor Thread-3 Permutation 4, Normalised statistic = 1.000049346835081
Compressor Thread-2 Permutation 4, Normalised statistic = 0.9974084403507368
Compressor Thread-0 Permutation 4, Normalised statistic = 0.9990879343584996
Compressor Thread-1 Permutation 5, Normalised statistic = 0.9980805782768426
Compressor Thread-4 Permutation 5, Normalised statistic = 0.9980805782768426
Compressor Thread-2 Permutation 5, Normalised statistic = 0.9977504649662995
Compressor Thread-0 Permutation 5, Normalised statistic = 1.0022052930436276
Compressor Thread-3 Permutation 5, Normalised statistic = 0.9962224146937859
Compressor Thread-1 Permutation 6, Normalised statistic = 0.9975020291758908
Compressor Thread-4 Permutation 6, Normalised statistic = 0.9986114821577159
Compressor Thread-0 Permutation 6, Normalised statistic = 0.9963211083639483
Compressor Thread-2 Permutation 6, Normalised statistic = 0.9986114821577159
Compressor Thread-3 Permutation 6, Normalised statistic = 0.9986182886177271
CURRENT RANK = 24
Remaining running time to full completion = 6662s
Estimated full completion date/time = 2020-12-17T22:45:25.671464
Compressor Thread-1 Permutation 7, Normalised statistic = 1.0004900651208062
Compressor Thread-4 Permutation 7, Normalised statistic = 0.9983375221422652
Compressor Thread-0 Permutation 7, Normalised statistic = 0.9989449986982645
Compressor Thread-3 Permutation 7, Normalised statistic = 0.9977572714263108
Compressor Thread-2 Permutation 7, Normalised statistic = 1.0006142830160105
Compressor Thread-1 Permutation 8, Normalised statistic = 0.9979937959116998
Compressor Thread-4 Permutation 8, Normalised statistic = 1.0005615329509236
Compressor Thread-2 Permutation 8, Normalised statistic = 0.9991253698885613
Compressor Thread-3 Permutation 8, Normalised statistic = 0.9980040056017165
Compressor Thread-0 Permutation 8, Normalised statistic = 0.997034085050121
Compressor Thread-1 Permutation 9, Normalised statistic = 0.9976687874461652
Compressor Thread-4 Permutation 9, Normalised statistic = 0.9992444829387572
Compressor Thread-3 Permutation 9, Normalised statistic = 1.0006278959360329
Compressor Thread-0 Permutation 9, Normalised statistic = 0.9974782065658516
Compressor Thread-2 Permutation 9, Normalised statistic = 0.9981282234969209
CURRENT RANK = 35
Remaining running time to full completion = 6652s
Estimated full completion date/time = 2020-12-17T22:45:25.671464
Compressor Thread-1 Permutation 10, Normalised statistic = 0.9990232729883933
Compressor Thread-4 Permutation 10, Normalised statistic = 1.0002280164103752
Compressor Thread-3 Permutation 10, Normalised statistic = 0.9977045213612239
Compressor Thread-0 Permutation 10, Normalised statistic = 0.9968809396998691
Compressor Thread-2 Permutation 10, Normalised statistic = 0.9977521665813023
Compressor Thread-1 Permutation 11, Normalised statistic = 0.9981962880970329
Compressor Thread-3 Permutation 11, Normalised statistic = 0.9967686331096844
Compressor Thread-4 Permutation 11, Normalised statistic = 0.997661980986154
Compressor Thread-2 Permutation 11, Normalised statistic = 0.9971566013303226
Compressor Thread-0 Permutation 11, Normalised statistic = 1.0008491058863969
Compressor Thread-1 Permutation 12, Normalised statistic = 0.9983307156822541
Compressor Thread-4 Permutation 12, Normalised statistic = 0.9963245115939539
Compressor Thread-2 Permutation 12, Normalised statistic = 0.9982626510821421
Compressor Thread-0 Permutation 12, Normalised statistic = 1.0003062907005038
Compressor Thread-3 Permutation 12, Normalised statistic = 0.9999931935399888
CURRENT RANK = 47
Remaining running time to full completion = 6642s
Estimated full completion date/time = 2020-12-17T22:45:25.671464
Compressor Thread-1 Permutation 13, Normalised statistic = 1.0001242178952043
Compressor Thread-4 Permutation 13, Normalised statistic = 0.9972944321455494
Compressor Thread-0 Permutation 13, Normalised statistic = 0.9981401348019405
Compressor Thread-2 Permutation 13, Normalised statistic = 0.9982609494671393
Compressor Thread-3 Permutation 13, Normalised statistic = 0.9991730151086397
Compressor Thread-1 Permutation 14, Normalised statistic = 0.9974458758807985
Compressor Thread-4 Permutation 14, Normalised statistic = 0.9974016338907257
Compressor Thread-2 Permutation 14, Normalised statistic = 0.9977232391262547
Compressor Thread-0 Permutation 14, Normalised statistic = 0.998043142746781
Compressor Thread-3 Permutation 14, Normalised statistic = 0.9977981101863779
Compressor Thread-1 Permutation 15, Normalised statistic = 0.9972535933854821
Compressor Thread-4 Permutation 15, Normalised statistic = 0.9967363024246312
Compressor Thread-3 Permutation 15, Normalised statistic = 0.9972195610854262
Compressor Thread-0 Permutation 15, Normalised statistic = 0.9966188909894381
Compressor Thread-2 Permutation 15, Normalised statistic = 0.9959790837483856
CURRENT RANK = 61
Remaining running time to full completion = 6632s
Estimated full completion date/time = 2020-12-17T22:45:25.671464
Compressor Thread-1 Permutation 16, Normalised statistic = 1.0022733576437397
Compressor Thread-4 Permutation 16, Normalised statistic = 1.0002399277153946
Compressor Thread-2 Permutation 16, Normalised statistic = 0.997076625425191
Compressor Thread-0 Permutation 16, Normalised statistic = 0.9970511012001491
Compressor Thread-3 Permutation 16, Normalised statistic = 0.9990096600683709
Compressor Thread-1 Permutation 17, Normalised statistic = 1.0006976621511476
Compressor Thread-4 Permutation 17, Normalised statistic = 0.9983919738223548
Compressor Thread-3 Permutation 17, Normalised statistic = 0.9984123932023884
Compressor Thread-2 Permutation 17, Normalised statistic = 0.998000602371711
Compressor Thread-0 Permutation 17, Normalised statistic = 0.9982881753071841
Compressor Thread-1 Permutation 18, Normalised statistic = 1.000695960536145
Compressor Thread-0 Permutation 18, Normalised statistic = 0.9975020291758908
Compressor Thread-4 Permutation 18, Normalised statistic = 0.9983511350622876
Compressor Thread-3 Permutation 18, Normalised statistic = 0.9994878138841574
Compressor Thread-2 Permutation 18, Normalised statistic = 0.997289327300541
CURRENT RANK = 72
Remaining running time to full completion = 6622s
Estimated full completion date/time = 2020-12-17T22:45:25.671464
Compressor Thread-0 Permutation 19, Normalised statistic = 0.9969081655399139
Compressor Thread-1 Permutation 19, Normalised statistic = 1.0016233407126705
Compressor Thread-3 Permutation 19, Normalised statistic = 0.9981707638719909
Compressor Thread-2 Permutation 19, Normalised statistic = 0.9990692165934688
Compressor Thread-4 Permutation 19, Normalised statistic = 1.0004968715808173
Compressor Thread-1 Permutation 20, Normalised statistic = 0.9968860445448775
Compressor Thread-0 Permutation 20, Normalised statistic = 1.0001072017451764
Compressor Thread-2 Permutation 20, Normalised statistic = 0.9970902383452135
Compressor Thread-4 Permutation 20, Normalised statistic = 0.9974288597307704
Compressor Thread-3 Permutation 20, Normalised statistic = 0.9962105033887663
Compressor Thread-1 Permutation 21, Normalised statistic = 1.000051048450084
Compressor Thread-0 Permutation 21, Normalised statistic = 0.9965950683793989
Compressor Thread-4 Permutation 21, Normalised statistic = 0.999084531128494
Compressor Thread-3 Permutation 21, Normalised statistic = 0.9988446034130993
Compressor Thread-2 Permutation 21, Normalised statistic = 0.9962071001587607
Compressor Thread-1 Permutation 22, Normalised statistic = 0.9962530437638363
CURRENT RANK = 84
Remaining running time to full completion = 6549s
Estimated full completion date/time = 2020-12-17T22:44:22.671464
Compressor Thread-2 Permutation 22, Normalised statistic = 1.0000816775201344
Compressor Thread-3 Permutation 22, Normalised statistic = 0.9969370929949615
Compressor Thread-4 Permutation 22, Normalised statistic = 0.9975905131560364
Compressor Thread-0 Permutation 22, Normalised statistic = 0.9997532658245941
Compressor Thread-1 Permutation 23, Normalised statistic = 0.9969643188350064
Compressor Thread-2 Permutation 23, Normalised statistic = 0.9986455144577718
Compressor Thread-3 Permutation 23, Normalised statistic = 0.9968366977097963
Compressor Thread-0 Permutation 23, Normalised statistic = 0.9978763844765066
Compressor Thread-4 Permutation 23, Normalised statistic = 0.9999795806199664
Compressor Thread-1 Permutation 24, Normalised statistic = 1.0008967511064752
Compressor Thread-2 Permutation 24, Normalised statistic = 0.9975360614759469
Compressor Thread-3 Permutation 24, Normalised statistic = 1.0000187177650308
Compressor Thread-0 Permutation 24, Normalised statistic = 1.0005615329509236
Compressor Thread-4 Permutation 24, Normalised statistic = 0.9989449986982645
Compressor Thread-1 Permutation 25, Normalised statistic = 0.997804916646389
Compressor Thread-2 Permutation 25, Normalised statistic = 1.0003318149255458
Compressor Thread-3 Permutation 25, Normalised statistic = 0.9998162255796977
Compressor Thread-4 Permutation 25, Normalised statistic = 0.9965236005492814
Compressor Thread-0 Permutation 25, Normalised statistic = 0.9995303542592274
CURRENT RANK = 98
Remaining running time to full completion = 6335s
Estimated full completion date/time = 2020-12-17T22:40:58.671464
Compressor Thread-1 Permutation 26, Normalised statistic = 0.9993601927589475
Compressor Thread-2 Permutation 26, Normalised statistic = 1.0007640251362568
Compressor Thread-0 Permutation 26, Normalised statistic = 0.9972757143805185
Compressor Thread-3 Permutation 26, Normalised statistic = 1.0031241651451392
Compressor Thread-4 Permutation 26, Normalised statistic = 0.999212152253704
Compressor Thread-1 Permutation 27, Normalised statistic = 0.9972348756204513
Compressor Thread-2 Permutation 27, Normalised statistic = 0.9969898430600483
Compressor Thread-3 Permutation 27, Normalised statistic = 0.998017618521739
Compressor Thread-0 Permutation 27, Normalised statistic = 1.001439566292368
Compressor Thread-4 Permutation 27, Normalised statistic = 0.9977589730413136
Compressor Thread-1 Permutation 28, Normalised statistic = 0.9980023039867137
Compressor Thread-2 Permutation 28, Normalised statistic = 0.9984549335774584
Compressor Thread-3 Permutation 28, Normalised statistic = 0.9999387418598993
Compressor Thread-0 Permutation 28, Normalised statistic = 0.9976534729111399
Compressor Thread-4 Permutation 28, Normalised statistic = 0.9989926439183429
CURRENT RANK = 110
Remaining running time to full completion = 6353s
Estimated full completion date/time = 2020-12-17T22:41:26.671464

----------------------------------
Ranked as > 110/10,000
*** PASSED permutation test. There is no evidence that the data is not IID ***
Based on 140 permutations.
Test results in file: /tmp/results.json

----------------------------------
Compressor Thread-1 Permutation 29, Normalised statistic = 0.9974645936458293
Compressor Thread-0 Permutation 29, Normalised statistic = 1.0009512027865648
Compressor Thread-3 Permutation 29, Normalised statistic = 0.9989943455333457
Compressor Thread-4 Permutation 29, Normalised statistic = 0.9996545721544318
Compressor Thread-2 Permutation 29, Normalised statistic = 0.9969915446750511
$ ea_iid  -v  /tmp/mata-hari-1mb-x1bit.bin
Opening file: '/tmp/mata-hari-1mb-x1bit.bin'
Loaded 1000000 samples of 2 distinct 1-bit-wide symbols
Calculating baseline statistics...
	Raw Mean: 0.496745
	Median: 0.500000
	Binary: true

Literal MCV Estimate: mode = 503255, p-hat = 0.50325500000000001, p_u = 0.50454288800445357
H_original: 0.986951                <<<<< Hmin. 98.7% purity.

Chi square independence
	score = 1966.613953
	degrees of freedom = 2046
	p-value = 0.893798

Chi square goodness of fit
	score = 3.380723
	degrees of freedom = 9
	p-value = 0.947272

** Passed chi square tests

LiteralLongest Repeated Substring results
	P_col: 0.500021
	Length of LRS: 41
	Pr(X >= 1): 0.203677
** Passed length of longest repeated substring test

Beginning initial tests...

Initial test results
              excursion: 296.514
     numDirectionalRuns: 79871
     lenDirectionalRuns: 12
  numIncreasesDecreases: 74699
          numRunsMedian: 500931
          lenRunsMedian: 25
           avgCollision: 19.7375
           maxCollision: 73
         periodicity(1): 24444
         periodicity(2): 24449
         periodicity(8): 24569
        periodicity(16): 24791
        periodicity(32): 24309
          covariance(1): 1.9731e+06
          covariance(2): 1.97385e+06
          covariance(8): 1.97307e+06
         covariance(16): 1.97408e+06
         covariance(32): 1.97312e+06
            compression: 155979


                statistic  C[i][0]  C[i][1]  C[i][2]
----------------------------------------------------
                excursion      55       0       6
       numDirectionalRuns       6       0      36
       lenDirectionalRuns       2       4      13
    numIncreasesDecreases      42       0       6
            numRunsMedian       6       0      85
            lenRunsMedian       5       1     169
             avgCollision       6       0       8
             maxCollision       3       3      91
           periodicity(1)      17       0       6
           periodicity(2)      19       0       6
           periodicity(8)       6       0       9
          periodicity(16)       6       0      71
          periodicity(32)      72       0       6
            covariance(1)      43       0       6
            covariance(2)       8       0       6
            covariance(8)      47       0       6
           covariance(16)       6       0      16
           covariance(32)      11       0       6
              compression       6       0     816
(* denotes failed test)

** Passed IID permutation tests
Slow IID test of REALLYREALLYRANDOM's Type 3 (Mata Hari) 1 bit entropy source.

Slow IID test for sampling at 1 bit/sample.

These two tests confirm that our sampling regime is IID. Therefore we can compress without worry the individual bits into a byte to increase the efficiency of transferring the entropy off-circuit. We can do this utilising a left shift and some ORs. Incidentally, the extra processing has the effect of ever so slightly decreasing the sample rate ($\tau \uparrow $) and thus enhancing IIDness, but we couldn’t exactly quantify this small difference ourselves $(\tau \approx 112 \: \mu \text{s})$. The internal Mata Hari sample loop is like so:-

  for (int16_t i = 0; i < totalSamples; i++) {
    uint8_t sample = 0;
    sample |= (analogRead(portNo) & 0b1);
    sample <<= 1;
    sample |= (analogRead(portNo) & 0b1);
    sample <<= 1;
    sample |= (analogRead(portNo) & 0b1);
    sample <<= 1;
    sample |= (analogRead(portNo) & 0b1);
    sample <<= 1;
    sample |= (analogRead(portNo) & 0b1);
    sample <<= 1;
    sample |= (analogRead(portNo) & 0b1);
    sample <<= 1;
    sample |= (analogRead(portNo) & 0b1);
    sample <<= 1;
    sample |= (analogRead(portNo) & 0b1);
    buffer[i] = sample;
  }

buffer is then off loaded via a binary Serial.write(buffer, totalSamples) statement. Some firmware is in ‘Related files’. Testing the resulting 8 bit binary output, we now get:-

Post processed signal and histogram from REALLYREALLYRANDOM's Type 3 (Mata Hari) entropy source.

Post processed signal and histogram.

Zoomed histogram showing bias towards zero from REALLYREALLYRANDOM's Type 3 (Mata Hari) entropy source.

Zoomed histogram showing a $2^{-9.5}$ bias towards zero.

We find from ent over the 8 bits/byte compressed output:-

$ ent -b  /tmp/mata-hari-10mb-x8bit.bin
Entropy = 0.999995 bits per bit.

Optimum compression would reduce the size
of this 80000000 bit file by 0 percent.

Chi square distribution for 80000000 samples is 587.11, and randomly
would exceed this value less than 0.01 percent of the times.

Arithmetic mean value of data bits is 0.4986 (0.5 = random).                    <<<<
Monte Carlo value for Pi is 3.151026060 (error 0.30 percent).
Serial correlation coefficient is -0.000459 (totally uncorrelated = 0.0).       <<<<

We can ignore all the metrics other than arithmetic mean and serial correlation. An arithmetic mean of 0.4968 suggests the expected bit bias of $\epsilon = 0.0014$, or $ \epsilon = 2^{-9.5}$ away from $0.5$ towards $0$, calculated as $ 9.5 = \frac{\log \big (\frac{1}{0.5-0.4986} \big)}{\log(2)} $. Not exactly NIST’s $\epsilon \le 2^{-64}$ next bit requirement, but then this is not a TRNG. It’s the entropy source for a TRNG. Randomness extraction follows which reduces the bias to a negligible degree. And notice the correlation. $|R| = 0.000459$ is very good indeed being an order of magnitude below what’s expected of IID data. Which we further confirm by three independent means…

$ java -jar "/mnt/bazinga/RRR/software/iid-tester/dist/iid-tester.jar"  /tmp/mata-hari-10mbx8.bin   /tmp/results.json
Starting IID test...
Baseline: 40184672
Starting a compressor thread...
Starting a compressor thread...
Starting a compressor thread...
Starting a compressor thread...
Starting a compressor thread...
Compressor Thread-1 started.
Compressor Thread-2 started.
Compressor Thread-4 started.
Compressor Thread-3 started.
Compressor Thread-0 started.
Compressor Thread-2 Permutation 1, Normalised statistic = 0.9999547837543629
Compressor Thread-4 Permutation 1, Normalised statistic = 0.999958093474049
Compressor Thread-0 Permutation 1, Normalised statistic = 0.9999588897975825
Compressor Thread-3 Permutation 1, Normalised statistic = 0.9999325862358662
Compressor Thread-1 Permutation 1, Normalised statistic = 0.9999399771136617
CURRENT RANK = 5
Compressor Thread-1 Permutation 2, Normalised statistic = 0.9999573220356259
Compressor Thread-0 Permutation 2, Normalised statistic = 0.9999440831568813
Compressor Thread-3 Permutation 2, Normalised statistic = 0.9999647626836422
Compressor Thread-2 Permutation 2, Normalised statistic = 0.9999846956570904
Compressor Thread-4 Permutation 2, Normalised statistic = 0.9999609303766371
CURRENT RANK = 10
Compressor Thread-1 Permutation 3, Normalised statistic = 0.9999429135566915
Compressor Thread-2 Permutation 3, Normalised statistic = 0.99996640510093
Compressor Thread-0 Permutation 3, Normalised statistic = 0.9999727010338668
Compressor Thread-4 Permutation 3, Normalised statistic = 0.9999217363277222
Compressor Thread-3 Permutation 3, Normalised statistic = 0.9999424158544831
CURRENT RANK = 15
Remaining running time to full completion = 20019s
Estimated full completion date/time = 2020-12-19T19:30:50.201137
Compressor Thread-0 Permutation 4, Normalised statistic = 0.9999443320079855
Compressor Thread-1 Permutation 4, Normalised statistic = 0.9999466214381444
Compressor Thread-4 Permutation 4, Normalised statistic = 0.9999415199905078
Compressor Thread-2 Permutation 4, Normalised statistic = 1.0000117457721194
Compressor Thread-3 Permutation 4, Normalised statistic = 0.9999769563877491
CURRENT RANK = 19
Remaining running time to full completion = 20009s
Estimated full completion date/time = 2020-12-19T19:30:50.201137
Compressor Thread-1 Permutation 5, Normalised statistic = 0.9999544104777065
Compressor Thread-0 Permutation 5, Normalised statistic = 0.9999307945079159
Compressor Thread-4 Permutation 5, Normalised statistic = 0.9999357466448898
Compressor Thread-3 Permutation 5Compressor Thread-2 Permutation 5, Normalised statistic = 0.9999695903950641
, Normalised statistic = 0.9999326857763079
CURRENT RANK = 24
Remaining running time to full completion = 19999s
Estimated full completion date/time = 2020-12-19T19:30:50.201137
Compressor Thread-1 Permutation 6, Normalised statistic = 0.999990394347377
Compressor Thread-2 Permutation 6, Normalised statistic = 0.9999869353170283
Compressor Thread-3 Permutation 6, Normalised statistic = 0.9999640161303295
Compressor Thread-0 Permutation 6, Normalised statistic = 0.9999667783775864
Compressor Thread-4 Permutation 6, Normalised statistic = 0.999937936534607
Compressor Thread-1 Permutation 7, Normalised statistic = 0.9999651359602985
Compressor Thread-0 Permutation 7, Normalised statistic = 0.9999865371552615
Compressor Thread-3 Permutation 7, Normalised statistic = 0.9999821076056065
Compressor Thread-2 Permutation 7, Normalised statistic = 0.9999465716679236
Compressor Thread-4 Permutation 7, Normalised statistic = 0.999987507674568
CURRENT RANK = 34
Remaining running time to full completion = 17124s
Estimated full completion date/time = 2020-12-19T18:43:05.201137
Compressor Thread-1 Permutation 8, Normalised statistic = 0.9999418434969433
Compressor Thread-2 Permutation 8, Normalised statistic = 0.9999591635337972
Compressor Thread-4 Permutation 8, Normalised statistic = 0.9999553312267921
Compressor Thread-0 Permutation 8, Normalised statistic = 0.9999522703582102
Compressor Thread-3 Permutation 8, Normalised statistic = 0.9999414951053974
CURRENT RANK = 39
Remaining running time to full completion = 17472s
Estimated full completion date/time = 2020-12-19T18:49:03.201137
Compressor Thread-2 Permutation 9, Normalised statistic = 0.9999390812496864
Compressor Thread-4 Permutation 9, Normalised statistic = 0.9999605073297599
Compressor Thread-1 Permutation 9, Normalised statistic = 0.9999547588692524
Compressor Thread-0 Permutation 9, Normalised statistic = 0.9999821822609377
Compressor Thread-3 Permutation 9, Normalised statistic = 0.9999419928076059
CURRENT RANK = 44
Remaining running time to full completion = 17741s
Estimated full completion date/time = 2020-12-19T18:53:42.201137
Compressor Thread-2 Permutation 10, Normalised statistic = 0.9999573966909572
Compressor Thread-4 Permutation 10, Normalised statistic = 0.9999463725870401
Compressor Thread-0 Permutation 10, Normalised statistic = 0.9999543607074857
Compressor Thread-1 Permutation 10, Normalised statistic = 0.9999651359602985
Compressor Thread-3 Permutation 10, Normalised statistic = 0.9999593377295701
CURRENT RANK = 49
Remaining running time to full completion = 17954s
Estimated full completion date/time = 2020-12-19T18:57:25.201137
Compressor Thread-2 Permutation 11, Normalised statistic = 0.9999565754823132
Compressor Thread-0 Permutation 11, Normalised statistic = 0.999973945289388
Compressor Thread-4 Permutation 11, Normalised statistic = 0.9999838246782256
Compressor Thread-1 Permutation 11, Normalised statistic = 0.9999716309741187
Compressor Thread-3 Permutation 11, Normalised statistic = 0.9999583672102637
Compressor Thread-2 Permutation 12, Normalised statistic = 0.9999690180375244
Compressor Thread-1 Permutation 12, Normalised statistic = 0.9999340544573811
Compressor Thread-4 Permutation 12, Normalised statistic = 1.0000074655331266
Compressor Thread-0 Permutation 12, Normalised statistic = 0.9999339549169395
Compressor Thread-3 Permutation 12, Normalised statistic = 0.9999645884878692
CURRENT RANK = 58
Remaining running time to full completion = 16607s
Estimated full completion date/time = 2020-12-19T18:35:08.201137
Compressor Thread-2 Permutation 13, Normalised statistic = 0.9999604077893183
Compressor Thread-4 Permutation 13, Normalised statistic = 0.999936990900411
Compressor Thread-1 Permutation 13, Normalised statistic = 0.9999415199905078
Compressor Thread-3 Permutation 13, Normalised statistic = 0.999946248161488
Compressor Thread-0 Permutation 13, Normalised statistic = 0.9999608806064163
CURRENT RANK = 63
Remaining running time to full completion = 16854s
Estimated full completion date/time = 2020-12-19T18:39:25.201137
Compressor Thread-2 Permutation 14, Normalised statistic = 0.9999868855468075
Compressor Thread-3 Permutation 14, Normalised statistic = 0.999961925781054
Compressor Thread-0 Permutation 14, Normalised statistic = 0.9999461983912672
Compressor Thread-4 Permutation 14, Normalised statistic = 0.9999592879593493
Compressor Thread-1 Permutation 14, Normalised statistic = 0.9999702871781558
CURRENT RANK = 68
Remaining running time to full completion = 17064s
Estimated full completion date/time = 2020-12-19T18:43:05.201137
Compressor Thread-2 Permutation 15, Normalised statistic = 0.9999614280788456
Compressor Thread-4 Permutation 15, Normalised statistic = 0.9999538132350564
Compressor Thread-0 Permutation 15, Normalised statistic = 0.999933457214731
Compressor Thread-3 Permutation 15, Normalised statistic = 0.9999416195309495
Compressor Thread-1 Permutation 15, Normalised statistic = 0.9999476417276717
CURRENT RANK = 73
Remaining running time to full completion = 17245s
Estimated full completion date/time = 2020-12-19T18:46:16.201137
Compressor Thread-2 Permutation 16, Normalised statistic = 0.9999393549859011
Compressor Thread-4 Permutation 16, Normalised statistic = 0.9999410969436306
Compressor Thread-0 Permutation 16, Normalised statistic = 0.9999582178996012
Compressor Thread-3 Permutation 16, Normalised statistic = 0.9999617515852811
Compressor Thread-1 Permutation 16, Normalised statistic = 0.999958566291147
CURRENT RANK = 78
Remaining running time to full completion = 17402s
Estimated full completion date/time = 2020-12-19T18:49:03.201137
Compressor Thread-2 Permutation 17, Normalised statistic = 0.9999513993793454
Compressor Thread-4 Permutation 17, Normalised statistic = 0.9999475173021196
Compressor Thread-0 Permutation 17, Normalised statistic = 0.9999719047103334
Compressor Thread-3 Permutation 17, Normalised statistic = 0.999941768841612
Compressor Thread-1 Permutation 17, Normalised statistic = 0.9999561026652153
Compressor Thread-2 Permutation 18, Normalised statistic = 1.0000161753217744
Compressor Thread-4 Permutation 18, Normalised statistic = 0.9999646133729796
Compressor Thread-0 Permutation 18, Normalised statistic = 0.9999470195999112
Compressor Thread-1 Permutation 18, Normalised statistic = 0.9999603580190974
Compressor Thread-3 Permutation 18, Normalised statistic = 0.9999593377295701
CURRENT RANK = 87
Remaining running time to full completion = 16557s
Estimated full completion date/time = 2020-12-19T18:35:08.201137
Compressor Thread-2 Permutation 19, Normalised statistic = 0.9999721037912167
Compressor Thread-4 Permutation 19, Normalised statistic = 0.9999784743794848
Compressor Thread-3 Permutation 19, Normalised statistic = 0.9999227317321391
Compressor Thread-0 Permutation 19, Normalised statistic = 0.99995874048692
Compressor Thread-1 Permutation 19, Normalised statistic = 0.9999979594209454
CURRENT RANK = 92
Remaining running time to full completion = 16723s
Estimated full completion date/time = 2020-12-19T18:38:04.201137
Compressor Thread-2 Permutation 20, Normalised statistic = 0.999953614154173
Compressor Thread-3 Permutation 20, Normalised statistic = 0.9999680226331075
Compressor Thread-4 Permutation 20, Normalised statistic = 0.9999761351791051
Compressor Thread-1 Permutation 20, Normalised statistic = 0.999953912775498
Compressor Thread-0 Permutation 20, Normalised statistic = 0.9999616022746185
CURRENT RANK = 97
Remaining running time to full completion = 16871s
Estimated full completion date/time = 2020-12-19T18:40:42.201137
Compressor Thread-4 Permutation 21, Normalised statistic = 0.9999425402800352
Compressor Thread-3 Permutation 21, Normalised statistic = 0.999942117233158
Compressor Thread-2 Permutation 21, Normalised statistic = 0.9999537136946147
Compressor Thread-1 Permutation 21, Normalised statistic = 0.9999611792277414
Compressor Thread-0 Permutation 21, Normalised statistic = 0.9999890256663038
CURRENT RANK = 102
Remaining running time to full completion = 17004s
Estimated full completion date/time = 2020-12-19T18:43:05.201137

----------------------------------
Ranked as > 102/10,000
*** PASSED permutation test. There is no evidence that the data is not IID ***
Based on 105 permutations.
Test results in file: /tmp/results.json

----------------------------------
Compressor Thread-4 Permutation 22, Normalised statistic = 0.9999939529181674
Compressor Thread-3 Permutation 22, Normalised statistic = 0.9999843970357652
Compressor Thread-2 Permutation 22, Normalised statistic = 0.9999586160613678
Compressor Thread-1 Permutation 22, Normalised statistic = 0.999959039108245
Compressor Thread-0 Permutation 22, Normalised statistic = 0.9999566003674237
$ python iid-test.py /tmp/mata-hari-10mb-x8bit.bin


LZMA tests...
Permutation 1 : NTS = 1.0
Permutation 2 : NTS = 1.0
Permutation 3 : NTS = 1.0
Permutation 4 : NTS = 1.0
Permutation 5 : NTS = 1.0
Permutation 6 : NTS = 1.0
Permutation 7 : NTS = 1.0
Permutation 8 : NTS = 1.0
Permutation 9 : NTS = 1.0
Permutation 10 : NTS = 1.0


BZ2 tests...
Permutation 1 : NTS = 0.9995848759456019
Permutation 2 : NTS = 0.9995054952708632
Permutation 3 : NTS = 0.9996467095723792
Permutation 4 : NTS = 0.999834780003603
Permutation 5 : NTS = 1.0000228901046078
Permutation 6 : NTS = 1.0000816067371336
Permutation 7 : NTS = 0.999754223964656
Permutation 8 : NTS = 1.0005593603783745
Permutation 9 : NTS = 0.999747255885505
Permutation 10 : NTS = 0.9999542118161385


ZLIB tests...
Permutation 1 : NTS = 1.0
Permutation 2 : NTS = 1.0
Permutation 3 : NTS = 1.0
Permutation 4 : NTS = 1.0
Permutation 5 : NTS = 1.0
Permutation 6 : NTS = 1.0
Permutation 7 : NTS = 1.0
Permutation 8 : NTS = 1.0
Permutation 9 : NTS = 1.0
Permutation 10 : NTS = 1.0
-----------------------------------
Broke file into 1,000,000 byte segments.
Tested 10,000,000 bytes for each compressor.
Using 3 compressors.
Minimum NTS = 0.9995054952708632
Maximum NTS = 1.0005593603783745
Mean NTS = 0.9999563803226289
66.7% unchanged by shuffle.
Probability of 13 heads and 17 tails = 0.5847


*** Accept Ho. The data file looks like IID ***
$ ea_iid  -v  /tmp/mata-hari-10mb-x8bit.bin
Opening file: '/tmp/mata-hari-10mb-x8bit.bin'
Loaded 10000000 samples of 256 distinct 8-bit-wide symbols
Number of Binary samples: 80000000
Calculating baseline statistics...
	Raw Mean: 127.137822
	Median: 127.000000
	Binary: false

Literal MCV Estimate: mode = 39996, p-hat = 0.0039995999999999999, p_u = 0.0040510108918163411
Bitstring MCV Estimate: mode = 40108361, p-hat = 0.50135451249999996, p_u = 0.50149850520805828
H_original: 7.947502
H_bitstring: 0.995683
min(H_original, 8 X H_bitstring): 7.947502          <<<< Hmin.= 7.9 bits/byte. Whoopie!

Chi square independence
	score = 64919.272275
	degrees of freedom = 65280
	p-value = 0.840940

Chi square goodness of fit
	score = 2489.421745
	degrees of freedom = 2295
	p-value = 0.002538

** Passed chi square tests

LiteralLongest Repeated Substring results
	P_col: 0.0039066
	Length of LRS: 5
	Pr(X >= 1) rounds to 1, but there is precision loss. The test verdict is still expected to be valid.
** Passed length of longest repeated substring test

Beginning initial tests...

Initial test results
              excursion: 463049
     numDirectionalRuns: 6.6688e+06
     lenDirectionalRuns: 9
  numIncreasesDecreases: 5.01931e+06
          numRunsMedian: 5.00035e+06
          lenRunsMedian: 23
           avgCollision: 19.7551
           maxCollision: 74
         periodicity(1): 39066
         periodicity(2): 39107
         periodicity(8): 38874
        periodicity(16): 38862
        periodicity(32): 38891
          covariance(1): 1.61632e+11
          covariance(2): 1.61654e+11
          covariance(8): 1.61637e+11
         covariance(16): 1.61612e+11
         covariance(32): 1.61639e+11
            compression: 1.06741e+07

Beginning permutation tests... these may take some time
 99.13% of Permutuation test rounds, 100.00% of Permutuation tests

                statistic  C[i][0]  C[i][1]  C[i][2]
----------------------------------------------------
                excursion       6       0    9457
       numDirectionalRuns       6       0     106
       lenDirectionalRuns       1       6       0
    numIncreasesDecreases       9       0       6
            numRunsMedian      10       0       6
            lenRunsMedian      10       1       5
             avgCollision       6       0     235
             maxCollision     128       2       4
           periodicity(1)       6       0      11
           periodicity(2)       6       0      12
           periodicity(8)      15       0       6
          periodicity(16)      52       0       6
          periodicity(32)      14       0       6
            covariance(1)       7       0       6
            covariance(2)       6       0      17
            covariance(8)       6       0       7
           covariance(16)     193       0       6
           covariance(32)      12       0       6
              compression       8       0       6
(* denotes failed test)

** Passed IID permutation tests
Slow IID test of REALLYREALLYRANDOM's Type 3 (Mata Hari) 8 bit entropy source.

Slow IID test for sampling at 8 samples/byte.

Fast IID test of REALLYREALLYRANDOM's Type 3 (Mata Hari) 8 bit entropy source.

Fast IID test for sampling at 8 samples/byte.

Both our and NIST’s IID tests confirm that the compressed data sample is IID. Therefore the entropy $(H_{\infty})$ rate is simply $-\log_2 (p_{max})$ or min(H_original, 8 X H_bitstring): 7.947502 taken from the NIST test output above.

Conclusion: That’s an internal on board generation rate of 7.9 bits/byte @ 8.8 kbits/s of pure IID entropy. Using a 9V battery. Of course this rate is constrained by transferring the entropy off board, and we have chosen 9600 baud for security and device compatibility reasons. Thus the Mata Hari kit feeds a client device over USB at 7.9 bits/byte @ 4.4 kbits/s.

Please just remember, this is an entropy source not a fully fledged TRNG. We then leverage the Leftover Hash Lemma to reduce the bias to a world beating $2^{-128}$.

Related files