1) Benchmarking CPU's for Snowie rollouts,
2) Measuring Snowie rollout runtime for the various parameter settings available,
3) Comparison of rollout results for different Snowie settings as well as different robots.

This article is a report on the current results of part 1.

A standard position and set of rollout parameters were chosen. The position is the opening setup with a diceroll of 62. The rollouts consisted of 36 trials of each of three candidate plays: 24/18, 13/11; 24/16; and 13/5. The rollout parameters were: cubeless money play, 3-ply truncated at 10 rolls, truncation if bearoff database reached, medium search space, 20% speed, seed = 12345. Several GoL members took on the task with their respective CPU's and they are credited below for their contributions.

Although cputime for a given rollout should roughly correlate with CPU clock speed, different processors handle programs differently, particularly when both integer and floating point arithmetic are used. The table below is a summary of the various results reported both publically on the forum and to me privately. Also included is a plot of Snowie's runspeed (1000/runtime where 1000 is chosen to make the resulting numbers manageable and 'runtime' is given in seconds). The Snowie runspeed is plotted as a function of CPU clockspeed.

Contributor     SW ver. Processor     Proc.  Mem.   OS     Tot Time  Normalized Projected   Ratio
                                      Speed                  (sec)     Speed      Speed   Norm/Proj
JP White          3.0   P-166         166    64MB   Win98     7555     0.13       0.15      0.87
Chuck Bower       3.0   PII-233       233    32MB   Win98     4042     0.25       0.25      0.99
Jason Lee         3.0   PII-233       233    64MB   Win98     3299     0.30       0.25      1.21
JP White          3.0   AMD K6-300    300    128MB  Win95     3917     0.26       0.35      0.73
Gregg Cattanach   3.2   PII-350       350    64MB   Win98     2174     0.46       0.42      1.09
David Montgomery  3.2   PII-400       400           Win98     2030     0.49       0.49      1.00
David Montgomery  2.3   PII-400       400           Win98     1948     0.51       0.49      1.04
Robi              3.2   PII-400       400    128MB  Win98     2104     0.48       0.49      0.96
Chuck Bower       3.2   PIII-450      450    128MB  WinNT4.0  1667     0.60       0.57      1.06
Gregg Cattanach   3.2   PIII-450      450    128MB  Win98     1752     0.57       0.57      1.01
JP White          3.2   AMD K6-2 450  450    128MB  Win98     2161     0.46       0.57      0.82
JP White          3.2   AMD K6-2 450  450    128MB  Win98     2253     0.44       0.57      0.78
Rob Maier         3.2   Celeron-466   466                     1810     0.55       0.59      0.94
Chris Yep         3.2   PIII-500      500    96MB   Win98     1571     0.64       0.64      1.00
TarHeelFan        2.3   PIII-600      600    256MB  Win2000   1236     0.81       0.79      1.03
Bob Stringer      3.2   PIII-1000     1000   256MB  Win98      782     1.28       1.37      0.93
Petter Bengsston  3.2   AMD-1000      1000   384MB  Win2000    636     1.57       1.37      1.15

You will notice that the data conform roughly to a linear relationship, which should not be surprising. I performed an 'eyeball' linear fit to the data. The second to last column of the table is a predicted speed based upon this fit. The last column is the ratio of the normalized speed (1000/actual runtime) divided by the predicted speed. Although one must be careful about drawing too much from this comparison, a few generalities seem to be evident.

The early AMD processors (K6 models) underperform their Intel cousins by about 20-25%. However the latest generations of processors reverses this trend. In particular the AMD Thunderbird (Athlon?) running at 1 GHz beats the prediction by 15%, while the Intel processor running the same clock speed might even be a few percent below the projection. Both of these occurences were predicted by forum posts prior to the results becoming available and I refer the interested reader to the thread which I initiated on January 6th for more details. Finally, admittedly based upon limited data, the Intel Celeron processor may be a bit slower than it's PII and PIII siblings, though if the effect is real it is only a few percent. I believe all of this is consistent with known performance differences of these processors, particluarly when performing floating point operations.

Again cautioning that the data is limited, there may be a few percent affect due to the version of Snowie being run (2.x or 3.y) with the older version running a bit faster. There is not an obvious connection relating to either operating system nor amount of available memory.

Concerning part II (runtime dependence on Snowie parameter settings), I have compliled quite a bit of data and will report the results next month. There is still considerable help needed for part III (rollout results for different robots and for different Snowie settings). Again I'm asking for help from GoL members. Here the position and candidate plays are the same as in the other parts, but the settings (including number of trials) can and should be different. In particular, since there is the potential of combining results, the use of a random seed would be safest, but in particular DON'T use 12345. If you have a favorite seed, that is probably OK.

Part III (when combined with part II) may help answer a longstanding question: what are the most efficient Snowie settings for getting fast, reliable rollout results? In addition, by comparing rollout results from different robots, we may be able to speculate about the relative strengths of different creations. I plan on setting up the old Expert Backgammon (version I). Gary Wong has submitted data for gnu-backgammon (but more would be welcome). Jellyfish is a must (any volunteers?) and I'm going to ask at least a couple authors of homegrown robots for their input. Again, any contributions you can make would be valuable and appreciated.