Mathematica Achieves Unparalleled Accuracy as a Statistical
Package
September 11, 2000An independent assessment of Mathematica
entitled "The
Accuracy of Mathematica 4 as a Statistical Package" was published
in the
September 2000 issue of Computational Statistics. In it, the
precision of
Mathematica's calculations was found to be "markedly superior" to
that of
other major statistical packages such as SAS, SPlus, SPSS, and Excel. The
paper was written by Dr. B. D. McCullough, a senior economist at the
Federal
Communications Commission, who has published several articles on the
reliability of statistical and econometric software.
Applying the same methodology to Mathematica as that used on other
packages, McCullough assessed reliability in three areas: linear and nonlinear
estimation, random number generation, and statistical distributions.
McCullough found that "Mathematica achieves unparalleled accuracy
and
reliability on the National Institute of Standards and Technology (NIST)
Standard Reference Datasets (StRD) and on the ELV benchmark for
statistical distributions."
The typical statistical package relies on fixed, machineprecision
calculation that involves approximations and roundoffs, thereby
introducing
error. However, "by virtue of its variableprecision arithmetic and
symbolic power, Mathematica's performance on these reliability
tests far
exceeds any finiteprecision statistical package," notes McCullough. But
since much data is recorded to only three or four significant figures,
what
makes Mathematica's accuracy so important?
McCullough states that the purpose of "benchmarking," as the assessment
tests are called, is "not to count digits, but to assess the quality of
the
implemented algorithm" used to perform a calculation. The questions of
interest to users are "Where does it break down?" and "Will the program
warn
me?" For example, Microsoft Excel, arguably the package most commonly used
for statistical calculation, has been found to perform inadequately in
almost all areas of the StRD.
StRD Results for Nonlinear Problem Ratkowsky 43
Coefficient

NIST

Excel

b1

699.64151270

676.0986499

b2

5.2771253025

39.7190456

b3

0.75962938329

4.559009025

b4

1.2792483859

13.02379155

McCullough and Wilson, Computational Statistics
and Data Analysis 31 (1999)
Even at default settings, Mathematica does not break down on any of
the
tests. Mathematica not only will inform the user if it cannot
complete the
procedure as defined but also will tell the user what settings need to be altered
in
order to complete the computation. By increasing the required precision of
the calculation, Mathematica is able to produce a perfect score on
all tests
in all four areas of the StRD. Although other programs are able to produce
"correct" solutions (accurate to three to four significant digits) on the tests, no
other program can come close to matching Mathematica's overall
performance.
Overall: 58 Problems (taking best options)
Comparison of McCullough Results to
Previous Package Assessments (2000)
McCullough highlights Wolfram Research's excellent technical support and
gigaNumerics, a project that uses more efficient data structures and
algorithms for speed gains in processing very large numbers, as additional
reasons for his confidence in Mathematica. Given the trend toward
increasingly large computer memory and processing speedand increasingly
large datasetsMcCullough says that the "potential [of cumulative rounding
error] for completely corrupting results obtained using traditional
methods and algorithms, is most worrisome. Hope may well lie in
Mathematica's
variableprecision arithmetic and Wolfram [Research]'s gigaNumerics project."
It is the combination of all of these readily accessible features that makes
Mathematica so accurate and straightforward to use, even for the
newcomer.
McCullough himself says he "shall be using Mathematica regularly
for
statistical purposes" as a supplement to his regular package. Wolfram
Research's commitment to "forwardlooking" development, and the pending
release of new statistics addon packages for Mathematica, will
only serve to make it a more natural application.
For more information on Mathematica in statistics, visit the
Mathematica
Statistics Solutions on our web site.

