Charlwood Integration Test Suite

Summary of Test Results

Kevin Charlwood's 2008 article Integration on Computer Algebra Systems included 50 indefinite integration problems involving a variety of elementary functions. This chart summarizes the performance of Rubi and several computer algebra systems on these problems.

The following table shows the grade given each system on each of the 50 problems in the Charlwood Test Suite:

 Problem # Rubi 4.8 Mathematica 10 Maple 18 FriCAS 1.2.1 Maxima 5.28 Derive 6.1 1 2 2 1 2 2 2 2 2 2 2 2 2 2 3 0 1 1 2 -1 0 4 1 1 1 1 0 1 5 1 1 1 0 0 0 6 2 0 2 2 0 1 7 2 1 2 2 2 2 8 2 0 1 0 0 0 9 2 1 0 2 2 2 10 2 1 2 2 0 0 11 2 2 0 2 0 2 12 1 1 1 2 0 0 13 1 1 0 2 0 0 14 2 2 0 1 0 2 15 2 1 0 2 0 2 16 2 2 0 1 2 2 17 2 1 0 1 0 2 18 2 2 1 2 2 2 19 2 2 2 2 2 2 20 2 2 0 2 2 2 21 2 1 0 1 0 0 22 2 1 0 2 0 1 23 2 2 0 2 0 2 24 2 2 2 2 2 2 25 2 2 0 1 2 2 26 2 2 0 1 0 2 27 2 2 2 1 2 2 28 2 2 0 1 2 2 29 2 2 0 2 0 2 30 2 2 0 2 0 2 31 2 2 0 1 2 2 32 2 2 2 2 2 2 33 2 2 1 1 2 2 34 2 2 2 2 2 2 35 2 2 0 2 0 2 36 2 2 2 1 2 2 37 2 1 2 1 2 2 38 2 1 1 2 0 0 39 2 1 1 2 0 0 40 2 2 2 1 1 0 41 2 1 0 1 0 0 42 1 1 1 1 0 1 43 2 1 2 1 1 2 44 1 0 1 1 0 2 45 0 1 0 1 0 0 46 2 2 1 2 2 2 47 2 1 1 1 0 1 48 2 1 2 2 1 2 49 2 2 1 1 0 2 50 2 2 2 2 0 2 Totals: 90 73 45 75 40 71 Percent: 90.0% 73.0% 45.0% 75.0% 40.0% 71.0%

Each result produced by the various systems is assigned a numeric grade based on a comparison with the optimal antiderivative as follows:

• If the system gives a result as good as the optimal antiderivative, 2 points are earned.
• If the system gives a mathematically correct result (i.e. its derivative equals the integrand) but is substantially more complex than the optimal antiderivative (e.g. significantly larger or involves the imaginary unit), 1 point is earned.
• If the system is unable to integrate the expression in closed-form or times-out after 120 seconds, 0 points are earned.
• If the system returns a mathematically incorrect result, 1 point is decremented.

So a perfect score on the Charlwood Test Suite would be a 100. Note that simplification of results using functions and/or domain restrictions on variables was not done. In other words, the comparisons are based on the results a novice user would see.