AUGI Gauge benchmark results for AutoCAD 2002, 2000i, 2000, and R14

by Mark Middlebrook

This is the report from my testing of AutoCAD 2002, 2000i, 2000, and Release 14 in May 2001 using the AUGI Gauge benchmark test suite. Autodesk hired me to do the testing and write the report, but I did the testing in my own office on my own test computer, and I wrote the report without direction from anyone at Autodesk.

Table of Contents

Introduction

About the AUGI Gauge
Testing Setup and Procedures
AUGI Gauge Results
Limitations
Conclusion

Appendix A: Charts

Introduction

With the release of AutoCAD® 2002 comes the perennial question, "what about performance?" Is the new version of AutoCAD faster or slower than previous versions? In what operations or what kinds of drawings is it faster or slower? Will AutoCAD users who upgrade need to consider buying more memory or a faster computer?

Since 1996, I have worked with Autodesk and then the Autodesk® User Group International (AUGI®) to develop the AUGI Gauge, a performance-testing engine and set of test suites. I also helped perform benchmark testing and publish results on AutoCAD Release 14 and AutoCAD 2000 when they first became available.

In May 2001, at the request of Autodesk, I used the AUGI Gauge to test the performance of AutoCAD 2002, AutoCAD 2000i, AutoCAD 2000, and AutoCAD Release 14. I used the same computer, AUGI Gauge test engine, and test suites that we used to compare AutoCAD 2000 and Release 14 during the AUGI benchmarking project in April 1999. (The report for the 1999 testing is available on the AUGI Web site. Its current address is www.augi.com/empower/benchmark/gauge/introduction.asp. In order to view the report, you will need to log in to the site as an AUGI member, which you can do by signing up for a free membership on-line.)

This report contains my results and analysis from testing the four versions of AutoCAD using the AUGI Gauge. Note that these results and analysis are my own. AUGI is not responsible for any of the results or analyses in this report.

About the AUGI Gauge

The AUGI Gauge is an AutoLISP® and Visual Basic® based performance-testing tool that one can use to develop benchmark scripts for testing different operations and different drawings. The AUGI Gauge test suite includes two test scripts:

A Real-World Drawings test script that exercises file input/output (I/O), object selection, editing and display operations on a series of 15 drawings ranging in size from 700 KB to 4 MB. The average drawing size with attached xrefs and images is 2 MB. The drawings represent a range of disciplines, object types, and drawing organization schemes (e.g., xrefs and paper space).
A Synthetic test script that exercises additional commands (such as creating objects and manipulating 3D views).

The current AUGI Gauge test suites are designed to compare raw performance in a range of average drawings. The test engine can be extended to other areas of interest, but you should avoid making unwarranted assumptions or extrapolations based on the AUGI Gauge (or any other benchmark) results. For example, the AUGI Gauge doesn't tell you how responsive the AutoCAD dialog boxes and other components of the user interface are.

See the report at www.augi.com/empower/benchmark/gauge/introduction.asp for more information about the history and current state of the AUGI Gauge.

Testing Setup and Procedures

I was able to use the same version of the AUGI Gauge (15.0) that we developed for testing AutoCAD 2000 in April 1999, without modification to the testing engine or test scripts. The AUGI Gauge AutoLISP test engine performs some complicated gymnastics in order to load drawings. In addition, all of the test operations are dependent on the specific command line sequences in AutoCAD 2000 and Release 14 - changes to AutoCAD commands often require modifications to the AUGI Gauge test scripts. Thus, the fact that the AUGI Gauge ran without my having to make modifications supports Autodesk's assertion of excellent application compatibility between AutoCAD 2000 and AutoCAD 2002, at least for AutoLISP applications. I ran the AUGI Gauge tests on the same computer that we used to do the AUGI testing in April 1999:

IBM IntelliStation E Pro 6893
450 MHz Pentium II processor
128MB memory
Matrox MGA-G200 AGP display adapter with 8 MB video memory
Screen resolution of 1024x68 at 24-bit color depth
Windows 2000 Professional operating system with NTFS disk volumes

This was a fast system two years ago, but is merely average now. In fact, my test machine corresponds to the AutoCAD 2002 recommended hardware of a Pentium II processor, 128 MB of RAM, and 1024x768 VGA video display.

Note that the April 1999 tests were run on Windows NT 4 WorkStation, while I used Windows 2000 Professional, so the old results are not comparable to the new ones.

I conducted the tests on these four AutoCAD versions with standard Autodesk-supplied display drivers:

AutoCAD 2002 first customer shipment (FCS) version
AutoCAD 2000i FCS version
AutoCAD 2000 FCS version
AutoCAD Release 14.01

Multiple Design Environment (MDE) was turned off in AutoCAD 2002, 2000i, and 2000, so that all versions of AutoCAD were testing one drawing at a time.

I defragmented the hard disk and rebooted the computer between each test run. I ran each test with each version of AutoCAD three times for a total of 24 test runs. I then assembled the test reports in a spreadsheet, checked them for consistency, averaged each set of three test runs, and graphed the results.

AUGI Gauge Results

The charts shown in Appendix A give the results from the Real-World drawings tests, grouped by operation type (file I/O, object selection, editing, and display). The charts show aggregate times for all 15 Real-World drawings with the vertical scale indicating times in seconds. Thus, shorter bars are better. (The numbers above the bars are rounded to the nearest whole number of seconds for clarity, which is why some bars with the same numbers are a slightly different height. The data from which the charts were generated have a precision of two decimal places.)

The File I/O, Selection & Change Properties, Editing and Display charts in Figures 1 through 4 show performance holding steady or improving for most operations in succeeding versions of AutoCAD.

Figure 1 shows that the Wblock, Xref attach, and Insert file operations took about the same amount of time in all four AutoCAD versions. AutoCAD 2002 File Open performance is midway between that of AutoCAD 2000 and 2000i, each of which improved a bit on its preceding version. File Saves took about the same amount of time in AutoCAD 2000 and AutoCAD Release 14, but improved considerably in AutoCAD 2000i, and improved a bit more in AutoCAD 2002. Dxfout and Dxfin performance improved by a wide margin in AutoCAD 2000 and improved some more in AutoCAD 2002.

In Figure 2, AutoCAD Release 14 shows a slight advantage in some object selection operations, but AutoCAD 2002 has a small advantage in Chprop operations (changing properties of a large number of objects).

Figures 3 and 4 show a similar picture for edit and display operations to that in Figure 1 - either no significant change or modest to noticeable improvement in AutoCAD 2002.

I think that much of the AUGI Gauge performance gain shown by the last three versions of AutoCAD have come from much-improved TrueType font (TTF) speed in AutoCAD. Figure 5 separates the Real-World drawings into two groups: four drawings with a significant amount of TTF text, and 11 drawings without (i.e., with all or most text using native AutoCAD SHX fonts). The chart in Figure 5 sums all of the operations - file I/O, object selection, editing, and display - and groups the totals by drawing type: TTF-heavy drawings versus those without much TTF text. For the non-TTF drawings, overall performance is within 5% for all four AutoCAD versions. For TTF-heavy drawings, overall performance improved almost 20% in AutoCAD 2000 and another 30% in AutoCAD 2000i and AutoCAD 2002. Autodesk continues to do an admirable job of closing the performance gap between TTF and SHX fonts.

Another interesting way of "slicing" the data is by DWG file size. Figure 6 groups the drawings into three categories: five drawings between 750 KB and 1 MB, five drawings between 1 MB and 2 MB, and five drawings between 2 MB and 5 MB. This chart shows almost identical performance in the larger drawings, better performance in succeeding AutoCAD versions for the mid-size drawings, and better performance still in the smaller drawings.

Bear in mind that that the drawing files selected for the Real-World test represent a broad cross-section of disciplines, AutoCAD object types, organizational schemes (xrefs, paper space, etc.), and other variables. Segregating the results by drawing size is only one of many ways to "slice" the data into categories. Each group of drawings (small, medium, and large) in the Real-World test represents a somewhat different mix of disciplines and object types. Different groups of small, medium, and large drawings might exhibit different performance results.

Limitations

My testing was limited to a single computer and operating system. AutoCAD and all of the drawings resided on the computer's local hard disk, so there was no testing of network performance. The current AUGI Gauge test suite doesn't include extremely large drawings (greater than 5 MB) or multiple drawings open at the same time. And of course no one set of drawings or test operations can tell the complete AutoCAD performance story.

Those who are interested in performing their own tests with the AUGI Gauge can download it from the AUGI Web site. The current version is at www.augi.com/empower/benchmark/gauge/download.asp and requires downloading about 32 MB of files. This version officially supports AutoCAD 2000 and Release 14, but as I said earlier in this report, I had no problem running it with AutoCAD 2002 and AutoCAD 2000i. (Be aware that benchmark testing is a finicky and time-consuming business. There are many hardware and operating system variables to control, as well as lots of possibilities for assembling the test results. The AUGI Gauge on-line help file includes suggestions for testing and reporting practices.)

As always, performance benchmarking doesn't tell the whole story about a new AutoCAD release. Customers need to consider user interface responsiveness and, of course, the additional features and usability improvements that come with each new version.