Death to Dhrystone?

EEMBC today announced its first open benchmark, called CoreMark. According to EEMBC, the benchmark is "small, yet sophisticated" and can be easily ported to run on a variety of CPUs, from 8-bit microcontrollers to 32-bit embedded processors.

Since its formation in 1997, EEMBC has sought to provide rigorous benchmarks that could be used in place of Dhrystone to better measure the performance of embedded processors. As most of you know, Dhrystone has many failings, including a susceptibility to compiler optimizations and a lack of memory operations. Although EEMBC has produced a slew of well-respected benchmarks covering automotive, consumer, multimedia, networking, and other applications, Dhrystone continues to be widely quoted, even by EEMBC member companies.

The problem lies in EEMBC's publication process. EEMBC members may not publish a score unless it has been certified by EEMBC. Because certification costs money, processor vendors are not motivated to certify their scores unless they achieve industry leadership. Thus, the number of published scores has slowed to a trickle in recent years. Many vendors publish scores for some benchmarks but not others, further limiting the opportunity to make comparisons.

To avoid the certification process, most EEMBC members share their scores with potential customers only under NDA. To generate comparison data, each vendor then has to run the benchmarks on their competitors' products. This situation creates many opportunities for cheating, since none of the results is certified or vetted in any way. Thus, large customers must rerun the benchmarks themselves to verify the results. Dhrystone remains the only benchmark that is consistently and publicly available for all CPUs.

The new CoreMark aims to solve this problem by eliminating the certification requirement and offering the benchmark source code openly. CoreMark results can be generated and published by any vendor, customer, or developer, which should quickly create a large body of scores. Once that happens, perhaps we can finally bury Dhrystone for good.

CoreMark is not as thorough as the traditional EEMBC benchmarks, however; for example, it does not show the impact of memory, I/O, and application-specific functions. Markus Levy, the president of EEMBC, hopes that CoreMark will prove to be a free sample that entices more vendors to use and publish results for other EEMBC benchmarks. As someone who compares microprocessors for a living, I hope he's right. --Linley

Linley Gwennap, president and principal analyst