When reading some web content on home computers comparisons, I started asking myself if there is a way of evaluating performances on a fair bases. This is rather difficult for 8 bit home computers, as each of them usually has a unique architecture. Different processors, different memory management schemes, different Operating Systems… all of this makes it extremely difficult to do benchmarks on these systems.
But, most of these systems do have one thing in common: a BASIC interpreter in ROM, available at start-up. Different home computers models may have different BASIC dialects, but very simple programs will run on all of them.
After reading this web page, I decided to test these programs on some home computers I own. I don’t have a big collection, so I couldn’t make a perfect choice. Still, in order not to make tests too long, I decided to only test five computers. Two 128 K machines, and three 64K ones:
- Commodore 64;
- Commodore 128;
- Sinclair ZX Spectrum 128 +2;
- MSX Panasonic CF 2700;
- Amstrad CPC 464
All these tests have been performed on PAL machines.
The Commodore 64 is a computer from 1982 we definetely know about. It has a 6510 CPU running at only 0,98 MHz on PAL systems. 64K RAM of memory, only 38911 RAM bytes available for BASIC programming. Commodore 64 BASIC is actually a Microsoft BASIC implementation. 65xx family CPUs have a small instruction set and they have few registers. They enjoy fast memory access and for this reason they are “memory oriented” CPUs.
The Commodore 128 may be regarded as a powered-up version of the Commodore 64, from 1985. It has 128K RAM and a 1Mhz/2MHz 8502 CPU, very similar to a 6510. This computer has two video chips: a variation of the C64 VIC-II chip, and a VDC, for 80 columns text. It also offers bitmap graphics, with a resolution of 640 x 200. Unfortunately, the VIC-II chip is not able to work properly when the CPU is at 2 MHz. So, when on C128 40 columns mode, the CPU must be forced to 1 MHz operation. 2 MHz can be activated on 40 columns mode actually, but no image will be shown. Still, as those benchmarks programs don’t use screen, blanking the screen may be tolerated. So, if you only have RF connection and you are not able to have the VDC video output, you can test those programs by adding a BASIC line with the FAST command at the very beginning of each program, and by adding another BASIC line with the command SLOW at the end of the programs. This way, screen will be blanked only during program execution. For our benchmark tests, the 2 MHz fast mode available on C128 80 columns mode will be used, so that programs will not need to be accomodated for the FAST/SLOW modes switching.
The Sinclair ZX Spectrum 128 +2 is a Spectrum version made by Amstrad in 1986. I choose this computer because is the only Spectrum I own at the moment. I had a 48K Spectrum but I sold it some time ago (and I do regret it). Being a Commodore 64 user, the Spectrum 128 keyboard is more confortable to me than the original 48K one. That helped me to type the benchmark programs with no problem. The +2 keyboard is excellent, despite the unusual position of the DELETE key. This computer offers 128K RAM of memory, and is powered by a Z80 CPU @ 3,54 MHz. This clock speed is much greater than the Commodore 64 one. The Z80 CPU is an 8 bit CPU, but it does have internal 16 bit registers. Furthermore, it has a bigger instructions set than 6502 CPUs and it is usually better for vectors calculations. Z80s are “register oriented” CPUs, as they have many registers in comparison to 6502s.
The PANASONIC CF 2700 is a MSX 1 machine. This is not very common, but this is the only MSX 1 machine I now have. By the way, I wanted another Z80 based machine to go along with the Spectrum, and here it is. It actually features a Z80 clone by NEC, but I don’t think it makes any difference. MSX machines have a vastly upgraded version of Microsoft BASIC, with many instructions.
The Amstrad CPC 464 was introduced in 1984 as a competitor to the Commodore 64 and ZX Spectrum. It was based on a Z80 CPU, running at 4.0 MHz, and on a Motorola 6845 video chip. Sound was offered by the AY-3-8910 chip. This model features 64 Kbytes of RAM. A BASIC interpreter made by Locomotive Software Ltd. is built in ROM.
The Commodore 64 has the older interpreter and the lowest CPU clock speed among these computers.
Now, let’s talk about those speed test programs.
With slight modifications of the generic versions, these benchmarks were made capable of running on all the machines with no custom code. This way, for each program the code is the same for every machine. For the Commodore 128, the FAST command was executed in direct mode before running the programs. Again, programs are taken from this page.
Benchmark 1
20 FOR k=1 TO 1000
40 NEXT k
FOR NEXT cycles are really common on BASIC programs, so their execution speed gives a good idea of the interpreter overall performance.
Benchmark 2
20 LET k=0
30 LET k=k+1
50 IF k<1000 THEN GOTO 30
Instead of the more sophisticated FOR NEXT cycle, the iteration is performed by using a counter and a GOTO instruction. This is less efficient but allows to make an estimation on how fast branches are executed.
Benchmark 3
20 LET k=0
30 LET k=k+1
40 LET a=k/k*k+k-k
50 IF k<1000 THEN GOTO 30
Calculations have been added.This way, speed on simple computations and on access to variables may be evaluated.
Benchmark 4
20 LET k=0
30 LET k=k+1
40 LET a=k/2*3+4-5
50 IF k<1000 THEN GOTO 30
Very similar to the previous test, but variables have been replaced by constants on calculations. If variable access is slow, this test will run faster than benchmark 3.
Benchmark 5
20 LET k=0
30 LET k=k+1
40 LET a=k/2*3+4-5
45 GOSUB 700
50 IF k<1000 THEN GOTO 30
60 STOP
700 RETURN
A call to a subroutine has been introduced. Subroutines are frequently used in BASIC programming, so if subroutines calls are slow, the whole program will be slow as well. I added a line: 60 STOP, so that no “RETURN WITHOUT GOSUB ERROR” will be shown.
Benchmark 6
20 LET k=0
25 DIM m(5)
30 LET k=k+1
40 LET a=k/2*3+4-5
45 GOSUB 700
46 FOR l=1 TO 5
48 NEXT l
50 IF k<1000 THEN GOTO 30
60 STOP
700 RETURN
This benchmark creates an array of five numeric elements. It also adds a FOR NEXT loop. The array will be needed on the next benchmark.
Benchmark 7
20 LET k=0
25 DIM m(5)
30 LET k=k+1
40 LET a=k/2*3+4-5
45 GOSUB 700
46 FOR l=1 TO 5
47 LET m(l)=a
48 NEXT l
50 IF k<1000 THEN GOTO 30
60 STOP
700 RETURN
Now, the array is used by assigning a value to each element of it. The slower this benchmark is in comparison to benchmark 6, the less efficient is array access.
Benchmark 8
20 LET k=0
25 DIM m(5)
30 LET k=k+1
40 LET a=k^2
45 LET b=LN(k)
47 LET c=SIN(k)
50 IF k<1000 THEN GOTO 30
This benchmark evaluates the efficiency of the implementation of mathematical functions. So, this program may give you an idea on how much a given interpreter is more or less adequate for creating efficient scientific programs.
THE RESULTS
The clock speed of the CPU doesn’t seem to always reflect the effective speed these programs will run at. Actually, the Commodore 64 computer, powered by a 0,98 MHz 6510 CPU, seems here to be faster than the MSX computer in average. It is also without doubt much faster on running those programs than Spectrum machines. Sinclair BASIC is actually the slowest on these performance tests. However, it should be noted that this interpreter was written for compactness rather than for speed.
However, the Amstrad CPC performs much better than the other Z80 based systems, sporting excellent execution times with these benchmarks, doing much better than the Commodore 64. The Amstrad machine also offers better performance than the Commodore 128 in 2MHz mode. On the first seven benchmarks, there’s no great difference between the Amstrad CPC and the Commodore 128. But, execution time of benchmark 8 on the Commodore 128 is twice as the time required by the Amstrad CPC. That means that Amstrad Locomotive BASIC provides much faster math routines than Commodore BASIC.
While doing tests on the Commodore 64, I intentionally kept inserted the Final Cartridge III I routinely use. This cartridge may slow down BASIC a bit, but as it is common practice to use cartridge tools on the C64 (for instance, to gain faster and easier disk access) I decided to keep it active while doing benchmarks. It shouldn’t make a great difference, so I don’t think this is unfair to the Commodore 64.
The MSX BASIC doesn’t seem very fast, although perfomances are much better than Sinclair BASIC. But, its performances are inferior in comparison to those of the Commodore 128 in 2MHz mode and of the Amstrad CPC 464. Indeed, some benchmarks on the MSX machine are faster than the ones on the Commodore 64, but benchmark 8 is much more faster on the Commodore 64.
It seems that, at least with some interpreters, 65xx CPUs have better average performances than Z80s CPUs. This may be due to the fact that executing BASIC programs requires many variables involved. So, maybe Z80 registers get saturated and fast memory access of 65xx CPUs allows for better performances on those particular situations.
Nevertheless, the Amstrad CPC (with a 4 MHz Z80 CPU) offers excellent performances on these benchmarks, turning out to be the fastest machine here (there are also other benchmarks were the Amstrad CPC offers very good performances). So, maybe it is just that MSX BASIC and Sinclair BASIC are both not well optimized for speed performances.
I do own a Commodore 16 and a Commodore VIC 20, but I decided not to test them, in order not to have too much Commodore machines on the “competition”.
Those tests are pretty aleatory. I think they give a good idea of the performances of the BASIC interpreters, but they cannot be regarded as fully representative of the overall performances of a given machine. Benchmarks written on machine language may give different results. But, BASIC allows to test the very same code on each machine, something Machine Language doesn’t make possible.
Futhermore, those programs are short. Each interpreter should be tested on very long programs as well. Accuracy on calculations that each interpreter offers should also be taken into account.
All that said, here are the results. I measured the time manually for each program execution, without using internal clocks on the computers that have it. This way, the same method of measurement has been used for all the machines.
SPECTRUM 128 +2 (grey)
Bench01: 6,2 sec Bench02: 12,8 sec Bench03: 26,2 sec Bench04: 25,5 sec Bench05: 32,0 sec Bench06: 1 min 12,0 sec Bench07: 1 min 43,0 sec Bench08: 4 min 6,0 sec
MSX PANASONIC CF 2700
Bench01: 2,0 sec Bench02: 6,0 sec Bench03: 17,0 sec Bench04: 18,5 sec Bench05: 19,5 sec Bench06: 42,0 sec Bench07: 45,0 sec Bench08: 3 min 33,0 sec
COMMODORE 64
Bench01: 1,7 sec Bench02: 11,0 sec Bench03: 21,0 sec Bench04: 23,0 sec Bench05: 24,8 sec Bench06: 36,7 sec Bench07: 58,9 sec Bench08: 2 min 0,0 sec
COMMODORE 128 (80 COL, 2MHz mode)
Bench01: 1,2 sec Bench02: 5,7 sec Bench03: 10,7 sec Bench04: 11,2 sec Bench05: 12,5 sec Bench06: 20,0 sec Bench07: 32,5 sec Bench08: 1 min 1,0 sec
AMSTRAD CPC 464
Bench01: 1,2 sec Bench02: 3,7 sec Bench03: 9,4 sec Bench04: 9,9 sec Bench05: 10,5 sec Bench06: 19,4 sec Bench07: 31,0 sec Bench08: 34,6 sec
It should be noted that benchmark 7 on the MSX computer is noticeably faster than the same benchmark on the Commodore 64. Also, there is little difference on execution times between benchmarks 6 and 7 on the MSX machine. This may indicate a fast array access on MSX BASIC and a slower array access on Commodore BASIC. Even on the Commodore 128 and on the Amstrad CPC 464 you may notice a remarkable difference in execution time between benchmarks 6 and 7. This difference is much more noticeable on Sinclair BASIC, which seems to also provide the slowest array access among all of the tested machines.
One probable reason you found MSX BASIC, a Microsoft BASIC, to be slow is that it uses 64-bit floating point while most 8-bit BASICs use 32, 40, or 48-bit floating point.
If you benchmark for floating point accuracy, MSX BASIC beats the others, according to my testing.
The only other 8-bit BASICs I’m aware of with 64-bit floating point are also Microsoft BASICs on the Tandy 100/102, the TRS-80 Model I, III & 4 series, the Apple III, and the Atari 8-bit computers when specifying double-precision numbers.
Nice comparison and benchmark!
In fact C16 worth a look, because the 8501 CPU it uses, can be forced into a 2.2Mhz mode (PAL machine swittched to NTSC, screen off (like the C128″s FAST command), and then it becaosme one of the fastest “classic” 8 bit processors ever made (I think it is even faster than a 4.0MHz Z80). In case of a Plus/4 it could be slower, than a C16, due to continuous ROM/RAM switches to let customers have the most memory available for the BASIC programming. This does not happen in case of C64 (that’s why has only 38K RAM) and C16 (no overlap).
On a C16 you have 13k out of 16k available, thus it has the same dynamic bank switching scheme as the Plus/4. In fact, if you expand the memory of a C16 you get the same amount of free memory than a Plus/4.
Thanks for reading and appreciation!
Results from 1979/1982 Atari 800 / 800XL (on NTSC, still on 16KB OS + 8KB basic ROM footprint, & fully interpreted):
Test 1: 0.5665s
Test 2: 2.7999s
Test 3: 7.1665s
Test 4: 7.5832s
Test 5: 9.199s
Test 6: 13.2165
Test 7: 20.9332
Test 8: 40.7332
TOTAL runtime (all 8 tests): 102.1998 secs (1m:42s)
MEAN runtime (non-weighted): 12.7748 secs.
Outperforming even the Amstrad CPC 464… and no compilers involved, yet.
This is a very interesting test. Thanks a lot for this.
With regard to clock speed – it is not possible to compare the 6502 (used on the Commodore) to the Z80 used on the CPC 464 and the Spectrum directly. The 6502 accessed memory every two clock cycles and the Z80 every four. This meant the Z80 could be clocked twice as fast as the 6502 for the same memory speed. This was reflected in instruction timings too. The fastest 6502 instructions ran in two clock cycles, the Z80 took four, The 6502 had few registers and relied heavily on zero page access for speed; the Z80 was much more register focused. All in all, a 2 MHz 6502 is roughly as fast as a 4 MHz Z80 and that seems reflected in your comparisons. The design of the BASIC interpreter is important too. The ZX Spectrum’s BASIC was notoriously slow – its floating point calculator is a plodding stack based implementation adapted from the ZX81, whereas the CPC 464 was much more optimized for speed.