It isn't a very reliable benchmark. Consecutively repeating the benchmark results in better and better results. Therefore, results can vary greatly even on the same Pocket PC. This is a sign that the processor cache isn't being flushed after every benchmark run.

As such, the only way to benchmark accurately using VOBenchmark is to do a soft-reset immediately before each test run. The soft-reset will terminate all running programs and flush whatever caches there are. Running VOBenchmark right after the soft-reset will yield the highest (slowest) but also the most accurate scores. To further improve the accuracy of the benchmark, you should run it at least 3 times, with soft-resets between them.

Note that VOBenchmark scores are not comparable unless the pre-test conditions are similar. That means if you want to benchmark your Pocket PC for submission to this comparison or for your own personal use, it is best to follow the steps above. Otherwise, your results will be skewed and cannot be compared with benchmark results of other Pocket PCs.

Although VOBenchmark shows us how fast the Pocket PC runs, it does not test the speed of the connection between the Pocket PC and the desktop PC. For that, we have prepared a special package of files to test the transfer rate between the Pocket PC and the desktop.

To use this benchmark, download the benchmark and extract it to an empty directory. Here are the details on the benchmark (extracted from the included Benchmark.txt file) :-

ARP Pocket PC File Transfer Test Rev. 1.0
----------------------------------------

Contents
---------
This is a simple collection of files that attempts to simulate the usual collection of files that a 
Pocket PC user would normally load into his/her Pocket PC. This collection consists of :-

- 3 games (192,000 bytes)
- 6 Pocket PC-optimized JPEG pictures (155,953 bytes)
- 3 Microsoft Reader eBooks (416,928 bytes)
- 1 Microsoft Word 2000 documents (209,513 bytes)
- 1 WMA files recorded at 48kbps stereo (1,013,349 bytes)
- Benchmark.txt (12,257 bytes)

Note that Benchmark.txt IS part of the test.

The total size of this collection of files is exactly 5,000,000 bytes so you will have to reserve
at least 2,000,000 bytes of free space on your Pocket PC.


Benchmarking
-------------
To get as accurate a result as possible, follow these steps :-

1. Run ActiveSync and synchronize everything (including Files) first.
2. Open up ActiveSync options and select Synchronize Manually.
3. Then, open "Pocket_PC My Documents" which is normally located on your PC's desktop.
4. Create a new empty directory in it and extract the benchmark files there.
5. Remove your Pocket PC from its cradle and soft-reset it. (This step is very important
because it makes sure there are no programs running in the background)
6. Replace your Pocket PC into its cradle and it should begin checking for changes.
7. Once it's done checking, ActiveSync should show 15 items (16 if you did not create the
temporary directory first) left to be synched under Files.
8. Prepare a stopwatch.
9. Simultaneously start the stopwatch and click on the Sync button.
10. Let ActiveSync transfer the files over to your Pocket PC.
11. Keep a close eye on the file transfers.
12. As soon as ActiveSync finishes copying the files over, stop the stopwatch and record the time.


Note - ActiveSync may say that it cannot sync the Word document file. This is a bug in ActiveSync.
To solve this, quit ActiveSync and restart it. It should sync the files properly now. Better still,
get a newer version of ActiveSync!

Calculations
------------
To get the exact transfer rate of your serial/USB connection, just divide 2,000,000 bytes by the
time that you recorded to get a transfer rate in bytes per second. For example, if a transfer takes 160
seconds to run, the transfer rate would be = 2,000,000 bytes / 160 seconds
= 12500 bytes / second.

For more accurate results, it's best to run this test three times and use the average time.

As you can see, the benchmark is only 2.0MB in size (1.65MB to download). That will enable quick and easy benchmarking of virtually all Pocket PCs. How quick? Well, on my slow serial connection, the test takes less than 5 minutes so if you have a USB connection, you can be assured that this file transfer benchmark will take less than 5 minutes to complete. 

Future Benchmarks

Current benchmarks are not as good as they should be. We hope that there will be better benchmarks soon. We have many ideas on what the ideal Pocket PC benchmark would be. So, if you are a WinCE programmer and if you are interested in helping us develop a proper benchmark suite for the Pocket PC, please contact us immediately. 

Submitting Your Results

Please support our Pocket PC benchmarking efforts by e-mailing us your benchmark results. When you do, please make sure you include details about :-

• your Pocket PC
• the connection you used in the benchmark, i.e. serial / USB / IR / Ethernet

Thanks! :D

Adrian Wong
Adrian's Rojak Pot
http://www.rojakpot.com/
http://www.adriansrojakpot.com/