Professional Documents
Culture Documents
www.veritest.com info@veritest.com
Executive Summary
Dell Inc. commissioned VeriTest, the testing service of Lionbridge Technologies Inc., to conduct a competitive analysis comparing the performance of integrated EV-DO (Rev 0) wireless cards. VeriTest conducted a series of tests to find which notebook had the best performance in terms of overall data throughput speed. The following notebooks were tested: Dell Latitude D620 HP Compaq NC6400 Lenovo ThinkPad T60
Key Findings
Average Download Speed
839
900
DELL
465
463
HP LENOVO
The Dell notebooks EV-DO rev 0 card is manufactured by Novatel. The HP and Lenovo EV-DO rev 0 cards are made by Sierra Wireless. Each of the notebooks had an Intel Core Duo 1.83GHz processor and 1 GB (2 x 512) of RAM. Further specs can be found in Appendix A. Tests were conducted in the following US cities. Two test locations were used in each city. Each location was tested on two separate days to account for network variability. Los Angeles, California Chicago, Illinois New York, New York
On average, the Dell D620 download speeds were 82% faster than the HP NC6400 and 84% faster than the Lenovo ThinkPad T60. There was little difference between notebooks in terms of upload speeds which averaged around 130 Kbps.
The result for each test was calculated by averaging the numbers from multiple test iterations after dropping the high and low data points. From seven to twenty-two iterations were conducted for each test. The following tests were conducted: A. Ping Test to measure latency and packet loss B. Speed Test to measure throughput using web speed tools at www.dslreports.com/stest C. Web Downloading Test to measure throughput when downloading 1MB and 2MB web files D. FTP Test to measure throughput when transferring 1MB and 2MB files to and from web servers E. Email Send/Receive Test to measure throughput when sending and receiving via MS Outlook files of sizes 500KB, 1MB and 2MB After all testing was conducted, the findings show that on average the Dell D620 provided 82% more throughput than the HP NC6400 and 84% more throughput than the Lenovo Thinkpad T60. The Dell system had an average throughput speed averaged across the FTP, Web Downloading and Email tests of 839 Kbps. The HP had an average throughput of 465 Kbps and the Lenovo had an average throughput of 463 Kbps. Please refer to the Test Methodology section of this report for complete details of how testing was conducted. All units were tested using default factory settings. VeriTest confirmed that all notebook computers were new and unopened. VeriTest made sure tests were completed with impartiality.
Test Results
The following section details the results from testing.
Ping Test
Goal: Determine if packet loss or latency varies by platform. Summary of Results: All platforms were comparable in this category. Comments: There was little difference in notebook ping times. Little to no packet loss occurred on all tested platforms.
Speed Test
Goal: Measure throughput with some commonly used web speed tools. Summary of Results: The Dell notebook had the highest average throughput, but the variability in the test results was significant. Due to this variability the results from this test were excluded from the overall averages. Comments: For this speed test, http://www.dslreports.com/stest was used as the reference. This service provides many servers to test against, each using their own proprietary java applet but all producing the same measurement. We used the following servers for testing: speakeasy.net, BroadStar.com and linkline.net. Results from the speed test varied widely compared to the Web,
FTP and Email tests. In some cases the speed test results were less than half of what was achieved with other throughput tests with significantly more variability. The variability of results when compared to the other tests was significant enough to exclude this test from the overall throughput averages. Below are the results from the speed test. Notebook Kilobits per Second DELL 674.74 HP 494.87 LENOVO 475.60 Table A. Average Download Speed
Notebook Kilobits per Second DELL 121.01 HP 119.26 LENOVO 118.42 Table B. Average Upload Speed
900 800 700 600 500 400 300 200 100 0 1MB 2MB
DELL HP LENOVO
Chart 2: Average Kbps for all test run averages for the Web Downloading Test. (Higher is better)
Notebook DELL
1MB(Kbps) 920
2MB(Kbps) 950
HP 539 536 LENOVO 543 539 Table C. Average Throughput of Web Downloading Test
FTP Test
Goal: Benchmark throughput when using FTP to transfer files. Summary of Results: The Dell D620 was 78 83% faster than the HP and 74 79% faster than the Lenovo when downloading 1 and 2 MB zip files via FTP. Upload results displayed no conclusive performance differences. Comments: Downloads are the main performance measure. There were little uplink differences. The Verizon network commonly capped uplinks at about 130 Kbps.
FTP Throughput
1,000
DELL HP LENOVO
1MB up
2MB up
1MB down
2MB down
Chart 3: Average throughput for all test run averages for the FTP Test.
Notebook 1MB(Kbps) 2MB(Kbps) DELL 916 953 HP 507 514 LENOVO 518 523 Table D. Average Download Speed of FTP Test Notebook 1MB(Kbps) 2MB(Kbps) DELL 136 131 HP 135 129 LENOVO 134 129 Table E. Average Upload Speed of FTP Test
Email Test
Goal: Benchmark throughput when sending and receiving email files.
Summary of Results: The Dell D620 was 92 - 94% faster than the HP and 95 107% faster than the Lenovo when downloading emails with 1 and 2 MB zip file attachments using Outlook. The Dell D620 was 78% faster than the HP and 86% faster than the Lenovo when downloading emails with 500 KB zip file attachments.
Seconds
DELL HP LENOVO
500K
1MB
2MB
Chart 4: Average time to download for all test run averages for the Email Test.
Notebook 500KB(Kbps) 1MB(Kbps) 2MB(Kbps) DELL 6 12 24 HP 11 23 45 LENOVO 11 23 48 Table F. Average Download Time of Emails Test in seconds
Test Methodology
Dell Inc. commissioned VeriTest to conduct a competitive analysis comparing performance of integrated EV-DO wireless cards. VeriTest conducted a series of throughput tests to find which notebook had the greatest performance.
Notebook Configurations
The notebooks were to be as close to an out-of-the-box configuration as possible, but some third party software, such as firewalls and virus protection, had to be disabled. This ensured that some of this software, such as email scanning software, did not affect download times. The Lenovo had Symantec Firewall and Virus protection which had to be disabled. The following is a list of the software that was required for testing which needed to be installed: o o Microsoft Office 2003 with Outlook logging enabled for the Email Test. Wget for DOS (an application distributed under the GNU license) for the Web Downloading Test.
Test Locations
The test locations included areas with both strong and weak signals. Locations used were deemed typical for users of integrated EV-DO and included hotels and coffee shops. Some locations provided unreliable connections resulting in data with such large standard deviations as to be statistically useless. Data from these locations was not used in this report. Outdoor locations were not used. Several test runs would take 8 hours or more requiring the availability of AC power to keep the notebooks charged.
There are many factors affecting actual throughput analysis such as structures, latency, network traffic, etc. Because these factors were out of our control and testing was not done in a controlled lab the results may not be reproducible. The tests were designed to be objective by ensuring that each notebook was affected by the same or similar environmental factors.
Calculating Average
Once a test run was complete the highest and lowest data points from each test for each notebook was removed and the remaining data points were averaged. Those results can be found in the Detailed Results section.
Test Details
The following describes each test in detail and includes information on test development, description of execution and data gathering. A. Ping Test Test Description A ping sends a data packet out to a location on a network and is generally used to test network connectivity. Test Development A batch file was written which would execute a ping command and send ten (10) 32 bytes to the following five sites and write the results to a file on the desktop:
Test Execution and Data Gathering The Ping Test always ran for two iterations and usually at the beginning of every test run. Double clicking the batch file would run the test. A log file was created on the desktop upon completion. B. Speed Test Test Description www.dslreports.com/stest is a commonly used tool for measuring throughput. The servers we connected for this study were BroadStar.com, linkline.net and speakeasy.net. Test Development No development was necessary for this test. Test Execution and Data Gathering A server was chosen to test with on the page from the above link. The instructions were followed to complete the test from that server using a java applet located on the page. Results were copied into a text file on the desktop and the speed data into an Excel spreadsheet to be averaged. C. Web Downloading Test Test Description Wget for DOS was used to download a 1 megabyte and 2 megabyte zip file from a web site. Zip files were used to eliminate the possibility of file transfer compression affecting test results. Test Development The Wget application was installed on each notebook. A batch file executed the wget command with the logging option to download the two zip files and log the results. The zip files were located on a low traffic web server from a local ISP, CableOne. Test Execution and Data Gathering Double clicking the batch file would run the test. A log file was created on the desktop upon completion. The start and stop time data as well as the KB/s from each download were copied into an Excel spreadsheet to be averaged. D. FTP Test Test Description FTP is one of the most common methods for transferring large files over the internet. To test throughput speeds via FTP Microsoft XPs integrated FTP client was used to download and upload a 1 megabyte zip file and a 2 megabyte zip file. Zip files were used to eliminate the possibility of file transfer compression affecting test results. Test Development
A batch file was written to log onto an FTP server and run put/get commands which would send and receive the files via FTP then log the results to a file on the Desktop. The FTP server used for this test was provided by a low traffic local ISP, CableOne.
Test Execution and Data Gathering Double clicking the batch file would run the test. Upon completion a log file was created on the desktop. The time in seconds it took for each download and upload to complete as well as the KB/s were copied into an Excel file to be averaged. E. Email Test Test Description Three emails were downloaded using Microsoft Outlook 2003 to test download speed via SMTP. The three emails each had a zip file attachment of a predetermined size. The sizes were 500 kilobytes, 1 megabyte and 2 megabytes. Zip files were used to eliminate the possibility of file transfer compression affecting test results. Test Development A PHP script was written to automate the process of sending the emails of different sizes. This script resided on a company Linux server. Each of the notebooks was configured with a different email address so that the three emails could be sent and tested by all three notebooks by running the script once. The email accounts were originally set up with Gmail.com but were moved due to traffic congestion to a local ISP, CableOne. Test Execution and Data Gathering The PHP script was executed via a web browser. This placed the test emails in the appropriate email accounts. We then opened Outlook and clicked Send/Receive to execute the mail download. The download times were placed in a log file and subsequently moved to an Excel file to be averaged.
5.1.2600.2180
Wget
1.8.2
No version info
dslreports.com/stest
10
TEST RUN: 1
Testing Location: August 23, 2006 Westfield Mall Los Angeles, CA RSSI (signal strength): -70 DBM Details: Testing took place on the 3rd floor close to the railing and near the Food court. It was a large open space with tall ceilings. The server we connected to for the speed test was speakeasy.net.
Web-based Speed Test Std Std Downlink Uplink Dev Dev Dell HP Lenovo Kilobits Seconds Kilobits Seconds Kilobits Seconds 1,078 N/A 991 N/A 950 N/A 154 N/A 53 N/A 45 N/A 130 N/A 128 N/A 128 N/A 3 N/A 3 N/A 3 N/A FTP Put 1MB Dell HP Lenovo Kilobits Seconds Kilobits Seconds Kilobits Seconds 141 63 139 64 139 64 Std Dev 1 0 1 0 0 0 Put 2MB 134 130 133 131 133 131 Std Dev 0 0 1 1 0 0 Get 1MB 1,580 6 712 12 689 13 Std Dev 21 0 19 0 3 0 Get 2MB 1,633 11 719 24 721 24 Std Dev 53 0 21 1 3 1 HTTP Download Std 2MB Dev File 67 1 22 1 28 1 1,515 12 718 24 697 25
Std Dev 93 1 28 1 40 1
500K File Dell HP Lenovo Kilobits Seconds Kilobits Seconds Kilobits Seconds 1,052 4 468 9 468 9
Email Download (Kbps is an estimate) Std 1MB Std 2MB Dev File Dev File 0 0 52 1 52 1 1,274 7 524 17 524 17 182 1 31 1 31 1 1,158 15 511 34 457 38
Std Dev 77 1 30 2 24 2
11
TEST RUN: 2
Testing Location: August 24, 2006 Westfield Mall Los Angeles, CA RSSI (signal strength): -70 DBM Details: Testing took place, again, on the 3rd floor close to the railing and near the Food court. It was a large open space with tall ceilings. The server we connected to for the speed test was speakeasy.net.
Web-based Speed Test Std Std Downlink Dev Uplink Dev Dell HP Lenovo Kilobits Seconds Kilobits Seconds Kilobits Seconds 1,458 N/A 964 N/A 968 N/A 98 N/A 48 N/A 50 N/A 127 N/A 126 N/A 127 N/A 5 N/A 3 N/A 5 N/A FTP Put 1MB Dell HP Lenovo Kilobits Seconds Kilobits Seconds Kilobits Seconds 139 64 139 64 140 63 Std Dev 1 0 0 0 0 0 Put 2MB 133 131 133 131 132 132 Std Dev 0 0 0 0 0 1 Get 1MB 1,503 6 709 13 677 13 Std Dev 52 0 20 0 3 0 Get 2MB 1,578 11 703 25 702 25 Std Dev 98 1 12 0 3 1 HTTP Download Std 2MB Dev File 66 1 13 1 20 1 1,437 12 754 23 743 23
500K File Dell HP Lenovo Kilobits Seconds Kilobits Seconds Kilobits Seconds 1,052 4 526 8 421 10
Email Download (Kbps is an estimate) Std 1MB Std 2MB Dev File Dev File 263 1 66 1 42 1 1,115 8 524 17 495 18 139 1 0 0 28 1 1,158 15 543 32 434 40
Std Dev 77 1 17 1 22 2
12
TEST RUN: 3
Testing Location: August 24, 2006 Starbucks Los Angeles, CA RSSI (signal strength): -55 DBM Details: Testing took place near a window; it was next to a large parking lot on both sides with relatively no large buildings. We were unable to get any speed test results in this test run due to time constraints. Web-based Speed Test Std Std Downlink Dev Uplink Dev
Dell HP Lenovo Kilobits Seconds Kilobits Seconds Kilobits Seconds
HTTP Download Std 2MB Dev File 129 927 2 19 62 649 1 27 43 653 1 27
Std Dev 85 2 18 1 64 3
Dell HP Lenovo
Std Dev 1 1 0 0 1 5
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 526 66 743 0 827 158 8 1 12 0 21 4 526 0 495 28 482 27 8 0 18 1 36 2 421 42 469 25 424 21 10 1 19 1 41 2
13
TEST RUN: 4
Testing Location: August 25, 2006 Starbucks Los Angeles, CA RSSI (signal strength): -55 DBM Details: Testing took place near a window; it was next to a large parking lot on both sides with relatively no large buildings. The server we connected to for the speed test was linkline.net. Web-based Speed Test Std Std Downlink Uplink Dev Dev 979 165 128 3 N/A N/A N/A N/A 944 86 128 3 N/A N/A N/A N/A 702 69 123 6 N/A N/A N/A N/A HTTP Download Std 2MB Dev File 159 945 2 19 66 669 1 26 60 659 1 26
Dell HP Lenovo
Dell HP Lenovo
Std Dev 1 0 1 0 0 1
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 1,052 263 892 267 827 118 4 1 10 3 21 3 526 66 524 31 543 17 8 1 17 1 32 1 468 52 425 20 354 51 9 1 21 1 49 7
14
TEST RUN: 5
Testing Location: September 6, 2006 Springhill Suites Marriott Chicago, IL RSSI (signal strength): -68 DBM Details: Testing took place near a window on the 6th floor, near the freeway with no other surrounding large buildings. The server we connected to for the speed test was BroadStar.com. Web-based Speed Test Std Std Downlink Uplink Dev Dev 415 129 125 3 N/A N/A N/A N/A 263 26 125 4 N/A N/A N/A N/A 262 43 123 4 N/A N/A N/A N/A FTP Std Dev 3 3 2 2 0 2 HTTP Download Std 2MB Dev File 63 708 1 25 14 462 1 37 24 404 1 43
Dell HP Lenovo
Std Dev 77 3 14 1 30 3
Dell HP Lenovo
Std Dev 2 1 2 1 0 1
Std Dev 88 3 21 2 3 2
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 526 66 524 93 511 75 8 1 17 3 34 5 324 25 343 13 347 14 13 1 26 1 50 2 248 29 270 25 255 23 17 2 33 3 68 6
15
TEST RUN: 6
Testing Location: September 7, 2006 McDonalds Downtown Chicago, IL RSSI (signal strength): -52 DBM Details: Testing took place upstairs on the 2nd floor, centrally located downtown and surrounded by large buildings. The server we connected to for the speed test was BroadStar.com. Web-based Speed Test Std Std Downlink Uplink Dev Dev 420 141 86 16 N/A N/A N/A N/A 259 26 80 9 N/A N/A N/A N/A 279 30 87 10 N/A N/A N/A N/A FTP Std Dev 13 21 9 19 1 25 HTTP Download Std 2MB Dev File 73 838 1 21 12 444 1 39 13 461 1 38
Dell HP Lenovo
Std Dev 84 2 17 2 16 1
Dell HP Lenovo
Std Dev 20 17 11 11 1 10
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 601 86 637 136 695 83 7 1 14 3 25 3 351 0 343 13 347 14 12 0 26 1 50 2 324 25 330 24 316 17 13 1 27 2 55 3
16
TEST RUN: 7
Testing Location: September 8, 2006 Springhill Suites Marriott Chicago, IL RSSI (signal strength): -68 DBM Details: Testing took place near a window on the 6th floor, near the freeway with no other surrounding large buildings. The server we connected to for the speed test was BroadStar.com. Web-based Speed Test Std Std Downlink Uplink Dev Dev 353 96 121 4 N/A N/A N/A N/A 238 27 121 3 N/A N/A N/A N/A 251 30 120 6 N/A N/A N/A N/A FTP Std Dev 2 2 2 2 0 4 HTTP Download Std 2MB Dev File 90 518 3 35 48 347 4 51 43 360 3 49
Dell HP Lenovo
Dell HP Lenovo
Std Dev 3 1 4 2 0 1
Std Dev 91 3 18 1 2 1
Std Dev 82 6 13 2 2 2
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 526 0 557 35 579 19 8 0 16 1 30 1 281 19 279 9 280 14 15 1 32 1 62 3 351 29 343 0 328 12 12 1 26 0 53 2
17
TEST RUN: 8
Testing Location: September 7, 2006 McDonalds Downtown Chicago, IL RSSI (signal strength): -52 DBM Details: Testing took place upstairs on the 2nd floor, centrally located downtown and surrounded by large buildings. The server we connected to for the speed test was BroadStar.com. Web-based Speed Test Std Std Downlink Uplink Dev Dev 337 75 115 12 N/A N/A N/A N/A 275 36 108 14 N/A N/A N/A N/A 271 46 111 8 N/A N/A N/A N/A FTP Std Dev 4 4 1 1 0 1 HTTP Download Std 2MB Dev File 63 802 1 22 15 448 1 38 23 455 1 38
Dell HP Lenovo
Std Dev 84 2 18 2 13 1
Dell HP Lenovo
Std Dev 3 1 1 0 0 1
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 601 86 686 53 695 56 7 1 13 1 25 2 383 35 357 14 354 14 11 1 25 1 49 2 324 25 343 13 328 12 13 1 26 1 53 2
18
TEST RUN: 9
Testing Location: September 10, 2006 Days Inn New York, NY RSSI (signal strength): -55 DBM Details: Testing took place on the 7th floor not near a window. Location was surrounded by large buildings. The server we connected to for the speed test was BroadStar.com. Web-based Speed Test Std Std Downlink Uplink Dev Dev 376 124 124 3 N/A N/A N/A N/A 212 42 122 4 N/A N/A N/A N/A 214 50 118 9 N/A N/A N/A N/A FTP Std Dev 1 1 1 1 0 1 HTTP Download Std 2MB Dev File 108 871 2 20 39 474 2 36 72 517 5 34
Dell HP Lenovo
Dell HP Lenovo
Std Dev 2 1 2 1 0 1
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 701 117 743 0 620 421 6 1 12 0 28 19 383 35 388 17 395 9 11 1 23 1 44 1 383 35 388 17 378 8 11 1 23 1 46 1
19
TEST RUN: 10
Testing Location: September 11, 2006 Days Inn New York, NY RSSI (signal strength): -55 DBM Details: Testing took place on the 7th floor not near a window. Location was surrounded by large buildings. The server we connected to for the speed test was BroadStar.com. Web-based Speed Test Std Std Downlink Uplink Dev Dev 326 40 126 3 N/A N/A N/A N/A 329 50 125 5 N/A N/A N/A N/A 343 48 118 23 N/A N/A N/A N/A FTP Std Dev 2 2 3 3 1 5 HTTP Download Std 2MB Dev File 50 982 1 18 43 513 2 34 128 479 13 42
Dell HP Lenovo
Dell HP Lenovo
Std Dev 1 1 7 4 1 2
Std Dev 30 0 10 0 8 3
Std Dev 62 1 14 1 3 2
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 701 234 743 62 724 60 6 2 12 1 24 2 383 70 343 66 328 105 11 2 26 5 53 17 421 42 405 18 395 27 10 1 22 1 44 3
20
TEST RUN: 11
Testing Location: September 12, 2006 Starbucks New York, NY RSSI (signal strength): -38 DBM Details: Testing took place on the main level and not near a window. Location was just above the subway and surrounded by taller buildings. The server we connected to for the speed test was linkline.net. Web-based Speed Test Std Std Downlink Dev Uplink Dev 775 88 124 2 N/A N/A N/A N/A 486 22 124 3 N/A N/A N/A N/A 486 59 123 2 N/A N/A N/A N/A FTP Std Dev 1 1 1 1 0 1 HTTP Download Std 2MB Dev File 57 870 1 20 38 472 1 37 25 515 1 33
Dell HP Lenovo
Std Dev 81 2 29 2 22 2
Dell HP Lenovo
Std Dev 1 0 2 1 0 0
Std Dev 72 2 31 3 4 2
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 601 86 686 53 668 51 7 1 13 1 26 2 324 25 330 24 334 32 13 1 27 2 52 5 383 35 372 15 370 24 11 1 24 1 47 3
21
TEST RUN: 12
Testing Location: September 12, 2006 Starbucks New York, NY RSSI (signal strength): -38 DBM Details: Testing took place on the main level and not near a window. Location was just above the subway and surrounded by taller buildings. The server we connected to for the speed test was linkline.net. Web-based Speed Test Std Std Downlink Dev Uplink Dev 905 140 125 2 N/A N/A N/A N/A 483 4 124 1 N/A N/A N/A N/A 507 16 123 2 N/A N/A N/A N/A FTP Std Dev 1 1 1 1 0 1 HTTP Download Std 2MB Dev File 31 990 0 17 19 484 1 35 6 520 1 33
Dell HP Lenovo
Std Dev 58 1 31 2 5 0
Dell HP Lenovo
Std Dev 3 2 1 1 0 0
Std Dev 48 1 20 1 1 0
Std Dev 42 1 15 1 1 1
Dell HP Lenovo
Email Download (Kbps is an estimate) 500K Std 1MB Std 2MB Std File Dev File Dev File Dev 701 117 743 62 789 36 6 1 12 1 22 1 383 35 357 29 370 16 11 1 25 2 47 2 421 42 388 17 395 0 10 1 23 1 44 0
22
All references, in part or in full, to test results presented in this report must directly reference or link to this complete report. All references should be made to this report which will be posted in the competitive analysis test section of the Lionbridge web site.
23