Performance Test Report

Last updated: 2019-11-06 16:26:18

PDF

Testing Tool

The performance test in this document is conducted with the TPC-C benchmark load built in HammerDB. TPC-C is a typical OLTP workload that simulates a scenario where a wholesaler with multiple warehouses ships goods to a large number of customers. The adjustment of the number of warehouses can reflect the data size that the database can sustain in the test.

Test Environment and Parameters

Physical machines where instances are located
High-performance physical machines with a specification of 80 cores, 767 GB memory, and 30 TB storage capacity.

Test instance specifications
The test instances are of 2008 Enterprise Edition, 2012 Enterprise Edition, 2016 Enterprise Edition, and 2017 Enterprise Edition. They cover almost all purchasable specifications, including 1-core 2 GB, 1-core 4 GB, 1-core 8 GB, 2-core 16 GB, 4-core 32 GB, 8-core 64 GB, 12-core 96 GB, 16-core 128 GB, and 32-core 256 GB.

Load generation environment
The machines on which HammerDB is installed are of the same models as the database instances, ensuring that the performance of the SQL Server instances can be fully measured in the stress test.

TPC-C benchmark parameters

  • Number of Warehouses = 100: Sets the number of warehouses to 100, which will generate a test database of approximately 55 GB in size.
  • Minutes of Rampup Time = 2: Sets the warm-up time before the test to 2 minutes.
  • Minutes Test Duration = 5: Sets the test duration to 5 minutes.

Number of virtual users
The number of virtual users is the number of concurrent connections. In this document, different numbers of concurrent connections are tested on instances with different specifications.

Concurrent Connections 2 4 8 16 32 64 128 256 512 1,024
1-core 2 GB - -
1-core 4 GB - -
1-core 8 GB - -
2-core 16 GB -
4-core 32 GB -
8-core 64 GB -
12-core 96 GB
16-core 128 GB
32-core 256 GB

Testing Method

  1. Prepare the TPC-C workload.
    • Number of Warehouses: The number of warehouses, which will affect the size of the test database generated.
    • Virtual Users to Build Schema: The number of concurrent connections when generating the load data, which cannot be larger than the number of warehouses. This value affects the efficiency of load data generation, so it is recommended to be the same as the number of CPU cores of the load generating device.
  2. Set the test script.
    • Total Transactions per User: The total number of transactions per user. You are recommended to set this parameter to a higher value so as to ensure that the user will not exit due to the completion of transactions during the stress test.
    • Minutes of Rampup Time: Warm-up time for the stress test.
    • Minutes for Test Duration: Duration of the stress test.
  3. Set the automated test script.
    • Minutes per Test in Virtual User Sequence: The interval between two automated test sessions during which the program completes various tasks such as creating virtual users, warming up, running the test, and stopping the test. This value should be greater than that of “Minutes of Rampup Time and Minutes for Test Duration”.
    • Active Virtual User Sequence (Space Separated): The number of virtual users generated by each iteration of the automated test. It can be understood as the number of concurrent connections.
  4. Select Autopilot > Autopilot in the left pane to start the test.
  5. The test result will be output in the hammerdb.log file.

Test Results

The TPM in HammerDB is obtained through the SQL Server performance counter "batch requests/sec", so the TPM actually refers to the batch requests per minute.

  • SQL Server 2008 Enterprise Edition
Instance Specification Storage Capacity Data Set Concurrent Connections TPM
1-core 2 GB 300 GB 55 GB 256 279,798
1-core 4 GB 300 GB 55 GB 256 284,680
1-core 8 GB 300 GB 55 GB 256 269,039
2-core 16 GB 800 GB 55 GB 256 368,366
4-core 32 GB 800 GB 55 GB 256 657,641
8-core 64 GB 1,500 GB 55 GB 256 1,164,062
12-core 96 GB 1,500 GB 55 GB 1024 1,348,121
16-core 128 GB 2,000 GB 55 GB 1024 1,357,678
32-core 256 GB 3,000 GB 55 GB 1024 1,401,600
  • SQL Server 2012 Enterprise Edition
Instance Specification Storage Capacity Data Set Concurrent Connections TPM
1-core 2 GB 300 GB 55 GB 256 229,854
1-core 4 GB 300 GB 55 GB 256 234,401
1-core 8 GB 300 GB 55 GB 256 236,773
2-core 16 GB 800 GB 55 GB 256 333,797
4-core 32 GB 800 GB 55 GB 256 608,801
8-core 64 GB 1,500 GB 55 GB 256 1,020,500
12-core 96 GB 1,500 GB 55 GB 1024 1,266,868
16-core 128 GB 2,000 GB 55 GB 1024 1,385,158
32-core 256 GB 3,000 GB 55 GB 1024 1,526,762
  • SQL Server 2016 Enterprise Edition
Instance Specification Storage Capacity Data Set Concurrent Connections TPM
1-core 2 GB 300 GB 55 GB 256 219,142
1-core 4 GB 300 GB 55 GB 256 222,796
1-core 8 GB 300 GB 55 GB 256 219,676
2-core 16 GB 800 GB 55 GB 256 336,843
4-core 32 GB 800 GB 55 GB 256 665,065
8-core 64 GB 1,500 GB 55 GB 256 1,070,826
12-core 96 GB 1,500 GB 55 GB 1024 1,337,473
16-core 128 GB 2,000 GB 55 GB 1024 1,705,660
32-core 256 GB 3,000 GB 55 GB 1024 2,280,252
  • SQL Server 2017 Enterprise Edition
Instance Specification Storage Capacity Data Set Concurrent Connections TPM
1-core 2 GB 300 GB 55 GB 256 201,851
1-core 4 GB 300 GB 55 GB 256 202,510
1-core 8 GB 300 GB 55 GB 256 208,685
2-core 16 GB 800 GB 55 GB 256 331,650
4-core 32 GB 800 GB 55 GB 256 625,370
8-core 64 GB 1,500 GB 55 GB 256 1,102,296
12-core 96 GB 1,500 GB 55 GB 1024 1,325,010
16-core 128 GB 2,000 GB 55 GB 1024 1,716,818
32-core 256 GB 3,000 GB 55 GB 1024 2,520,856