Testing Tool

The performance test in this document is conducted with the TPC-C benchmark load built in HammerDB. TPC-C is a typical OLTP workload that simulates a scenario where a wholesaler with multiple warehouses ships goods to a large number of customers. The adjustment of the number of warehouses can reflect the data size that the database can sustain in the test.

• HammerDB download address
• HammerDB User Manual
• Overview of the TPC-C test load built in HammerDB

Test Environment and Parameters

Physical machines where instances are located
High-performance physical machines with a specification of 80 cores, 767 GB memory, and 30 TB storage capacity.

Test instance specifications
The test instances are of 2008 Enterprise Edition, 2012 Enterprise Edition, 2016 Enterprise Edition, and 2017 Enterprise Edition. They cover almost all purchasable specifications, including 1-core 2 GB, 1-core 4 GB, 1-core 8 GB, 2-core 16 GB, 4-core 32 GB, 8-core 64 GB, 12-core 96 GB, 16-core 128 GB, and 32-core 256 GB.

Load generation environment
The machines on which HammerDB is installed are of the same models as the database instances, ensuring that the performance of the SQL Server instances can be fully measured in the stress test.

TPC-C benchmark parameters

• Number of Warehouses = 100: Sets the number of warehouses to 100, which will generate a test database of approximately 55 GB in size.
• Minutes of Rampup Time = 2: Sets the warm-up time before the test to 2 minutes.
• Minutes Test Duration = 5: Sets the test duration to 5 minutes.

Number of virtual users
The number of virtual users is the number of concurrent connections. In this document, different numbers of concurrent connections are tested on instances with different specifications.

Testing Method

1. Prepare the TPC-C workload.
   • Number of Warehouses: The number of warehouses, which will affect the size of the test database generated.
   • Virtual Users to Build Schema: The number of concurrent connections when generating the load data, which cannot be larger than the number of warehouses. This value affects the efficiency of load data generation, so it is recommended to be the same as the number of CPU cores of the load generating device.
     
2. Set the test script.
   • Total Transactions per User: The total number of transactions per user. You are recommended to set this parameter to a higher value so as to ensure that the user will not exit due to the completion of transactions during the stress test.
   • Minutes of Rampup Time: Warm-up time for the stress test.
   • Minutes for Test Duration: Duration of the stress test.
     
3. Set the automated test script.
   • Minutes per Test in Virtual User Sequence: The interval between two automated test sessions during which the program completes various tasks such as creating virtual users, warming up, running the test, and stopping the test. This value should be greater than that of “Minutes of Rampup Time and Minutes for Test Duration”.
   • Active Virtual User Sequence (Space Separated): The number of virtual users generated by each iteration of the automated test. It can be understood as the number of concurrent connections.
     
4. Select Autopilot > Autopilot in the left pane to start the test.
   
5. The test result will be output in the hammerdb.log file.
   

Test Results

The TPM in HammerDB is obtained through the SQL Server performance counter "batch requests/sec", so the TPM actually refers to the batch requests per minute.

• SQL Server 2008 Enterprise Edition

• SQL Server 2012 Enterprise Edition

• SQL Server 2016 Enterprise Edition

• SQL Server 2017 Enterprise Edition