This document describes how to improve the Prompt Cache hit rate of the TokenHub platform through proper request design, thereby reducing the time to first Token (TTFT) and inference costs.
I. Using prompt_cache_key
1.1 What Is prompt_cache_key
prompt_cache_key is a request-level cache identifier field that tells the caching system which requests share the same prefix and can reuse the KV Cache.
Its core principle is: assign the overall context ID (conversation_id) instead of a single session ID (session_id).
1.2 Usage
Add the prompt_cache_key field to the request body:
{
"model": "your-model",
"prompt_cache_key": "conv-6900xxxx",
"messages": [
{"role": "system", "content": "You are an assistant..."},
{"role": "user", "content": "Hello"}
]
}
1.3 Best Practices
All requests within the same conversation context use the same prompt_cache_key.
Use different prompt_cache_key values for different conversations to avoid cache pollution.
It is recommended to use the business-side conversation_id or an equivalent globally unique identifier as the value.
II. Using X-Session-ID
2.1 What Is X-Session-ID
X-Session-ID is a session identifier passed through the HTTP Header, used to route consecutive requests from the same user to the same inference instance, thereby improving the local KV Cache hit rate on that instance.
2.2 Usage
Add the following to the request Header:
X-Session-ID: session-abc123
Complete request example:
curl -X POST 'https://tokenhub-intl.tencentcloudmaas.com/v1/chat/completions' \\
-H 'Content-Type: application/json' \\
-H 'Authorization: Bearer your-api-key' \\
-H 'X-Session-ID: session-abc123' \\
-d '{
"model": "your-model",
"messages": [...]
}'
2.3 Best Practices
Keep X-Session-ID unchanged across multiple turns of conversation for the same user.
Use different Session IDs for different users or conversation contexts.
It works best when used together with prompt_cache_key.
III. Cache TTL Mechanism
3.1 What Is Cache TTL
TTL (Time To Live) is the lifetime of cached data. After the TTL expires, the cached KV data is evicted, and subsequent requests require recalculation.
3.2 Impact of TTL on Hit Ratio
Within the TTL validity period, requests with the same prefix can directly reuse cached KV data. After the TTL expires, recalculation is required even if the prefix remains the same, causing TTFT to rise again. Users can optimize request design to prevent the cache from being invalidated unexpectedly and prematurely, thereby maximizing hit benefits within the TTL validity period. Key focus: Keep the request prefix stable and avoid frequent changes to the content within the prefix.
3.3 Notes
Avoid Writing Time-Related Content in System Prompts
Note:
Do not include dynamic content such as the current date or time in the System Prompt.
Reason: Time changes cause the System Prompt content to change, which leads to prefix mismatch and complete cache invalidation. For example, when the date changes after midnight, all caches become invalid instantly, causing TTFT to surge and resulting in a noticeably sluggish user experience.
Correct approach:
{
"messages": [
{"role": "system", "content": "Today is May 9, 2026. You are an assistant..."}
]
}
{
"messages": [
{"role": "system", "content": "You are an assistant..."},
{"role": "user", "content": "Today is May 9, 2026. Please help me..."}
]
}
Place dynamic content at the end of the messages (in the user message) rather than in the system prompt, so that prefix caching is not affected.
IV. Request Structure Design Principles
A well-designed request structure is the foundation for improving cache hit rates.
4.1 Core Principles
Key unchanged: Keep the role of each message in messages stable.
Number of keys unchanged: Keep the message quantity structure consistent.
Key order unchanged: Keep the message order consistent.
4.2 Principles for Message Structure Changes
Append keys at the end: New conversation turns are appended only at the end of the messages array. Do not insert or modify existing messages in the middle.
{
"messages": [
{"role": "system", "content": "You are an assistant"},
{"role": "user", "content": "Question 1"}
]
}
{
"messages": [
{"role": "system", "content": "You are an assistant"},
{"role": "user", "content": "Question 1"},
{"role": "assistant", "content": "Answer 1"},
{"role": "user", "content": "Question 2"}
]
}
In this way, the KV Cache of the first two messages can be fully reused.
V. Recommendations for New Version Releases
During product release updates, if System Prompt changes are involved, large-scale cache invalidation may occur. Recommendations:
1. Pre-warm the cache before release: Use a small amount of simulated session data to access the API and build the KV Cache in advance.
2. Avoid traffic spikes: Use canary releases to prevent a sudden drop in Cache Rate caused by a release and a traffic spike occurring at the same time.
3. Monitor Cache Rate: Closely monitor the cache hit rate metric after release, and promptly investigate any abnormal drops.
VI. Summary
|
prompt_cache_key
| Identify identical contexts to improve cache reuse. | Request body field, with the value conversation_id |
X-Session-ID
| Route to the same instance to improve local cache hits. | HTTP Header |
Stable System Prompt | Prevent cache invalidation caused by prefix changes. | Do not write dynamic time content. |
Append Messages at the End. | Maintain prefix consistency. | Append messages only at the end. |
Pre-release Warm-up | Prevent cold start impact. | Build KV by simulating requests in advance. |