tencent cloud

Tencent Kubernetes Engine

문서Tencent Kubernetes Engine

Buffer Operator Description

다운로드
포커스 모드
폰트 크기
마지막 업데이트 시간: 2026-10-23 16:19:17
AI 번역

Product Overview

What Is Buffer Operator

Buffer Operator is a Kubernetes cluster resource watermark management component provided by Tencent Cloud TKE. It proactively reserves cluster resources by creating low-priority placeholder Pods and works with Cluster Autoscaler to achieve scale-out in advance, allowing business Pods to preempt resources and start within seconds when resources are tight.
Buffer Operator has the following features:
Resource Reservation: Maintain the specified resource watermark by configuring the number of Buffer Pods.
Node-Level Buffer: A zero-resource NodeAnchor Pod is deployed on each node, and the Pod Anti-Affinity mechanism is used to pin the node, preventing Cluster Autoscaler from scaling in empty nodes when the resource watermark is low.
Passive Scale-Out Trigger: When business Pods are scheduled, they preempt low-priority placeholder Pods and immediately obtain the required resources.
Dynamic Adjustment: Periodically monitors available cluster resources and automatically scales the number of Placeholder Pods.
Multi-Cloud Support: Compatible with Tencent Cloud TKE, Alibaba Cloud ACK, AWS EKS, Azure AKS, and self-managed Kubernetes clusters.

Why Buffer Operator

With Buffer Operator, you can obtain the following benefits:
Proactive Scale-Out: Avoids triggering scale-out only when resources are insufficient, reducing scheduling latency for business Pods.
Resource Guarantee: Flexibly control the amount of reserved resources through configurable watermark parameters to ensure that the cluster always has sufficient resources to handle sudden business demands.
Automated Management: Automatically calculates cluster resource usage and dynamically adjusts placeholder Pods without manual intervention.
Observability: Built-in Prometheus Metrics and OpenTelemetry Traces support comprehensive monitoring and alarm.
For more reasons to choose this product, see Product Advantages and Application Scenarios.

Product Architecture

Buffer Operator mainly consists of the following core components:
Resource Monitor: Monitors resource changes of Nodes and Pods, caches resource information, and reduces the pressure on the API Server.
Resource Calculator: Calculates the total and available cluster resources, and calculates the amount of resources to reserve based on the watermark configuration.
Placeholder Manager: Creates and deletes placeholder Pods, manages Pod affinity and anti-affinity rules, and controls the scaling frequency.
BufferConfig Controller: Listens for BufferConfig CRD changes, coordinates the work of various components, and updates the BufferConfig status.
The following figure shows the architecture diagram.
┌─────────────────────────────────────────────────────┐
│                  Buffer Operator                     │
├─────────────────┬──────────────┬────────────────────┤
│ ResourceMonitor │  Calculator  │ PlaceholderManager │
│ (internal/      │ (internal/   │ (internal/         │
│  monitor)       │  calculator) │  manager)          │
├─────────────────┼──────────────┼────────────────────┤
│ Monitor Node/Pod   │ Calculate required    │ Build/Scale         │
│ Resources, node pools      │ buffer-pod   │ buffer-pod /        │
│                 │ / NodeAnchor │ NodeAnchor          │
│                 │ Quantity          │ Pod (OwnerRef +    │
│                 │              │ Anti-Affinity)      │
└─────────────────┴──────────────┴────────────────────┘
          ↓                ↓                ↓
┌─────────────────────────────────────────────────────┐
│              Kubernetes Cluster                      │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐           │
│  │NodeAnchor│  │BufferPod │  │ Business │           │
│  │(zero res)│  │(low prio)│  │ Pod (high)│ ──preempt│
│  └──────────┘  └──────────┘  └──────────┘           │
└─────────────────────────────────────────────────────┘

Basic Concepts

Concept
Description
BufferConfig
Custom resource (CRD) that defines the behavior configuration of the cluster resource buffer pool.
Buffer Pod
A placeholder Pod that actually occupies resources, used to reserve CPU and memory resources.
NodeAnchor Pod
A zero-resource placeholder Pod, one per node, used to support node-level watermark management.
Placeholder Pod
A collective term for placeholder Pods, including Buffer Pods and NodeAnchor Pods.
PriorityClass
PriorityClass, used to distinguish placeholder Pods (low priority) from business Pods (high priority).

Related Products

Related Products
Relationship Description
TKE cluster
Buffer Operator runs in a TKE cluster and manages the cluster resource level.
Cluster Autoscaler
Works with Cluster Autoscaler to implement automatic scaling.
PriorityClass
Relies on PriorityClass to implement the Pod priority preemption mechanism.

Strengths

Active Scale-Out

Buffer Operator proactively triggers Cluster Autoscaler scale-out by creating placeholder Pods in advance, preventing Pod scheduling failures caused by insufficient resources during peak business periods. Compared with passive scale-out, it reduces the startup time of business Pods from minutes to seconds.

Flexible Configuration

Multiple configuration methods are supported:
Fixed Quantity Mode: Maintains a specified number of Buffer Pods.
Automatic Mode: Automatically calculates the number of Buffer Pods based on cluster resource usage.
Node Pool-Level Configuration: Configure buffer policies separately for each node pool.

Automated Management

The component automatically performs the following tasks:
Periodically monitor cluster resource usage.
Automatically scale placeholder Pods based on the configured water level.
Handle abnormal conditions of placeholder Pods (such as a Pod being accidentally deleted).

Use Cases

Burst Traffic Scenarios

Target Users: Users whose business traffic fluctuates significantly, such as e-commerce, live streaming, and gaming.
Pain Points and Requirements: Business traffic exhibits distinct peak and trough patterns. During peak periods, rapid scale-out is required, while traditional on-demand scaling has high latency.
Solution: Buffer Operator reserves resource headroom so that business Pods can preempt resources within seconds during peak periods, ensuring business continuity.

HA Scenarios

Target Users: Users who have high requirements for service availability.
Pain Points and Requirements: It is necessary to ensure that the cluster always has sufficient resources for fault recovery and disaster recovery switchover.
Solution: Configure NodeAnchor to prevent nodes from being scaled in unexpectedly, and reserve Buffer Pods as a resource buffer to improve cluster fault tolerance.

Cost Optimization Scenarios

Target Users: Users who need to optimize resource costs while ensuring service quality.
Pain Points and Requirements: Users want to scale in nodes during off-peak hours to save costs, but worry about insufficient resources during peak hours.
Solution: Configure a reasonable resource watermark to ensure sufficient resources for business needs while avoiding resource waste caused by excessive reservation.

Prerequisites

Before using Buffer Operator, ensure that the following conditions are met:
A Kubernetes cluster has been deployed, with version 1.24 or later.
Cluster Autoscaler has been installed.
A PriorityClass has been created to distinguish placeholder Pods from business Pods.
RBAC access control has been enabled for the cluster.

Quick Start

Create a BufferConfig CR to define the resource buffering policy:
apiVersion: buffer.tke.cloud.tencent.com/v1alpha1
kind: BufferConfig
metadata:
  name: default-buffer
  namespace: default
spec:
  bufferPods:
    number: 2
    strategy: fixed
  nodeBuffer:
    strategy: node-only
    count:
      type: absolute
      value: 0
  placeholderPod:
    priorityClassName: buffer-priority-low
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
    bufferPod:
      priorityClassName: buffer-priority-low
      resources:
        requests:
          cpu: "100m"
          memory: "128Mi"
  control:
    reconcileInterval: 30
    scaleUpCooldown: 300
    scaleDownCooldown: 600
    enableNodeAnchor: true
Default low-priority PriorityClass (used by placeholder Pods):
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: buffer-priority-low
value: -10
globalDefault: false
description: "Low priority, used for placeholder Pods and can be preempted by business Pods"
Default high-priority PriorityClass (used by business Pods):
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-priority-high
value: 1000
globalDefault: false
description: "High priority, used for business Pods and can preempt placeholder Pods"

BufferConfig Configuration

BufferConfig is the core custom resource of Buffer Operator, used to define the behavior of the cluster resource buffer pool.

CR Structure

The API version of BufferConfig is buffer.tke.cloud.tencent.com/v1alpha1, and the CRD name is BufferConfig.
apiVersion: buffer.tke.cloud.tencent.com/v1alpha1
kind: BufferConfig
metadata:
  name: buffer-config-name
  namespace: namespace-name
spec:
  # Buffer Pod Configuration
  bufferPods:
    number: 2
    strategy: fixed
  
  # Node-Level Buffer Configuration (Optional)
  nodeBuffer:
    strategy: node-only
    count:
      type: absolute
      value: 0
  
  # Placeholder Pod Template Configuration
  placeholderPod:
    image: registry.k8s.io/pause:3.9
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
    nodeAnchor:
      priorityClassName: buffer-priority-low
    bufferPod:
      resources:
        requests:
          cpu: "100m"
          memory: "128Mi"
  
  # Control Parameters (Optional)
  control:
    reconcileInterval: 30
    scaleUpCooldown: 300
    scaleDownCooldown: 600
    enableNodeAnchor: true

spec.bufferPods (Required)

Defines the number of Buffer Pods to maintain.
Parameter
Type
Required
Default Value
Value Range
Description
number
Integer
Yes
-
≥0
Number of Buffer Pods to maintain
strategy
String
No
fixed
fixed,auto
Buffer policy
Description of strategy values:
fixed: Fixed quantity mode, maintains a specified number of Buffer Pods.
auto: Automatic mode, automatically calculates the number of Buffer Pods based on cluster resource usage.
Configuration Example:
spec:
  bufferPods:
    number: 2
    strategy: fixed

spec.nodeBuffer (Optional)

Define the node-level buffer configuration.
Parameter
Type
Required
Default Value
Value Range
Description
strategy
String
No
both
both,resource-only,node-only
Buffer policy
count
NodeCountBufferSpec
Yes
-
-
Node buffer quantity configuration
nodePools
[]NodePoolBufferSpec
No
-
-
Configure buffer by node pool
Description of strategy values:
both: Enables both resource buffering and node buffering.
resource-only: Enables only resource buffering.
node-only: Enables only node buffering.
Description of the count field:
Parameter
Type
Required
Value Range
Description
type
String
Yes
percentage,absolute
Configuration Type
value
Integer
Yes
Percentage: 0-100; Absolute value: Number of nodes
Value
Configuration Example:
spec:
  nodeBuffer:
    strategy: node-only
    count:
      type: absolute
      value: 0

spec.placeholderPod (Required)

Define the configuration template for placeholder Pods.
Parameter
Type
Required
Default Value
Description
image
String
No
registry.k8s.io/pause:3.9
Container image
resources
ResourceRequirements
Yes
-
Resource requests of each placeholder Pod
priorityClassName
String
No
-
PriorityClass name
nodeSelector
map[string]string
No
-
Node selector
tolerations
[]Toleration
No
-
Tolerations
nodeAnchor
NodeAnchorPodSpec
No
-
NodeAnchor Pod configuration
bufferPod
BufferPodSpec
No
-
Buffer Pod configuration
Description of the nodeAnchor field:
A NodeAnchor Pod is a zero-resource placeholder Pod with one instance per node, used to support node-level watermark management. The Operator automatically adds the cluster-autoscaler.kubernetes.io/safe-to-evict=true annotation to NodeAnchor Pods, ensuring that Cluster Autoscaler can scale in nodes normally.
Parameter
Type
Required
Description
priorityClassName
String
No
PriorityClass name
nodeSelector
map[string]string
No
Node selector
tolerations
[]Toleration
No
Tolerations
Description of the bufferPod field:
A BufferPod is a buffer Pod that actually occupies resources.
Parameter
Type
Required
Description
resources
ResourceRequirements
Yes
Resource requests of each Buffer Pod
priorityClassName
String
No
PriorityClass name
nodeSelector
map[string]string
No
Node selector
tolerations
[]Toleration
No
Tolerations
Attention:
When the bufferPod field is not empty, internal/manager.buildBufferPod completely overrides the nodeSelector / tolerations of the top-level placeholderPod. Therefore, if you need to specify scheduling constraints, you must configure them again in bufferPod.
Configuration Example:
spec:
  placeholderPod:
    image: registry.k8s.io/pause:3.9
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
    priorityClassName: buffer-priority-low
    nodeSelector:
      node-pool: "default"
    tolerations:
      - key: "node.kubernetes.io/not-ready"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 300
    nodeAnchor:
      priorityClassName: buffer-priority-low
    bufferPod:
      resources:
        requests:
          cpu: "1000m"
          memory: "2Gi"
        limits:
          cpu: "1000m"
          memory: "2Gi"
      priorityClassName: buffer-priority-low
      nodeSelector:
        node-pool: "default"

spec.control (Optional)

Define control parameters.
Parameter
Type
Required
Default Value
Value Range
Description
reconcileInterval
Integer
No
30
≥10
Reconciliation interval (in seconds)
scaleUpCooldown
Integer
No
300
≥0
Scale-out cooldown period (in seconds)
scaleDownCooldown
Integer
No
600
≥0
Scale-in cooldown period (in seconds)
enableNodeAnchor
*bool
No
true
true,false
Whether to enable NodeAnchor
Attention:
enableNodeAnchor uses a pointer type (*bool) so that users can explicitly set it to false. If a regular bool + omitempty is used, the false value is discarded during JSON encoding, causing the apiserver to reapply the kubebuilder default value (true).
Configuration Example:
spec:
  control:
    reconcileInterval: 30
    scaleUpCooldown: 300
    scaleDownCooldown: 600
    enableNodeAnchor: true

spec.nodeBuffer.nodePools (Optional)

Configure buffer policies separately for each node pool. This is suitable for scenarios with multiple node pools where each node pool has different resource specifications.
Parameter
Type
Required
Description
name
String
Yes
Node pool name (matching node labels)
nodeSelector
map[string]string
Yes
Label selector used to identify node pools
count
NodeCountBufferSpec
Yes
Buffer configuration of the node pool
Configuration Example:
spec:
  bufferPods:
    number: 0
  nodeBuffer:
    strategy: both
    nodePools:
      - name: "cpu-pool"
        nodeSelector:
          node-pool: "cpu-pool"
        count:
          type: absolute
          value: 2
      - name: "memory-pool"
        nodeSelector:
          node-pool: "memory-pool"
        count:
          type: percentage
          value: 20

Configuration Scenario Examples

Scenario 1: Fixed Number of Buffer Pods

This is suitable for scenarios where you know exactly how many resources need to be reserved.
spec:
  bufferPods:
    number: 4
    strategy: fixed
  placeholderPod:
    bufferPod:
      resources:
        requests:
          cpu: "1000m"
          memory: "2Gi"

Scenario 2: Configuring Buffers by Node Pool

This is suitable for scenarios with multiple node pools where each node pool has different resource specifications.
spec:
  bufferPods:
    number: 0
  nodeBuffer:
    strategy: both
    nodePools:
      - name: "cpu-pool"
        nodeSelector:
          node-pool: "cpu-pool"
        count:
          type: absolute
          value: 2
      - name: "memory-pool"
        nodeSelector:
          node-pool: "memory-pool"
        count:
          type: percentage
          value: 20

Scenario 3: Enabling NodeAnchor Only (Zero Resource Placeholder)

This is suitable for scenarios where you only need to ensure that nodes are not scaled in by Cluster Autoscaler, without reserving actual resources.
spec:
  bufferPods:
    number: 0
  nodeBuffer:
    strategy: node-only
  control:
    enableNodeAnchor: true

Monitoring and Alarms

Buffer Operator exposes Prometheus metrics at the :8080/metrics endpoint and supports the following monitoring metrics:

Tuning-Related Metrics

Metric Name
Type
Description
buffer_operator_reconcile_duration_seconds
Histogram
Distribution of reconciliation duration
buffer_operator_reconcile_total
Counter
Total number of reconciliations
buffer_operator_reconcile_errors_total
Counter
Number of reconciliation errors

Resource-Related Metrics

Metric Name
Type
Description
buffer_operator_placeholder_pods_current
Gauge
Current number of placeholder Pods
buffer_operator_placeholder_pods_desired
Gauge
Desired number of placeholder Pods
buffer_operator_cluster_cpu_total_cores
Gauge
Total number of CPU cores in the cluster
buffer_operator_cluster_memory_total_bytes
Gauge
Total number of memory bytes in the cluster
buffer_operator_cluster_cpu_available_cores
Gauge
Available CPU cores
buffer_operator_cluster_memory_available_bytes
Gauge
Available memory in bytes

Recommended Alarm Rules

groups:
  - name: buffer-operator
    rules:
      # High Reconciliation Failure Rate
      - alert: BufferOperatorHighReconcileErrorRate
        expr: rate(buffer_operator_reconcile_errors_total[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High Buffer Operator reconciliation failure rate"
          
      # Placeholder Pod Count Deviation
      - alert: BufferOperatorPlaceholderMismatch
        expr: abs(buffer_operator_placeholder_pods_current - buffer_operator_placeholder_pods_desired) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Large deviation between the actual and expected number of placeholder Pods"

Pod Tag Conventions

All Placeholder Pods managed by the Operator carry the following labels:
Tag
Value
Purpose
buffer.tke.cloud.tencent.com/placeholder
"true"
Identify Placeholder pods
buffer.tke.cloud.tencent.com/placeholder-type
buffer-pod / node-anchor
Distinguish categories
buffer.tke.cloud.tencent.com/config
BufferConfig name
Query the owning CR

Use Limits

Configuration Limits

Limit
Description
reconcileInterval
Minimum value: 10 seconds.
scaleUpCooldown
No lower limit. Recommended value: 300 seconds (5 minutes).
scaleDownCooldown
No lower limit. Recommended value: 600 seconds (10 minutes).
When nodeBuffer.count.type is percentage
The value range is 0-100.

Environment Requirement

Limit
Description
Kubernetes Version
1.24 or later
Cluster Autoscaler
Must be deployed and running properly.
RBAC
Grant read permissions on nodes and pods, and create/delete permissions on pods.

FAQs

What Are the Differences Between Buffer Pods and NodeAnchor Pods?

Buffer Pod is a placeholder Pod that actually occupies resources. It is used to reserve CPU and memory resources for business Pods to preempt.
NodeAnchor Pod is a zero-resource placeholder Pod with one instance per node, used to support node-level watermark management and prevent Cluster Autoscaler from scaling in empty nodes at low watermark.

How to Configure Business Pods to Preempt Placeholder Pods?

Business Pods need to be configured with a high-priority PriorityClass (for example, business-priority-high with a value of 1000), while placeholder Pods use a low-priority PriorityClass (for example, buffer-priority-low with a value of -10). When cluster resources are insufficient, business Pods automatically preempt the resources of placeholder Pods.

Will Nodes Still Be Scaled In After enableNodeAnchor Is Set to false?

Yes. After enableNodeAnchor is set to false, the Operator will not create NodeAnchor Pods, nodes will lose placeholder protection, and Cluster Autoscaler may scale in nodes when resources are insufficient.

How to View the Running Status of BufferConfig?

Run the following command to view the BufferConfig status:
kubectl get bufferconfig <name> -n <namespace> -o yaml
Status information is contained in the status field, including:
conditions: BufferConfig status conditions
statistics: Resource statistics
lastScaleTime: The time of the last scaling operation
lastScaleDirection: The direction of the last scaling operation
다음 주제: TKE Serverless 클러스터 업데이트(2023년)

도움말 및 지원

문제 해결에 도움이 되었나요?

피드백

키워드로 검색
    Enter keywords to search documentation