This document describes how to host a Spring Cloud application to TKE.
Hosting Spring Cloud applications to TKE has the following advantages:
This document describes how to host a Spring Cloud application to TKE by forking the open-source PiggyMetrics on GitHub and adapting it to Tencent Cloud products.
The PiggyMetrics homepage is as shown below:
PiggyMetrics is a microservice-architecture application for personal finances developed by using the Spring Cloud framework.
PiggyMetrics consists of the following microservices:
|API gateway||It's a Spring Cloud Zuul-based gateway and the aggregated portal for calling backend APIs, providing reverse routing and load balancing (Eureka + Ribbon) as well as rate limiting (Hystrix). Client single-page applications and the Zuul gateway are deployed together to simplify deployment.|
|Service registration and discovery||A Spring Cloud Eureka registry. Business services are registered through Eureka when they are enabled, and service discovery is performed through Eureka when services are called.|
|Authorization and authentication service||An authorization and authentication center based on Spring Security OAuth2. The client gets the access token through the Auth Service during logins, and so does service call. Each resource server verifies the token through the Auth Service.|
|Configuration service||A configuration center based on Spring Cloud Config to centrally manage configuration files for all Spring services.|
|Soft loading and rate limiting||Ribbon and Hystrix based on Spring Cloud. Zuul calls backend services through Ribbon for soft loading and Hystrix for rate limiting.|
|Metrics and dashboard||Hystrix Dashboard based on Spring Cloud Turbine, aggregating all the PiggyMetrics streams generated by Hystrix and displaying them on the Hystrix Dashboard.|
In the following best practice, applications deployed in CVM are containerized and hosted to TKE. In this use case, one VPC is used and divided into two subnets:
The VPC is divided as shown below:
The network planning for the PiggyMetrics instance is as shown below:
|Region/AZ||Nanjing/Nanjing Zone 1|
|Subnet-Basic||Nanjing Zone 1, CIDR block: 10.0.1.0/24|
|Subnet-K8S||Nanjing Zone 1, CIDR block: 10.0.2.0/24|
|Nacos cluster||Nacos cluster built with three 1-core 2 GB MEM Standard SA2 CVM instances with IP addresses of 10.0.1.9, 10.0.1.14, and 10.0.1.15|
The add-ons used in the PiggyMetrics instance are as shown below:
|K8S||1.8.4||Tencent Cloud||TKE management mode|
|MongoDB||4.0||Tencent Cloud||TencentDB for MongoDB WiredTiger engine|
|CLS||N/A||Tencent Cloud||Log service|
|TSW||N/A||Tencent Cloud||Accessed with SkyWalking 8.4.0 Agent, which can be downloaded here|
|Java||1.8||Open-source community||Docker image of Java 8 JRE|
|Spring Cloud||Finchley.RELEASE||Open-source community||Spring Cloud website|
Tencent Cloud Tencent Container Registry (TCR) are available in Personal Edition and Enterprise Edition as differentiated below:
PiggyMetrics is a Dubbo containerized demo project, so TCR Personal Edition perfectly meets its needs. However, for enterprise users, TCR Enterprise Edition is recommended. To use an image repository, see Basic Image Repository Operations.
Tencent Service Watcher (TSW) provides cloud-native service observability solutions that can trace upstream and downstream dependencies in distributed architectures, draw topologies, and provide multidimensional call observation by service, API, instance, and middleware.
TSW is architecturally divided into four modules:
You can use an open-source probe or SDK to collect data. If you are migrating to the cloud, you can change the reporting address and authentication information only and keep most of the configurations on the client.
Data is reported to the server via the Pulsar message queue, converted by the adapter into an OpenTracing-compatible format, and assigned to real-time and offline computing as needed.
The storage layer can adapt to use cases with different data types, writing at the server layer, and query and reading requests at the data usage layer.
The data usage layer provides underlying support for console operations, data display, and alarming.
The architecture is as shown below:
In the TencentDB for MongoDB console, create an instance and run the following command to initialize it:
# Download the MongoDB client, decompress it, and enter the `bin` directory.
tar -zxvf mongodb-linux-x86_64-3.6.18.tgz
# Run the following command to initialize MongoDB, where `mongouser` is the admin account created when the MongoDB instance is created.
./mongo -u mongouser -p --authenticationDatabase "admin" [mongodb IP]/piggymetrics mongo-init.js
A guest user of the
piggymetricslibrary is created in the MongoDB initialization script mongo-init.js by default, which can be modified as needed.
In the CLB console, create a private network CLB instance for
Subnet-K8S (the ID of this CLB instance will be used later).
TSW is currently in beta test and supports both Java and Go.
The following uses
account-service as an example to describe how to write a Dockerfile. The project directory structure of
account-service is displayed, Dockerfile is in the root directory of the project, and account-service.jar is the packaged file that needs to be added to the image.
➜ account-service tree ├── Dockerfile ├── skywalking │ ├── account.config │ └── skywalking-agent.zip ├── pom.xml ├── src │ .... ├── target │ ..... │ └── account-service.jar └── account-service.iml
Here, SkyWalking Agent is used as the TSW access client that reports call chain information to the TSW backend. For more information on how to download SkyWalking Agent, see PiggyMetrics deployment architecture and add-ons.
The Dockerfile of
account-service is as shown below:
# Working directory in the container
# Add the locally packaged application to the image.
# Copy SkyWalking Agent to the image.
# Decompress SkyWalking Agent and delete the original compressed file.
RUN unzip skywalking-agent.zip && rm -f skywalking-agent.zip
# Add the SkyWalking configuration file.
COPY ./skywalking/account.config ./skywalking-agent/config/agent.config
# Start the application.
CMD ["java", "-Xmx256m", "-javaagent:/app/skywalking-agent/skywalking-agent.jar", "-jar", "/app/account-service.jar"]
# Port description of the application
As each Run command in the Dockerfile will generate an image layer, we recommend you combine these commands into one.
TCR provides both automatic and manual methods to build an image. For more information, see Image Building Overview. To demonstrate the build process, the manual method is used.
The image name needs to be in line with the convention of
namespacecan be the project name to facilitate image management and use. In this document,
piggymetricsrepresents all the images under the PiggyMetrics project.
ImageNamecan contain the
subpath, generally used for multi-project use cases of enterprise users. In addition, if a local image is already built, you can run the
docker tagcommand to rename the image in line with the naming convention.
Run the following command to build an image as shown below:
# Recommended build method, which eliminates the need for secondary tagging operations
sudo docker build -t ccr.ccs.tencentyun.com/[namespace]/[ImageName]:[image tag]
# Build a local `account-service` image. The last `.` indicates that the Dockerfile is stored in the current directory (`account-service`).
➜ account-service docker build -t ccr.ccs.tencentyun.com/piggymetrics/account-service:1.0.0 .
# Rename existing images in line with the naming convention
sudo docker tag [ImageId] ccr.ccs.tencentyun.com/[namespace]/[ImageName]:[image tag]
After the build is complete, you can run the following command to view all the images in your local repository.
docker images | grep piggymetrics
A sample is as shown below:
The PiggyMetrics project uses TCR Personal Edition (TCR Enterprise Edition is recommended for enterprise users).
piggymetricsnamespace in the pop-up window. All the images of the PiggyMetrics project are stored under this namespace as shown below:
Log in to TCR and upload an image.
Run the following command to log in to TCR.
docker login --username=[Tencent Cloud account ID] ccr.ccs.tencentyun.com
- You can get your Tencent Cloud account ID on the Account Info page.
- If you forget your TCR login password, you can reset it in My Images of TCR Personal Edition.
- If you are prompted that you have no permission to run the command, add
sudobefore the command and run it as shown below. In this case, you need to enter two passwords, the server admin password required for
sudoand the TCR login password.As shown below:
sudo docker login --username=[Tencent Cloud account ID] ccr.ccs.tencentyun.com
Run the following command to push the locally generated image to TCR.
docker push ccr.ccs.tencentyun.com/[namespace]/[ImageName]:[image tag]
As shown below:
In My Images, you can view all the uploaded images.
The default image type is
Private. If you want to let others use the image, you can set it to
Publicin Image Info as shown below:
When a cluster is created, we recommend you enable Placement Group on the Select Model page. It helps distribute CVM instances across different hosts to increase the system reliability.
PiggyMetrics-k8s-democluster to enter the Basic Info page to view the cluster configuration information.
user home/.kube(choose another if the
configfile has content) to ensure that the default cluster can be accessed each time. If you choose not to store the token in the
.kube, see the Instructions on Connecting to Kubernetes Cluster via kubectl under Cluster API Server Info in the console as shown below:
A namespace is a logical environment in a Kubernetes cluster, allowing you to divide teams or projects. You can create a namespace in the following three methods, and method 1 is recommended.
ConfigMap allows you to decouple the configuration from the running image, making the application more portable. The PiggyMetrics backend service needs to get the MongoDB host and port information from the environment variables and store them by using the ConfigMap.
You can use ConfigMap to store configuration information in the following two methods:
The following is the ConfigMap YAML for PiggyMetrics, where values of pure digits require double quotation marks.
piggymetrics-envfor storing the configuration. The
piggymetricsnamespace is as shown below:
A Secret can be used to store sensitive information such as passwords, tokens, and keys to reduce exposure risks. PiggyMetrics uses it to store account and password information.
You can use a Secret to store sensitive information in the following two methods:
The following is the YAML for creating a Secret in PiggyMetrics, where the
value of the Secret needs to be a Base64-encoded string.
piggymetrics-keysas shown below:
A StatefulSet is used to manage stateful applications. A Pod created accordingly has a persistent identifier in line with the specifications, which will be retained after the Pod is migrated, terminated, or restarted. When using persistent storage, you can map storage volumes to identifiers.
The basic add-ons and services under the PiggyMetrics project such as configuration services, registry, and RabbitMQ have their own data stored and are therefore suitable for deployment through StatefulSet.
Below is a sample deployment YAML for
- name: http
- name: config
- containerPort: 8888
A Deployment declares the Pod template and controls the Pod running policy, which is suitable for deploying stateless applications. PiggyMetrics backend services such as Account are stateless and can use the Deployment.
YAML parameters for the
account-service Deployment are as follows:
|replicas||Indicates the number of Pods to be created.|
|imagePullSecrets||The key to pull an image, which can be obtained from Cluster > Configuration Management > Secret. It is not required for public images.|
|ports||Specifies the port number of the container. It is
Below is a complete sample YAML file for the
# account-service Deployment
- name: account-service
# MongoDB IP address
- name: MONGODB_HOST
# MongoDB username
- name: MONGODB_USER
# MongoDB password
- name: MONGODB_PASSWORD
# TSW access point
- name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
# TSW access token
- name: SW_AGENT_AUTHENTICATION
# Container port
- containerPort: 6000
imagePullSecrets: # Token to pull the image
- name: qcloudregistrykey
You can specify the Service type with Kubernetes
ServiceType, which defaults to
ClusterIP. Valid values of
ServiceType include the following:
The frontend pages and the gateway of PiggyMetrics are packaged together and need to provide services, so
ServiceType is set to
LoadBalancer. TKE enriches the
LoadBalancer mode by configuring the Service through annotations.
If you use the
service.kubernetes.io/qcloud-loadbalancer-internal-subnetid annotations, a private network CLB instance will be created when the Service is deployed. In general, we recommend you create the CLB instance in advance and use the
service.kubernetes.io/loadbalance-id annotations in the deployment YAML to improve the efficiency.
Below is the deployment YAML for
# Deploy `gateway service`.
# Replace it with the ID of the CLB instance of `Subnet-K8S`.
- name: http
selector: # Map the backend `gateway` to the Service.
At this point, you have completed the deployment of PiggyMetrics in TKE and can view the deployment result in the following steps:
The container log collection feature is disabled by default and needs to be enabled as instructed below:
CLS is region-specific. To reduce the network latency, we recommend you select a region closest to your business when creating log resources, which are mainly logsets and log topics. A logset represents a project, a log topic represents a class of services, and a single logset can contain multiple log topics.
PiggyMetrics is deployed in Nanjing region, so you need to select Nanjing region on the Log Topic page when creating logsets:
As PiggyMetrics has multiple backend microservices, you can create a log topic for each microservice to facilitate log categorization.
- A log topic is created for each PiggyMetrics service.
- You need the log topic ID when creating log rules for containers.
You can configure container log collection rules in the console or with CRD.
Log rules specify the location of a log in a container:
You can also configure log collection rules via Custom Resource Definition (CRD). PiggyMetrics uses a container file path for collection and single-line text. The following is a configuration YAML for
account-service log collection. For more information on CRD collection configuration, see Using CRD to Configure Log Collection via YAML.
# Single-line text
# Log topic ID
# Standard container output
You can't find logs if no indexes are created.
TSW is currently in beta test and can be deployed in Guangzhou and Shanghai. Here, deployment in Shanghai is used as an example (PiggyMetrics is deployed in Nanjing).
Enter the Access Point and Token of the TSW obtained in the previous step in
agent.authentication respectively in the
agent.config of SkyWalking.
agent.service_name is the service name, and
agent.namespace can be used to group microservices under the same domain.
account-service configuration is as shown below:
You can also configure SkyWalking Agent by using environment variables. PiggyMetrics uses the ConfigMap and Secret to configure environment variables:
As shown below:
At this point, you have completed TSW access. After starting the container service, you can view the call chain, service topology, and SQL analysis in the TSW console.
account-serviceresponded too slowly and all
statistic-servicerequests failed in the past hour.
statistics-serviceto enter the information page. Click API Observation, and you can see that the API
NestedServletExceptionexception, which makes the API unavailable.