Data subscription Database table to Redis

Last updated: 2020-02-19 19:29:21


This article will use a simple case to illustrate the function of pulling the corresponding table to Redis in data subscription, and provide a simple RedisDemo download . The following operations will be done in the CentOS operating system.

Configure the environment

  • Java environment configuration
yum install java-1.8.0-openjdk-devel 

Get the key

Login Access Management console Get the key.

Select data subscription

  1. Login DTS console Select "data subscription" on the left to enter the data subscription page.
  2. Select the name of the TencentDB instance to be synchronized, then click Launch, return to the data subscription, and click the data subscription you created. For a detailed introduction, please refer to How to get a data subscription .
  3. Check the corresponding DTS channel, IP and Port, and then combine the previous key Enter into the corresponding
 context.setSecretId("AKIDfdsfdsfsdt1331431sdfds"); Please enter the secretID you got from Cloud API
        context.setSecretKey("test111usdfsdfsddsfRkeT"); Please enter the secretID you got from Cloud API
// Get the corresponding IP and port through data retrieval in the data migration service,and enter here 
       context.setServiceIp(""); Please enter the IP you got from Data Subscription Configuration
        context.setServicePort(7507); Please enter the PORT you got from Data Subscription Configuration

        // Create clients
        //SubscribeClient client=new DefaultSubscribeClient(context,true);
        final DefaultSubscribeClient client = new DefaultSubscribeClient(context);

        final Jedis jedis = new Jedis("", 6379); Please enter your redis CVM and port

        final String targetDatabase = "test"; Please enter the database name you want to subscribe.
        final String targetTable = "alantest"; Please enter the table name you want to subscribe.There are 2 fields,id and name.(id is the main key.)

        // Create listener
        ClusterListener listener = new ClusterListener() {
            public void notify(List<ClusterMessage> messages) throws Exception {
        //                System.out.println("--------------------:" + messages.size());
                for(ClusterMessage m:messages){
                    DataMessage.Record record = m.getRecord();
                    //Filter the subscription information you have no interest in
                if(!record.getDbName().equalsIgnoreCase(targetDatabase) || !record.getTablename().equalsIgnoreCase(targetTable)){
                        //Note: Ack the information even you have no interest in                         m.ackAsConsumed();

                    if(record.getOpt() != DataMessage.Record.Type.BEGIN && record.getOpt() != DataMessage.Record.Type.COMMIT){
                        List<DataMessage.Record.Field> fields = record.getFieldList();

                        //INSERT RECORD
                        //String pk = record.getPrimaryKeys();

                        if(record.getOpt() == DataMessage.Record.Type.INSERT){

                String keyid="";
                String value="";
                            for (DataMessage.Record.Field field : fields) {

                                //Get the id first, and you need a primary key, then find a column named name, assign a value to redis and insert the value corresponding to the key and name.                if(field.getFieldname().equalsIgnoreCase("id")){

                jedis.set(keyid, value);

Compilation operation and verification

  1. [root@VM_71_10_centos ~]# javac -classpath binlogsdk-2.6.0-release.jar:jedis-2.9.0.jar -encoding UTF-8 
  2. Execute Launch, if there is no abnormal error report, it is normal in service, and then check the landing file set before.

 java -XX:-UseGCOverheadLimit -Xms2g -Xmx2g -classpath .:binlogsdk-2.6.0-release.jar:jedis-2.9.0.jar RedisDemo
  1. Check to insert the database and update operation, and observe from redis to find that the insertion and modification are indeed successful. Finally, the delete operation is carried out, and the corresponding data of redis is deleted.
MySQL [test]> insert into alantest values(1001,'alan1');
Query OK, 1 row affected (0.00 sec)

MySQL [test]> update alantest set name='alan2' where id=1001;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

------------------------> get 1001

MySQL [test]> update alantest set name='alan3' where id=1001;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

------------------------> get 1001

MySQL [test]> delete from alantest where id=1001;
Query OK, 1 row affected (0.00 sec)

-----------------------> get 1001