tencent cloud

文档Tencent Cloud TCHouse-PData IngestionImporting TencentDB Data Offline with DataX

Importing TencentDB Data Offline with DataX

下载
聚焦模式
字号
最后更新时间: 2024-11-27 15:36:05
DataX is an open-source CLI that supports importing full or incremental data from TencentDB to Tencent Cloud TCHouse-P. The tool is developed in Java and uses JDBC to connect the source database to the target database. It can run on Windows and Linux. Install the Java environment before use.
DataX installation:
1. Download the source code here and compile it.
2. Directly use datax-v1.0.4-hashdata.tar.gz, an already compiled version.
The following section introduces DataX modified by HashData, which is more efficient to import data to Tencent Cloud TCHouse-P. Tests show that it can import more than 100,000 entries per second. The following is the configuration file to import data from MySQL to Tencent Cloud TCHouse-P:
{
"job": {
"setting": {
"speed": {
"channel": 3,
"byte": 1048576,
"record": 1000
},
"errorLimit": {
"record": 2,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "****",
"password": "****",
"column": [
"*"
],
"splitPk": "id",
"connection": [
{
"table": [
"test1"
],
"jdbcUrl": [
"jdbc:mysql://***:***/db1?serverTimezone=Asia/Shanghai"
]
}
]
}
},
"writer": {
"name": "gpdbwriter",
"parameter": {
"username": "******",
"password": "******",
"column": [
"*"
],
"preSql": [
"truncate table test1"
],
"postSql": [
"select count(*) from test2"
],
"segment_reject_limit": 0,
"copy_queue_size": 2000,
"num_copy_processor": 1,
"num_copy_writer": 1,
"connection": [
{
"jdbcUrl": "jdbc:postgresql://****:**/db1",
"table": [
"test1"
]
}
]
}
}
}
]
}
}
Parameter description:
1. The writer should be gpdbwriter. postgresqlwriter can also be used to write data to Tencent Cloud TCHouse-P, with a poor insertion efficiency though.
2. For specific meanings and parameter tuning, see DataX.
3. We recommend you add the serverTimezone=Asia/Shanghai parameter to the JDBC URL of mysqlreader to avoid data inconsistency caused by time zone issues.

帮助和支持

本页内容是否解决了您的问题?

填写满意度调查问卷,共创更好文档体验。

文档反馈