数据传输 | dtle 使用初探

于由马莹乐发布

作者：马莹乐

爱可生研发团队成员，负责 mysql 中间件和数据库管理平台的测试。擅长找茬（测试技术爱好者），欢迎大家试用 dtle~

本文来源：原创投稿

*爱可生开源社区出品，原创内容未经授权不得随意使用，转载请联系小编并注明来源。

dtle 是一款爱可生 MySQL 开源数据传输中间件，此文简要介绍初步使用方法。
项目地址：https://github.com/actiontech/dtle

一、下载安装

环境说明：
  • 准备三台主机，主机名分别为：node4、node5、node6
  • 三台均为 agent，另将 node4 和 node5 作为 manager

1.1 下载

下载 dtle 最新的 release rpm 包

地址：https://github.com/actiontech/dtle/releases

dtle 安装包下载：

  1. wget https://github.com/actiontech/dtle/releases/download/v2.19.11.0/dtle-2.19.11.0.x86_64.rpm

1.2 安装

安装步骤：

https://actiontech.github.io/dtle-docs-cn/4/4.0_installation.html

分别将 node4，node5 和 node6 的 dtle 安装完成。

1.3 启动 dtle

dtle 安装完成后，并未启动，需要先配置 dtle 的 conf 文件，再启动。

1.4 配置 dtle.conf

配置相关说明：

https://actiontech.github.io/dtle-docs-cn/4/4.1nodeconfiguration.html

dtle.conf 准备：以下配置文件替分别替换原有 dtle.conf（/opt/dtle/etc/dtle/dtle.conf）

node4 的 dtle.conf
  1. # Setup data dir
  2. data_dir = "/opt/dtle/data"
  3. log_level = "DEBUG"
  4. log_file = "/opt/dtle.log"
  5. #log_to_stdout = true

  7. bind_addr = "172.100.9.4"

  9. # Modify our port to avoid a collision with server
  10. ports {
  11.     http = 8190
  12. }

  14. # Enable the manager
  15. manager {
  16.     enabled = true

  18.     # Self-elect, should be 3 or 5 for production,
  19.     # Addresses to attempt to join when the server starts.
  20.     join = [ "172.100.9.4","172.100.9.5" ]
  21. }

  23. # Enable the agent
  24. agent {
  25.     enabled = true
  26.     managers = ["172.100.9.4:8191","172.100.9.5:8191"]
  27. }

  29. metric{
  30.    collection_interval = "15s"
  31.    publish_allocation_metrics = "true"
  32.    publish_node_metrics = "true"
  33. }

  35. addresses {
  36.     http = "172.100.9.4"
  37.     rpc = "172.100.9.4"
  38.     serf = "172.100.9.4"
  39. }
  40. advertise {
  41.     http = "172.100.9.4"
  42.     rpc = "172.100.9.4"
  43.     serf = "172.100.9.4"
  44. }
node5 的 dtle.conf
  1. # Setup data dir
  2. data_dir = "/opt/dtle/data"
  3. log_level = "DEBUG"
  4. log_file = "/opt/dtle.log"
  5. #log_to_stdout = true

  7. bind_addr = "172.100.9.5"

  9. # Modify our port to avoid a collision with server
  10. ports {
  11.     http = 8190
  12. }

  14. # Enable the manager
  15. manager {
  16.     enabled = true

  18.     # Self-elect, should be 3 or 5 for production,
  19.     # Addresses to attempt to join when the server starts.
  20.     join = [ "172.100.9.4","172.100.9.5" ]
  21. }

  23. # Enable the agent
  24. agent {
  25.     enabled = true
  26.     managers = ["172.100.9.4:8191","172.100.9.5:8191"]
  27. }

  29. metric{
  30.    collection_interval = "15s"
  31.    publish_allocation_metrics = "true"
  32.    publish_node_metrics = "true"
  33. }

  35. addresses {
  36.     http = "172.100.9.5"
  37.     rpc = "172.100.9.5"
  38.     serf = "172.100.9.5"
  39. }
  40. advertise {
  41.     http = "172.100.9.5"
  42.     rpc = "172.100.9.5"
  43.     serf = "172.100.9.5"
  44. }

node6 的 dtle.conf

  1. # Setup data dir
  2. data_dir = "/opt/dtle/data"
  3. log_level = "DEBUG"
  4. log_file = "/opt/dtle.log"
  5. #log_to_stdout = true

  7. bind_addr = "172.100.9.6"

  9. # Modify our port to avoid a collision with server
  10. ports {
  11.     http = 8190
  12. }

  14. # Enable the manager
  15. manager {
  16.     enabled = false

  18.     # Self-elect, should be 3 or 5 for production,
  19.     # Addresses to attempt to join when the server starts.
  20.     join = [ "172.100.9.4","172.100.9.5" ]
  21. }

  23. # Enable the agent
  24. agent {
  25.     enabled = true
  26.     managers = ["172.100.9.4:8191","172.100.9.5:8191"]
  27. }

  29. metric{
  30.    collection_interval = "15s"
  31.    publish_allocation_metrics = "true"
  32.    publish_node_metrics = "true"
  33. }

  35. addresses {
  36.     http = "172.100.9.6"
  37.     rpc = "172.100.9.6"
  38.     serf = "172.100.9.6"
  39. }
  40. advertise {
  41.     http = "172.100.9.6"
  42.     rpc = "172.100.9.6"
  43.     serf = "172.100.9.6"
  44. }

1.5 启动 dtle

启动命令说明：

https://actiontech.github.io/dtle-docs-cn/4/4.2_command.html

1.6 查看 3 个节点 dtle 进程

node 4

node 5

node 6

1.7 尝试查看 dtle 架构是否符合预期

命令说明：

https://actiontech.github.io/dtle-docs-cn/4/4.2_command.html

其中，members 展示的是集群中 manager 节点的信息。

HTTP API：

https://actiontech.github.io/dtle-docs-cn/4/4.4httpapi.html

输出内容为 json 格式，可以安装 jq，使之格式化展示。

二、源端/目标端准备

MySQL 迁移用户准备。

2.1 创建用户

源端用户创建，

目标端用户创建，

有关源端/目标端用户的最小权限说明：

https://actiontech.github.io/dtle-docs-cn/4/4.5mysqluser_privileges.html

2.2 数据准备

源端数据准备，

目标端测试前库表情况，

2.3 创建 job

创建全量 + 增量，表级别迁移 job，分别使用 node5 和 node6 节点的 dtle 作为源端和目标端迁移的 agent。

job 配置相关参数：

https://actiontech.github.io/dtle-docs-cn/4/4.3jobconfiguration.html

2.3.1 准备 job.json

job.json

  1. {
  2.     "Name":"have_a_try",
  3.     "Failover":false,
  4.     "Orders":[],
  5.     "Type":"synchronous",
  6.     "Tasks":[
  7.         {
  8.             "Type":"Src",
  9.             "NodeId":"ee97dc49-85ed-febc-4d3c-cfbfa87f46bd",
  10.             "Config":{
  11.                 "Gtid":"",
  12.                 "DropTableIfExists":false,
  13.                 "SkipCreateDbTable":false,
  14.                 "ApproveHeterogeneous":true,
  15.                 "ReplChanBufferSize":"600",
  16.                 "ChunkSize":"2000",
  17.                 "ExpandSyntaxSupport":false,
  18.                 "MsgBytesLimit":"20480",
  19.                 "MsgsLimit":"65536",
  20.                 "BytesLimit":"67108864",
  21.                 "GroupMaxSize":"1",
  22.                 "GroupTimeout":"100",
  23.                 "SqlFilter":[],
  24.                 "ReplicateDoDb":[
  25.                     {
  26.                         "TableSchema":"test",
  27.                         "Tables":[
  28.                             {
  29.                                 "TableName":"test1"
  30.                             }
  31.                         ]
  32.                     }
  33.                 ],
  34.                 "ConnectionConfig":{
  35.                     "Host":"172.100.9.1",
  36.                     "Port":"3306",
  37.                     "User":"src_test",
  38.                     "Password":"test"
  39.                 }
  40.             }
  41.         },
  42.         {
  43.             "Type":"Dest",
  44.             "NodeId":"e623aedd-5c37-da67-4ddf-1a82ce1ac298",
  45.             "Config":{
  46.                 "ParallelWorkers":"1",
  47.                 "ConnectionConfig":{
  48.                     "Host":"172.100.9.2",
  49.                     "Port":"3306",
  50.                     "User":"dest_test",
  51.                     "Password":"test"
  52.                 }
  53.             }
  54.         }
  55.     ]
  56. }

其中 NodeId 可通过命令或者 HTTP API 查询得知。

2.3.2 创建 job

创建 job 试用 HTTP API：

https://actiontech.github.io/dtle-docs-cn/4/4.4httpapi.html

创建完成后查询 job 状态为 running。

job 详细信息可使用 HTTP API 查询，示例：

  1. [
  2.     {
  3.         "CreateIndex":109,
  4.         "ID":"ba13f3e1-0630-f141-4c60-a1f36c61cdb3",
  5.         "JobModifyIndex":2372,
  6.         "JobSummary":{
  7.             "Constraints":null,
  8.             "CreateIndex":109,
  9.             "Datacenters":[
  10.                 "dc1"
  11.             ],
  12.             "EnforceIndex":false,
  13.             "Failover":false,
  14.             "ID":"ba13f3e1-0630-f141-4c60-a1f36c61cdb3",
  15.             "JobModifyIndex":2372,
  16.             "ModifyIndex":2372,
  17.             "Name":"have_a_try",
  18.             "Orders":[

  20.             ],
  21.             "Region":"global",
  22.             "Status":"running",
  23.             "StatusDescription":"",
  24.             "Tasks":[
  25.                 {
  26.                     "Config":{
  27.                         "GroupMaxSize":"1",
  28.                         "ChunkSize":"2000",
  29.                         "BinlogFile":"1.000002",
  30.                         "MsgsLimit":"65536",
  31.                         "GroupTimeout":"100",
  32.                         "DropTableIfExists":false,
  33.                         "ExpandSyntaxSupport":false,
  34.                         "BytesLimit":"67108864",
  35.                         "MsgBytesLimit":"20480",
  36.                         "ReplChanBufferSize":"600",
  37.                         "NatsAddr":"172.100.9.6:8193",
  38.                         "ApproveHeterogeneous":true,
  39.                         "SqlFilter":[

  41.                         ],
  42.                         "BinlogPos":4261,
  43.                         "ConnectionConfig":{
  44.                             "Host":"172.100.9.1",
  45.                             "Port":"3306",
  46.                             "User":"src_test",
  47.                             "Password":"*"
  48.                         },
  49.                         "TrafficAgainstLimits":0,
  50.                         "Gtid":"8868d98f-af5e-11e8-9aa9-0242ac110002:1-16",
  51.                         "ReplicateDoDb":[
  52.                             {
  53.                                 "TableSchema":"test",
  54.                                 "Tables":[
  55.                                     {
  56.                                         "TableName":"test1"
  57.                                     }
  58.                                 ]
  59.                             }
  60.                         ],
  61.                         "SkipCreateDbTable":false
  62.                     },
  63.                     "ConfigLock":{

  65.                     },
  66.                     "Constraints":null,
  67.                     "Driver":"MySQL",
  68.                     "Leader":false,
  69.                     "NodeID":"ee97dc49-85ed-febc-4d3c-cfbfa87f46bd",
  70.                     "NodeName":"",
  71.                     "Type":"Src"
  72.                 },
  73.                 {
  74.                     "Config":{
  75.                         "BinlogFile":"1.000002",
  76.                         "BinlogPos":4261,
  77.                         "ConnectionConfig":{
  78.                             "Host":"172.100.9.2",
  79.                             "Port":"3306",
  80.                             "User":"dest_test",
  81.                             "Password":"*"
  82.                         },
  83.                         "Gtid":"8868d98f-af5e-11e8-9aa9-0242ac110002:1-16",
  84.                         "ParallelWorkers":"1",
  85.                         "NatsAddr":"172.100.9.6:8193"
  86.                     },
  87.                     "ConfigLock":{

  89.                     },
  90.                     "Constraints":null,
  91.                     "Driver":"MySQL",
  92.                     "Leader":true,
  93.                     "NodeID":"e623aedd-5c37-da67-4ddf-1a82ce1ac298",
  94.                     "NodeName":"",
  95.                     "Type":"Dest"
  96.                 }
  97.             ],
  98.             "Type":"synchronous"
  99.         },
  100.         "ModifyIndex":2372,
  101.         "Name":"have_a_try",
  102.         "Status":"running",
  103.         "StatusDescription":"",
  104.         "Type":"synchronous"
  105.     }
  106. ]

2.4 查询目标端迁移情况

目标端迁移前后对比：

新增两个库其中 test 库是我们 job 迁移的结果。

dtle 库相关说明：

https://actiontech.github.io/dtle-docs-cn/3/3.3impacton_dest.html

2.5 在源端执行普通 DML 操作

2.6 在源端执行普通 DDL 操作

小结

更多用法，详见项目文档，欢迎试用~

https://actiontech.github.io/dtle-docs-cn/
如果有更多关于 DTLE 相关的技术问题，
可以扫码或直接加入官方QQ技术交流群（852990221）进行交流。
分类： DTLE 数据传输组件
标签：

相关文章

DTLE 数据传输组件

数据传输 | lower_case_table_names 设置对数据一致性的影响

作者通过全面系统的测试，揭秘 lower_case_table_nam 阅读更多…

DTLE 数据传输组件

数据传输 | DTLE 4.2211.0 来啦!

4.22.11.0 版本DTLE Release Notes 开源数据 阅读更多…

DTLE 数据传输组件

数据传输 | DTLE 4.22.07.2 来啦!特定场景性能提升2-4倍！

4.22.07.2 版本DTLE Release Notes 开源数据 阅读更多…