From a5520113e16be66725f364a75d3b26c681f183c0 Mon Sep 17 00:00:00 2001 From: Aolin Date: Mon, 23 May 2022 15:20:48 +0800 Subject: [PATCH 1/4] This is an automated cherry-pick of #900 Signed-off-by: ti-chi-bot --- en/shard-merge-best-practices.md | 4 ++++ zh/shard-merge-best-practices.md | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/en/shard-merge-best-practices.md b/en/shard-merge-best-practices.md index a6ba152ed..02897ddf2 100644 --- a/en/shard-merge-best-practices.md +++ b/en/shard-merge-best-practices.md @@ -117,6 +117,10 @@ Then you can perform the following steps to fix the `ERROR 1062 (23000): Duplica 3. Run `query-status` to verify whether the data migration task is successfully processed and whether the data from upstream has already been merged and migrated to the downstream database. +## Special processing when the upstream RDS contains sharded tables + +If the upstream data source is an RDS and it contains sharded tables, the table names in MySQL binlog might be invisible when connecting to a SQL client. For example, if the upstream is a UCloud distributed database, the table name in the binlog might have an extra prefix `_0001`. Therefore, you need to configure [table routing](dm-key-features.md#table-routing) based on the table names in binlog, instead of those in the SQL client. + ## Create/drop tables in the upstream In [Merge and Migrate Data from Sharded Tables](feature-shard-merge-pessimistic.md#principles), it is clear that the coordination of sharding DDL lock depends on whether the downstream database receives the DDL statements of all upstream sharded tables. In addition, DM currently **does not support** dynamically creating or dropping sharded tables in the upstream. Therefore, to create or drop sharded tables in the upstream, it is recommended to perform the following steps. diff --git a/zh/shard-merge-best-practices.md b/zh/shard-merge-best-practices.md index 9d7423f21..2d48f202a 100644 --- a/zh/shard-merge-best-practices.md +++ b/zh/shard-merge-best-practices.md @@ -116,7 +116,11 @@ CREATE TABLE `tbl_multi_pk` ( ## 上游 RDS 封装分库分表的处理 +<<<<<<< HEAD 上游数据源为 RDS 且使用了其分库分表功能的情况下，MySQL binlog 中的表名在 SQL client 连接时可能并不可见。例如在 UCloud 分布式数据库 [UDDB](https://www.ucloud.cn/site/product/uddb.html) 中，其 binlog 表名可能会多出 `_0001` 的后缀。这需要根据 binlog 中的表名规律，而不是 SQL client 所见的表名，来配置 [table routing 规则](key-features.md#table-routing)。 +======= +上游数据源为 RDS 且使用了其分库分表功能的情况下，MySQL binlog 中的表名在 SQL client 连接时可能并不可见。例如在 UCloud 分布式数据库 [UDDB](https://docs.ucloud.cn/uddb/README) 中，其 binlog 表名可能会多出 `_0001` 的后缀。这需要根据 binlog 中的表名规律，而不是 SQL client 所见的表名，来配置 [table routing 规则](dm-key-features.md#table-routing)。 +>>>>>>> 2b5ac2e4 (Fix broken link in shard-merge-best-practices.md and addd a missing section (#900)) ## 合表迁移过程中在上游增/删表 From 9a389f8df2e54acf2e828885381087f2e909b1aa Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Mon, 23 May 2022 15:23:24 +0800 Subject: [PATCH 2/4] resolve conflict --- zh/shard-merge-best-practices.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/zh/shard-merge-best-practices.md b/zh/shard-merge-best-practices.md index 2d48f202a..a9c185317 100644 --- a/zh/shard-merge-best-practices.md +++ b/zh/shard-merge-best-practices.md @@ -69,11 +69,11 @@ CREATE TABLE `tbl_no_pk` ( ``` 2. 在 `task.yaml` 文件中增加如下配置跳过自增主键冲突检查： - + ```yaml ignore-checking-items: ["auto_increment_ID"] ``` - + 3. 启动数据迁移任务，执行全量与增量数据迁移。 4. 通过 `query-status` 验证数据迁移任务是否正常，在下游数据库中验证合表中是否已经存在了来自上游的数据。 @@ -116,11 +116,7 @@ CREATE TABLE `tbl_multi_pk` ( ## 上游 RDS 封装分库分表的处理 -<<<<<<< HEAD -上游数据源为 RDS 且使用了其分库分表功能的情况下，MySQL binlog 中的表名在 SQL client 连接时可能并不可见。例如在 UCloud 分布式数据库 [UDDB](https://www.ucloud.cn/site/product/uddb.html) 中，其 binlog 表名可能会多出 `_0001` 的后缀。这需要根据 binlog 中的表名规律，而不是 SQL client 所见的表名，来配置 [table routing 规则](key-features.md#table-routing)。 -======= 上游数据源为 RDS 且使用了其分库分表功能的情况下，MySQL binlog 中的表名在 SQL client 连接时可能并不可见。例如在 UCloud 分布式数据库 [UDDB](https://docs.ucloud.cn/uddb/README) 中，其 binlog 表名可能会多出 `_0001` 的后缀。这需要根据 binlog 中的表名规律，而不是 SQL client 所见的表名，来配置 [table routing 规则](dm-key-features.md#table-routing)。 ->>>>>>> 2b5ac2e4 (Fix broken link in shard-merge-best-practices.md and addd a missing section (#900)) ## 合表迁移过程中在上游增/删表 From 150688b1e2fac59f1f0463173c689f80c4e7dfd7 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Mon, 23 May 2022 15:26:47 +0800 Subject: [PATCH 3/4] Update zh/shard-merge-best-practices.md --- zh/shard-merge-best-practices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/zh/shard-merge-best-practices.md b/zh/shard-merge-best-practices.md index a9c185317..be9589bb3 100644 --- a/zh/shard-merge-best-practices.md +++ b/zh/shard-merge-best-practices.md @@ -116,7 +116,7 @@ CREATE TABLE `tbl_multi_pk` ( ## 上游 RDS 封装分库分表的处理 -上游数据源为 RDS 且使用了其分库分表功能的情况下，MySQL binlog 中的表名在 SQL client 连接时可能并不可见。例如在 UCloud 分布式数据库 [UDDB](https://docs.ucloud.cn/uddb/README) 中，其 binlog 表名可能会多出 `_0001` 的后缀。这需要根据 binlog 中的表名规律，而不是 SQL client 所见的表名，来配置 [table routing 规则](dm-key-features.md#table-routing)。 +上游数据源为 RDS 且使用了其分库分表功能的情况下，MySQL binlog 中的表名在 SQL client 连接时可能并不可见。例如在 UCloud 分布式数据库 [UDDB](https://docs.ucloud.cn/uddb/README) 中，其 binlog 表名可能会多出 `_0001` 的后缀。这需要根据 binlog 中的表名规律，而不是 SQL client 所见的表名，来配置 [table routing 规则](key-features.md#table-routing)。 ## 合表迁移过程中在上游增/删表 From 89f32a198b064ef1d4a1ccbb6fab85bce5a16acf Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Mon, 23 May 2022 15:28:03 +0800 Subject: [PATCH 4/4] Update en/shard-merge-best-practices.md --- en/shard-merge-best-practices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/shard-merge-best-practices.md b/en/shard-merge-best-practices.md index 02897ddf2..a189a9726 100644 --- a/en/shard-merge-best-practices.md +++ b/en/shard-merge-best-practices.md @@ -119,7 +119,7 @@ Then you can perform the following steps to fix the `ERROR 1062 (23000): Duplica ## Special processing when the upstream RDS contains sharded tables -If the upstream data source is an RDS and it contains sharded tables, the table names in MySQL binlog might be invisible when connecting to a SQL client. For example, if the upstream is a UCloud distributed database, the table name in the binlog might have an extra prefix `_0001`. Therefore, you need to configure [table routing](dm-key-features.md#table-routing) based on the table names in binlog, instead of those in the SQL client. +If the upstream data source is an RDS and it contains sharded tables, the table names in MySQL binlog might be invisible when connecting to a SQL client. For example, if the upstream is a UCloud distributed database, the table name in the binlog might have an extra prefix `_0001`. Therefore, you need to configure [table routing](key-features.md#table-routing) based on the table names in binlog, instead of those in the SQL client. ## Create/drop tables in the upstream