Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge DM dev docs to TiDB dev docs #8042

Merged
merged 17 commits into from
Dec 23, 2021
Merged

Merge DM dev docs to TiDB dev docs #8042

merged 17 commits into from
Dec 23, 2021

Conversation

qiancai
Copy link
Collaborator

@qiancai qiancai commented Dec 21, 2021 •
edited by wd0517
Loading

What is changed, added or deleted? (Required)

To help users access TiDB Data Migration (DM) documentation more quickly and conveniently, this PR merges TiDB DM dev docs into TiDB dev docs.

The major changes are as follows:

  1. Copy the DM dev documentation in https://github.com/pingcap/docs-dm/pull/869/files to the TiDB dev documentation, and add the DM dev TOC to TiDB dev TOC.
  2. In DM dev documentation, update the external links of TiDB documentation to internal links.
  3. Update the external links of DM documentation to internal links.
  4. Add dm-releases-notes.md.

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions (in Chinese).

  • master (the latest development version)
  • v5.3 (TiDB 5.3 versions)
  • v5.2 (TiDB 5.2 versions)
  • v5.1 (TiDB 5.1 versions)
  • v5.0 (TiDB 5.0 versions)
  • v4.0 (TiDB 4.0 versions)
  • v3.1 (TiDB 3.1 versions)
  • v3.0 (TiDB 3.0 versions)
  • v2.1 (TiDB 2.1 versions)

What is the related PR or file link(s)?

  • This PR is translated from:
  • Other reference link(s):

Do your changes match any of the following descriptions?

  • Delete files
  • Change aliases
  • Need modification after applied to another branch
  • Might cause conflicts after applied to another branch

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 21, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • TomShawn
  • lilin90

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot requested a review from TomShawn December 21, 2021 15:20
@ti-chi-bot ti-chi-bot added missing-translation-status This PR does not have translation status info. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 21, 2021
@qiancai qiancai added the translation/doing This PR’s assignee is translating this PR. label Dec 21, 2021
@ti-chi-bot ti-chi-bot removed the missing-translation-status This PR does not have translation status info. label Dec 21, 2021
@qiancai qiancai mentioned this pull request Dec 21, 2021
Merged
13 tasks
@TomShawn TomShawn added the type/refactor The issue belongs to a refactor work. label Dec 22, 2021
TomShawn
TomShawn reviewed Dec 22, 2021
View reviewed changes
TomShawn
TomShawn reviewed Dec 22, 2021
View reviewed changes
TomShawn
TomShawn reviewed Dec 22, 2021
View reviewed changes
dm/dm-faq.md Outdated
- TiDB Server 的全局变量 `max_allowed_packet`
- 任务配置文件中的配置项 `target-database.max-allowed-packet`（详情参见 [DM 任务完整配置文件介绍](/dm/task-configuration-file-full.md)）

设置为比默认 67108864 (64M) 更大的值。详见 [Loader 解决方案](https://docs.pingcap.com/zh/tidb/stable/loader-overview#解决方案)。
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loader 在 v5.0 中就废弃了

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sunzhaoyang 请帮忙看下是否有 Loader 的替代方案呢？谢谢

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个 bug 在 dm v2.0 GA 版本就已经修复了。我觉得可以采用如下两个方案：

  1. 去掉这个链接，如果还有用老版本的用户遇到这个问题，直接告诉他需要怎么workaround 就够了，不用在强调 loader.
  2. 将链接强制跳转到 3.0/4.0 版本的 loader 配置页面，即 https://docs.pingcap.com/zh/tidb/v4.0/loader-overview/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

赞同去掉链接。根据前面的信息设置更大的值就可以解决了。

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

TomShawn
TomShawn reviewed Dec 22, 2021
View reviewed changes
dm/dm-faq.md Outdated
该情况有两种解决方案：

- 如果数据量较小（TB 级以下）或任务有合库合表：清空下游数据库的已导入数据，同时清空导出数据目录，使用 dmctl 删除并 `start-task --remove-meta` 重建任务。后续尽量保证全量导出导入阶段 DM 没有冗余 worker 以及避免在该时段内重启或升级 DM 集群。
- 如果数据量较大（数 TB 或更多）：清空下游数据库的已导入数据，将 lightning 部署到数据所在的 DM worker 节点，使用 [lightning local backend 模式](https://docs.pingcap.com/zh/tidb/dev/deploy-tidb-lightning) 导入 DM dump 单元导出的数据。全量导入完成后，修改任务的 `task-mode` 为 `incremental`，修改 `mysql-instance.meta.pos` 为 dump 单元导出数据 `metadata` 中记录的位置，启动一个增量任务。
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Global comment: Please check all links that start with https://docs.pingcap.com/zh/tidb/ and https://pingcap.com/docs-cn/.

Suggested change
- 如果数据量较大（数 TB 或更多）：清空下游数据库的已导入数据，将 lightning 部署到数据所在的 DM worker 节点，使用 [lightning local backend 模式](https://docs.pingcap.com/zh/tidb/dev/deploy-tidb-lightning) 导入 DM dump 单元导出的数据。全量导入完成后，修改任务的 `task-mode` 为 `incremental`，修改 `mysql-instance.meta.pos` 为 dump 单元导出数据 `metadata` 中记录的位置，启动一个增量任务。
- 如果数据量较大（数 TB 或更多）：清空下游数据库的已导入数据，将 lightning 部署到数据所在的 DM worker 节点，使用 [lightning local backend 模式](https://docs.pingcap.com/zh/tidb/dev/deploy-tidb-lightning) 导入 DM dump 单元导出的数据。全量导入完成后，修改任务的 `task-mode` 为 `incremental`，修改 `mysql-instance.meta.pos` 为 dump 单元导出数据 `metadata` 中记录的位置，启动一个增量任务。

TomShawn
TomShawn reviewed Dec 22, 2021
View reviewed changes
TomShawn
TomShawn reviewed Dec 22, 2021
View reviewed changes
dm/dm-precheck.md Outdated
TiDB 和 MySQL 的兼容性存在以下一些区别：

- TiDB 不支持外键
- 字符集的兼容性不同，详见 [TiDB 支持的字符集](https://pingcap.com/docs-cn/dev/reference/sql/character-set/)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

qiancai and others added 4 commits December 22, 2021 21:11
@qiancai @TomShawn
Apply suggestions from code review
39196b2 
Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>
@qiancai @TomShawn
Update dm/dm-open-api.md
3070d84 
Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>
@qiancai
update_external_links_of_tidb_docs_to_internal_links
0bcf6e9
@qiancai
fix_broken_links
5bec6a3
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 23, 2021
@qiancai
Copy link
Collaborator Author

qiancai commented Dec 23, 2021

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: a3d0fa6

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 23, 2021
@ti-chi-bot ti-chi-bot merged commit 1a20622 into pingcap:master Dec 23, 2021
This was referenced Dec 23, 2021
Closed
Closed
Merged
@Rustin170506 Rustin170506 mentioned this pull request Dec 30, 2021
Merged
4 tasks
@glorv glorv mentioned this pull request Jan 17, 2022
Merged
8 tasks
@shichun-0415 shichun-0415 mentioned this pull request Jan 24, 2022
Merged
8 tasks
@TomShawn TomShawn added translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. and removed translation/doing This PR’s assignee is translating this PR. labels Feb 8, 2022
@qiancai qiancai mentioned this pull request Feb 14, 2022
Merged
7 tasks
@Austaras Austaras mentioned this pull request Feb 23, 2022
Merged
7 tasks
@buchuitoudegou buchuitoudegou mentioned this pull request Mar 24, 2022
Closed
4 tasks
@luzizhuo luzizhuo mentioned this pull request Apr 25, 2022
Closed
8 tasks
This was referenced May 23, 2022
Closed
Merged
@ti-chi-bot ti-chi-bot mentioned this pull request May 23, 2022
Merged
8 tasks
@shichun-0415 shichun-0415 mentioned this pull request Jun 14, 2022
Merged
8 tasks
This was referenced Jun 20, 2022
Merged
Merged
Merged
This was referenced Jun 20, 2022
Merged
Merged
@Oreoxmt Oreoxmt mentioned this pull request Sep 28, 2022
Merged
7 tasks
@lilin90 lilin90 mentioned this pull request Nov 18, 2024
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@glorv glorv glorv left review comments

@sunzhaoyang sunzhaoyang sunzhaoyang approved these changes

@lilin90 lilin90 lilin90 approved these changes

@TomShawn TomShawn TomShawn approved these changes

Assignees

@qiancai qiancai

Labels
size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. type/refactor The issue belongs to a refactor work.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@qiancai @ti-chi-bot @sunzhaoyang @glorv @lilin90 @TomShawn