Migrate experimental flags in robustness tests

[siyuanfoundation]

What would you like to be added?

With #19141 and #18023, we will be removing all experimental flags in 3.7.

The robustness tests are used by different binary versions, for the same parameter, we would need to use the experimental flag in 3.5 and 3.4, while using the non-experimental flag in 3.6.

The affected flags include:

• experimental-snapshot-catchup-entries
• experimental-compaction-batchLimit
• experimental-watch-process-notify-interval

Why is this needed?

The robustness test would fail when we delete the deprecated flags if we don't fix this.

[kavirajk]

@siyuanfoundation I would like to pick this up if that's ok.

Also quick clarification. I see few more experimental flags are to be removed in v1.37.

-bash-5.2$ rg "TODO: Delete in v3.7" -A 3 | rg "Experimental" | rg "json" | awk '{print$2}'
ExperimentalTxnModeWriteWithSharedBuffer
ExperimentalEnableLeaseCheckpointPersist
ExperimentalCompactionBatchLimit
ExperimentalCompactionSleepInterval
ExperimentalWatchProgressNotifyInterval
ExperimentalWarningApplyDuration
ExperimentalBootstrapDefragThresholdMegabytes
ExperimentalWarningUnaryRequestDuration
ExperimentalMaxLearners
ExperimentalPeerSkipClientSanVerification
ExperimentalMemoryMlock
ExperimentalTxnModeWriteWithSharedBuffer
ExperimentalStopGRPCServiceOnDefrag

Those things can also cause failure in robustness test IIUC? no? But I see you mentioned only subset of it.

[siyuanfoundation]

[@siyuanfoundation](https://github.com/siyuanfoundation) I would like to pick this up if that's ok.

Also quick clarification. I see few more experimental flags are to be removed in v1.37.

-bash-5.2$ rg "TODO: Delete in v3.7" -A 3 | rg "Experimental" | rg "json" | awk '{print$2}'
ExperimentalTxnModeWriteWithSharedBuffer
ExperimentalEnableLeaseCheckpointPersist
ExperimentalCompactionBatchLimit
ExperimentalCompactionSleepInterval
ExperimentalWatchProgressNotifyInterval
ExperimentalWarningApplyDuration
ExperimentalBootstrapDefragThresholdMegabytes
ExperimentalWarningUnaryRequestDuration
ExperimentalMaxLearners
ExperimentalPeerSkipClientSanVerification
ExperimentalMemoryMlock
ExperimentalTxnModeWriteWithSharedBuffer
ExperimentalStopGRPCServiceOnDefrag
Those things can also cause failure in robustness test IIUC? no? But I see you mentioned only subset of it.

These flags are not currently used in robustness tests. If they are being used, then yes they would break.

[ivanvc]

Covered in our triage meeting. @kavirajk, are you interested in working on this? If so, please comment /assign so the k8s-bot can assign it to you

[kavirajk]

/assign