OCPBUGS-52367: Fix collecting IPsec data for upgrade

[pperiyasamy]

After IPsec is upgraded from 4.14 into later versions, API option for IPsec in networks.operator.openshift.io cluster object continues to have {} in the ipsecConfig and may not be migrated to newer version containing IPsec mode option. This is supported in OCP for backward compatibility, so mg needs fixing to collect IPsec data when API configured with old option.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pperiyasamy
Once this PR has been reviewed and has the lgtm label, please assign sferich888 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• collection-scripts/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pperiyasamy]

/assign @huiran0826

[pperiyasamy]

/retest

[openshift-ci]

@pperiyasamy: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[huiran0826]

pre-merge tested it, looks good

oc get network.operator -o jsonpath='{.items[*].spec.defaultNetwork}'    
{"ovnKubernetesConfig":{"egressIPConfig":{},"gatewayConfig":{"ipv4":{},"ipv6":{},"routingViaHost":false},"genevePort":6081,"ipsecConfig":{},"mtu":8855,"policyAuditConfig":{"destination":"null","maxFileSize":50,"maxLogFiles":5,"rateLimit":20,"syslogFacility":"local0"}},"type":"OVNKubernetes"}
% ls -lrt
total 48
-rw-r--r--   1 huirwang  staff  1672 Mar  4 19:46 ovn-ipsec-host-m26pz_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  4 19:46 ovn-ipsec-host-hj97b_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  4 19:46 ovn-ipsec-host-xbgbz_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  4 19:46 ovn-ipsec-host-n9rdl_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  4 19:46 ovn-ipsec-host-dxbk8_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  4 19:46 ovn-ipsec-host-cqv42_ipsec.conf
drwxr-xr-x   5 huirwang  staff   160 Mar  4 19:54 ovn-ipsec-host-cqv42_ipsec.d
drwxr-xr-x   5 huirwang  staff   160 Mar  4 19:54 ovn-ipsec-host-dxbk8_ipsec.d
drwxr-xr-x   5 huirwang  staff   160 Mar  4 19:54 ovn-ipsec-host-hj97b_ipsec.d
drwxr-xr-x   5 huirwang  staff   160 Mar  4 19:54 ovn-ipsec-host-m26pz_ipsec.d
drwxr-xr-x   5 huirwang  staff   160 Mar  4 19:54 ovn-ipsec-host-n9rdl_ipsec.d
drwxr-xr-x   5 huirwang  staff   160 Mar  4 19:54 ovn-ipsec-host-xbgbz_ipsec.d
drwxr-xr-x   8 huirwang  staff   256 Mar  4 19:54 status
drwxr-xr-x   8 huirwang  staff   256 Mar  4 19:54 trafficstatus
drwxr-xr-x  14 huirwang  staff   448 Mar  4 19:54 xfrm

/label qe-approved

[huiran0826]

Double checked for full mode, ipsec logs were collected in must-gather

 % oc get network.operator -o jsonpath='{.items[*].spec.defaultNetwork.ovnKubernetesConfig.ipsecConfig}' | jq .
{
  "mode": "Full"
}
% ls -lrt
total 48
-rw-r--r--   1 huirwang  staff  1672 Mar  5 18:16 ovn-ipsec-host-q7vhv_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  5 18:16 ovn-ipsec-host-kxssh_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  5 18:16 ovn-ipsec-host-k8bng_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  5 18:16 ovn-ipsec-host-gz4r8_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  5 18:16 ovn-ipsec-host-d22t5_ipsec.conf
-rw-r--r--   1 huirwang  staff  1672 Mar  5 18:16 ovn-ipsec-host-7gdf4_ipsec.conf
drwxr-xr-x   4 huirwang  staff   128 Mar  5 18:20 ovn-ipsec-host-7gdf4_ipsec.d
drwxr-xr-x   4 huirwang  staff   128 Mar  5 18:20 ovn-ipsec-host-d22t5_ipsec.d
drwxr-xr-x   4 huirwang  staff   128 Mar  5 18:20 ovn-ipsec-host-gz4r8_ipsec.d
drwxr-xr-x   4 huirwang  staff   128 Mar  5 18:20 ovn-ipsec-host-k8bng_ipsec.d
drwxr-xr-x   4 huirwang  staff   128 Mar  5 18:20 ovn-ipsec-host-kxssh_ipsec.d
drwxr-xr-x   4 huirwang  staff   128 Mar  5 18:20 ovn-ipsec-host-q7vhv_ipsec.d
drwxr-xr-x   8 huirwang  staff   256 Mar  5 18:20 status
drwxr-xr-x   8 huirwang  staff   256 Mar  5 18:20 trafficstatus
drwxr-xr-x  14 huirwang  staff   448 Mar  5 18:20 xfrm

[pperiyasamy]

/assign @sferich888

[openshift-ci-robot]

@pperiyasamy: This pull request references Jira Issue OCPBUGS-52367, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.0) matches configured target version for branch (4.19.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @huiran0826

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

After IPsec is upgraded from 4.14 into later versions, API option for IPsec in networks.operator.openshift.io cluster object continues to have {} in the ipsecConfig and may not be migrated to newer version containing IPsec mode option. This is supported in OCP for backward compatibility, so mg needs fixing to collect IPsec data when API configured with old option.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sferich888]

I worry that what we collect here could collect a large number of files (that can't be limited) if ipsec configuration are in place. Thus on a 500 node cluster the collection time and space requirements to collect this data may be a problem for customers; should we provide a way to limit the collection with a label?

[pperiyasamy]

I worry that what we collect here could collect a large number of files (that can't be limited) if ipsec configuration are in place. Thus on a 500 node cluster the collection time and space requirements to collect this data may be a problem for customers; should we provide a way to limit the collection with a label?

@sferich888 The collected ipsec logs are mostly about collecting ipsec config files and few command outputs (like ip xfrm state, ip xfrm policy, ipsec whack --status and ipsec whack --trafficstatus). of course ip/ipsec command output has results for its peer nodes, but don't think it consumes lot of space though files are too many. The libreswan log (which may consume more space) is collected only for 4.14 deployment. From 4.15, this is collected only from sos report.