Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collecting node logs #517

Merged
merged 18 commits into from
Nov 9, 2021
Merged

Collecting node logs #517

merged 18 commits into from
Nov 9, 2021

Conversation

rluders
Copy link
Contributor

@rluders rluders commented Oct 12, 2021

This PR implements a new data enhancement to collect the node logs from the journal unit (kubelet).

Categories

  • Bugfix
  • Enhancement
  • Backporting
  • Others (CI, Infrastructure, Documentation)

Sample Archive

  • docs/insights-archive-sample/config/node/logs/ip-10-0-135-228.us-east-2.compute.internal.log

Documentation

  • docs/gathered-data.md

Unit Tests

  • pkg/gatherers/clusterconfig/node_logs_test.go (WIP)

Privacy

Yes. There are no sensitive data in the newly collected information.

Changelog

Breaking Changes

No

References

https://issues.redhat.com/browse/CCXDEV-5831
https://bugzilla.redhat.com/show_bug.cgi?id=???
https://access.redhat.com/solutions/???

@openshift-ci
Copy link

openshift-ci bot commented Oct 12, 2021

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 12, 2021
@openshift-ci
Copy link

openshift-ci bot commented Oct 12, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rluders

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 12, 2021
@rluders rluders force-pushed the ccx-5831-collect-kubelet-logs branch from 58b668a to b9dba33 Compare October 15, 2021 13:27
@rluders rluders marked this pull request as ready for review October 21, 2021 08:07
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 21, 2021
@rluders rluders requested a review from tremes October 21, 2021 08:08
@rluders
Copy link
Contributor Author

rluders commented Oct 25, 2021

/retest

@xJustin
Copy link
Contributor

xJustin commented Nov 1, 2021

/label docs-approved

@openshift-ci openshift-ci bot added the docs-approved Signifies that Docs has signed off on this PR label Nov 1, 2021
@rluders
Copy link
Contributor Author

rluders commented Nov 1, 2021

/test e2e-agnostic-upgrade

@sferich888
Copy link

/label px-approved

@openshift-ci openshift-ci bot added the px-approved Signifies that Product Support has signed off on this PR label Nov 1, 2021
@quarckster
Copy link
Contributor

/label qe-approved

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Nov 1, 2021
@rluders rluders force-pushed the ccx-5831-collect-kubelet-logs branch from 37fc99d to d79dff6 Compare November 2, 2021 09:58
@rluders rluders requested a review from tremes November 2, 2021 09:58
@rluders rluders requested a review from tremes November 3, 2021 13:51
Serhii1011010
Serhii1011010 reviewed Nov 3, 2021
View reviewed changes
pkg/gatherers/clusterconfig/node_logs.go
"io"
"strconv"

"github.com/openshift/insights-operator/pkg/gatherers/common"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we had a convention where we have 3 groups of imports: from the standard library, external packages, operator's code

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that I would group imports like this:

    "bufio"
	"bytes"
	"compress/gzip"
	"context"
	"fmt"
	"io"
	"strconv"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/klog/v2"
	"k8s.io/client-go/rest"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	corev1client "k8s.io/client-go/kubernetes/typed/core/v1"

	"github.com/openshift/insights-operator/pkg/gatherers/common"
	"github.com/openshift/insights-operator/pkg/recorder"
	"github.com/openshift/insights-operator/pkg/utils/marshal"
	"github.com/openshift/insights-operator/pkg/record"

but maybe we don't have any convention for it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't...but it's true that it's grouped differently than here. I mean into 3 groups (as Serhii mentioned) where the last two somehow intersect. Isn't this part of go fmt or our linting??

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha! I can check if it's possible to add this validation to the lining. I think that the pre-commit should format the code before pushing it. I need to check why it is not happening.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. So, a quick investigation point that we have two different styles of coding format to import: check here and here. If we want to use the one by goimports I'll check how to do linting it and add it to the pre-commit script later. For now, the better approach is to configure the IDE to do it.

@rluders rluders force-pushed the ccx-5831-collect-kubelet-logs branch from 3f127dc to f09b404 Compare November 5, 2021 16:56
@rluders rluders requested a review from Serhii1011010 November 5, 2021 16:59
@rluders
Copy link
Contributor Author

rluders commented Nov 8, 2021

/test e2e-agnostic-upgrade

@rluders rluders requested a review from tremes November 8, 2021 12:49
tremes
tremes reviewed Nov 8, 2021
View reviewed changes
pkg/gatherers/clusterconfig/node_logs.go
func nodeLogString(ctx context.Context, req *rest.Request) (string, error) {
in, err := req.Stream(ctx)
if err != nil {
return "", err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can probably lead to an empty file/record, but it's probably not a big deal.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that It is possible. Don't you think that it should be handled by the recorder? I mean, the recorder should not record empty files. Right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably...

@tremes
Copy link
Contributor

tremes commented Nov 8, 2021

Reviewed and tested few times. Thank you!

@tremes
Copy link
Contributor

tremes commented Nov 8, 2021

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2021
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

16 similar comments
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit e0497d0 into openshift:master Nov 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tremes tremes tremes left review comments

@mfojtik mfojtik Awaiting requested review from mfojtik

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@Serhii1011010 Serhii1011010 Awaiting requested review from Serhii1011010

Assignees

@tremes tremes

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. docs-approved Signifies that Docs has signed off on this PR lgtm Indicates that a PR is ready to be merged. px-approved Signifies that Product Support has signed off on this PR qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@rluders @xJustin @sferich888 @quarckster @tremes @openshift-bot @Serhii1011010 @openshift-merge-robot