fix: pvc mounted on pods for cache handling #2912
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
| GitGuardian id | Secret | Commit | Filename | |
|---|---|---|---|---|
| 141558 | Generic High Entropy Secret | b3c0a2e | scripts/devtron-reference-helm-charts/deployment-chart_1-1-0/env-values.yaml | View secret |
| 141558 | Generic High Entropy Secret | b3c0a2e | scripts/devtron-reference-helm-charts/reference-chart_4-17-0/env-values.yaml | View secret |
| 2763127 | Generic High Entropy Secret | b3c0a2e | scripts/devtron-reference-helm-charts/deployment-chart_1-1-0/secrets-test-values.yaml | View secret |
| 2763127 | Generic High Entropy Secret | b3c0a2e | scripts/devtron-reference-helm-charts/reference-chart_4-17-0/secrets-test-values.yaml | View secret |
| 5236769 | Generic High Entropy Secret | b3c0a2e | scripts/devtron-reference-helm-charts/deployment-chart_1-1-0/test-values.json | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Our GitHub checks need improvements? Share your feedbacks!
pkg/pipeline/WorkflowService.go
Outdated
| @@ -112,6 +114,9 @@ type WorkflowRequest struct { | |||
| CiBuildDockerMtuValue int `json:"ciBuildDockerMtuValue"` | |||
| IgnoreDockerCachePush bool `json:"ignoreDockerCachePush"` | |||
| IgnoreDockerCachePull bool `json:"ignoreDockerCachePull"` | |||
| CacheInvalidate bool `json:"cacheInvalidate"` | |||
| IsPvcMounted bool `json:"IsPvcMounted"` | |||
| PvcCachePath string `json:"pvcCachePath"` | |||
There was a problem hiding this comment.
why this flag is needed
There was a problem hiding this comment.
I didn't introduce the flag, but it's not being used anywhere so removed it
pkg/pipeline/WorkflowService.go
Outdated
| workflowRequest.IgnoreDockerCachePull = true | ||
| } | ||
| if impl.ciConfig.IgnoreDockerCacheForCI && workflowRequest.CacheInvalidate { | ||
| workflowRequest.IsPvcMounted = false |
There was a problem hiding this comment.
this is wrong, whats the intent of it.
There was a problem hiding this comment.
was just doing some checks, didn't need it so removed it
pkg/pipeline/CiConfig.go
Outdated
| @@ -79,6 +79,9 @@ type CiConfig struct { | |||
| CiRunnerDockerMTUValue int `env:"CI_RUNNER_DOCKER_MTU_VALUE" envDefault:"-1"` | |||
| IgnoreDockerCacheForCI bool `env:"CI_IGNORE_DOCKER_CACHE"` | |||
| VolumeMountsForCiJson string `env:"CI_VOLUME_MOUNTS_JSON"` | |||
| BuildPvcCachePath string `env:"BUILD_PVC_CACHE_PATH" envDefault:"/devtroncd-cache"` | |||
There was a problem hiding this comment.
rename these Params, BUILD_PVC_CACHE_PATH, DEFAULT_PVC_CACHE_PATH, something related to intent
|
Kudos, SonarCloud Quality Gate passed!
|
kripanshdevtron
approved these changes
Feb 2, 2023
ShashwatDadhich
added a commit
that referenced
this pull request
Feb 12, 2023
* fix: Installed apps rbac optimisation (#2918) * rbac enforced in batch * minor restructuring * minor fix * code comments added * throwing unauthorised error * fix: pvc mounted on pods for cache handling (#2912) * first cut pvc for ci * isPvcMounted flag introduced * cache invalidate and global invalidate handling * cache path made configurable * pvc mounted false * PVC mounted at two new paths for storing build and buildx cache * mnor fix * removed redundant objects from struct * chenged names of env variables for paths * buildx cache path restored after doing POC --------- Co-authored-by: Prashant Ghildiyal <prashant@devtron.ai> Co-authored-by: Kripansh <kripansh@devtron.ai> * updated log type for pipeline status methods (#2926) * docs: minor updates ingress (#2931) * minor updates ingress * minor updates ingress * finished on time update fix (#2932) Co-authored-by: Kripansh <kripansh@devtron.ai> * fix delete pipeline app status check (#2939) * feat: added option to propagate custom tag to k8s resources for that application (#2841) * wip * sql file renamed * propagation in labels support for create app api * validating labels if propagate is true * refactoring * sql file no. changed * removed required field from bool * sql file no. changed * removed cascade * alter app lable key * db migration seq fix for global tags --------- Co-authored-by: vikramdevtron <vikram@devtron.ai> * excluding inactive ci-pipelin-material from pre-cd stage (#2920) * feat: cluster bearer token hide from dashboard (#2894) * cluster token config removed mandatory * api spec added for cluster update and create, and cluster list api changes for token * fix check config for cluster token * docs: mount pvc (#2941) * added pvc * added pvc * added pvc * added pvc * added pvc * feat: Resource browser child ref (#2913) * child rbac handling init commit * resource manifest validate handling added * handle rbac case * wire gen fix * gvk passed and ap resource handling * code cleaning * dead code cleaning * removed unused func * fix: k8s log stream cpu issue (#2929) * updated buffer size * refactored code for getting logs from k8s * updated Dockerfile * updated Dockerfile * downgraded go version * removed redundant log * sorting_changes (#2935) * support given for cloning linked ci pipelines via workflow cloning API (#2944) * docs: Added preset plugins (#2904) * added preset plugins * added preset plugins * Added semgrep, codacy * updates * updated screenshots * updates --------- Co-authored-by: Gireesh Naidu <111440205+gireesh-devtron@users.noreply.github.com> Co-authored-by: Prakash Kumar <prakash.kumar@devtron.ai> Co-authored-by: Prashant Ghildiyal <prashant@devtron.ai> Co-authored-by: Kripansh <kripansh@devtron.ai> Co-authored-by: kartik-579 <84493919+kartik-579@users.noreply.github.com> Co-authored-by: SNe789 <113748500+SNe789@users.noreply.github.com> Co-authored-by: Vikram <73224103+vikramdevtron@users.noreply.github.com> Co-authored-by: Manish Agrawal <85211469+manish-agrawal-ai@users.noreply.github.com> Co-authored-by: vikramdevtron <vikram@devtron.ai> Co-authored-by: kripanshdevtron <107392309+kripanshdevtron@users.noreply.github.com> Co-authored-by: Dhananjay Sharma <124036328+dhananjay0601@users.noreply.github.com>
ShashwatDadhich
added a commit
that referenced
this pull request
Mar 20, 2023
* Made chenges * Made chenges * Made chenges * Clone job function added * fixed typo * fixed typo * Merging main to job-feature-branch (#2955) * fix: Installed apps rbac optimisation (#2918) * rbac enforced in batch * minor restructuring * minor fix * code comments added * throwing unauthorised error * fix: pvc mounted on pods for cache handling (#2912) * first cut pvc for ci * isPvcMounted flag introduced * cache invalidate and global invalidate handling * cache path made configurable * pvc mounted false * PVC mounted at two new paths for storing build and buildx cache * mnor fix * removed redundant objects from struct * chenged names of env variables for paths * buildx cache path restored after doing POC --------- Co-authored-by: Prashant Ghildiyal <prashant@devtron.ai> Co-authored-by: Kripansh <kripansh@devtron.ai> * updated log type for pipeline status methods (#2926) * docs: minor updates ingress (#2931) * minor updates ingress * minor updates ingress * finished on time update fix (#2932) Co-authored-by: Kripansh <kripansh@devtron.ai> * fix delete pipeline app status check (#2939) * feat: added option to propagate custom tag to k8s resources for that application (#2841) * wip * sql file renamed * propagation in labels support for create app api * validating labels if propagate is true * refactoring * sql file no. changed * removed required field from bool * sql file no. changed * removed cascade * alter app lable key * db migration seq fix for global tags --------- Co-authored-by: vikramdevtron <vikram@devtron.ai> * excluding inactive ci-pipelin-material from pre-cd stage (#2920) * feat: cluster bearer token hide from dashboard (#2894) * cluster token config removed mandatory * api spec added for cluster update and create, and cluster list api changes for token * fix check config for cluster token * docs: mount pvc (#2941) * added pvc * added pvc * added pvc * added pvc * added pvc * feat: Resource browser child ref (#2913) * child rbac handling init commit * resource manifest validate handling added * handle rbac case * wire gen fix * gvk passed and ap resource handling * code cleaning * dead code cleaning * removed unused func * fix: k8s log stream cpu issue (#2929) * updated buffer size * refactored code for getting logs from k8s * updated Dockerfile * updated Dockerfile * downgraded go version * removed redundant log * sorting_changes (#2935) * support given for cloning linked ci pipelines via workflow cloning API (#2944) * docs: Added preset plugins (#2904) * added preset plugins * added preset plugins * Added semgrep, codacy * updates * updated screenshots * updates --------- Co-authored-by: Gireesh Naidu <111440205+gireesh-devtron@users.noreply.github.com> Co-authored-by: Prakash Kumar <prakash.kumar@devtron.ai> Co-authored-by: Prashant Ghildiyal <prashant@devtron.ai> Co-authored-by: Kripansh <kripansh@devtron.ai> Co-authored-by: kartik-579 <84493919+kartik-579@users.noreply.github.com> Co-authored-by: SNe789 <113748500+SNe789@users.noreply.github.com> Co-authored-by: Vikram <73224103+vikramdevtron@users.noreply.github.com> Co-authored-by: Manish Agrawal <85211469+manish-agrawal-ai@users.noreply.github.com> Co-authored-by: vikramdevtron <vikram@devtron.ai> Co-authored-by: kripanshdevtron <107392309+kripanshdevtron@users.noreply.github.com> Co-authored-by: Dhananjay Sharma <124036328+dhananjay0601@users.noreply.github.com> * made changes to the queries * Update AppListingRepository.go * Update version.go * Update version.go * Update 1_insert.up.sql * Update 1_insert.up.sql * Update AppCloneService.go * Update AppCloneService.go * Uodated app listing api * Uodated app listing api restHandler * Updated app listing api restHandler * Updated app listing api service layer * Updated app listing api service layer * Updated app listing api service layer * Updated app listing api service layer * Updated app listing api service layer * Updated app listing api service layer * MAde the patch-ci-pipeline api * MAde the patch-ci-pipeline api * Reduced Code duplicacy * Corrected the code of clone job. * Corrected the code of clone job. * Updated the api specs * Updated the api specs * Updated the api specs * Updated the api specs * Updated the api specs * Copied the earlier deleted files * Copied the earlier deleted files * Update jobs.yaml * Updated the api end points and wrote the api specs. * Updated the api end points and wrote the api specs. * Updated the api end points and wrote the api specs. * Updated the api end points and wrote the api specs. * wire err fix * Wrote the sql migrate down command. * Removed the extra api endpoints. * Removed the commented code * Removed the commented code. * Update version.go * Added logger wherever required. * Fixed a bug. * Updated the api specs. * Added the user authentications * Added the user authentications * Added the user authentications * Added the user authentications * Added the user authentications * Added the user authentications * Added the user authentications for triggerCiPipeline * Added the user authentications for triggerCiPipeline * Updated the api specs and rbac checks * Updated the api specs and rbac checks * Updated the stage status api in case of jobs. * Updated the job autocomplete api. * Made the status changes * Fixed the bug of last succeeded time * undid the reverted changes * Job count added * Job count corrected * Job count bug fixed * Sorting order corrected * Sorting order DESC added * Corrected the app overview page deleted pipeline bug. * Corrected the app overview page deleted pipeline bug. * added column is_artifact_uploaded to the ci_artifact table. * Added the check of deleted pipeline. * Added the check of deleted pipeline. * Added the check of deleted pipeline. * Clone app * Multiple git repo patch * removed the commented lines. * Minor bug fixes * Delete Git material bug fix. * Fixed minor bug * Wrote the sql down command. * minor fix * resolved the comments of PR. * fixed a minor bug * Changed from app_store to app_type * Changed from app_store to app_type * changes as mentioned in the PR * Checked the ci_artifacts * Changes done as mentioned in the PR * Changes done as mentioned in the PR * data optimization added * Fixed minor bug * Fixed minor bug * Jobs sql up number update * Bug fix * Bug fix * Bug fix * Bug fix * Bug fix * Bug fix * Bug fix * Artifact logger added * Additional loggers removed * Logger No tasks configured for this job added * Removed No tasks configured for this job added * Added a validation for AppType * Added comments for validation for AppType * Added a default value to IsArtifactUploaded * Removed default value to IsArtifactUploaded * Added false to isArtifactUploaded * Removed false to isArtifactUploaded * Optimized sql up and down. * changed the sql number * Optimized the sql query. * Removed the history tables from sql down. * Removed the history tables from sql down. --------- Co-authored-by: ShashwatDevtron <shashwat@devtron.ai> Co-authored-by: Gireesh Naidu <111440205+gireesh-devtron@users.noreply.github.com> Co-authored-by: Prakash Kumar <prakash.kumar@devtron.ai> Co-authored-by: Prashant Ghildiyal <prashant@devtron.ai> Co-authored-by: Kripansh <kripansh@devtron.ai> Co-authored-by: kartik-579 <84493919+kartik-579@users.noreply.github.com> Co-authored-by: SNe789 <113748500+SNe789@users.noreply.github.com> Co-authored-by: Vikram <73224103+vikramdevtron@users.noreply.github.com> Co-authored-by: Manish Agrawal <85211469+manish-agrawal-ai@users.noreply.github.com> Co-authored-by: vikramdevtron <vikram@devtron.ai> Co-authored-by: kripanshdevtron <107392309+kripanshdevtron@users.noreply.github.com> Co-authored-by: Dhananjay Sharma <124036328+dhananjay0601@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
For shorted build time, PVC has been added in pods at app level and pipeline level.
Fixes #2924
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Checklist:
Does this PR introduce a user-facing change?