Skip to content

Commit 8185c3a

Browse files
mike-hoangmike-hoang
authored
adding support for public and private git providers (#160)
Signed-off-by: Michael Hoang <mhoang@redhat.com>
1 parent e75481b commit 8185c3a

13 files changed

+1981
-165
lines changed

‎README.md

+53-9
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,42 @@ The Devfile Parser library is a Golang module that:
1212
2. writes to the devfile.yaml with the updated data.
1313
3. generates Kubernetes objects for the various devfile resources.
1414
4. defines util functions for the devfile.
15+
5. downloads resources from a parent devfile if specified in the devfile.yaml
16+
17+
## Private repository support
18+
19+
Tokens are required to be set in the following cases:
20+
1. parsing a devfile from a private repository
21+
2. parsing a devfile containing a parent devfile from a private repository [1]
22+
3. parsing a devfile from a private repository containing a parent devfile from a public repository [2]
23+
24+
Set the token for the repository:
25+
```go
26+
parser.ParserArgs{
27+
...
28+
// URL must point to a devfile.yaml
29+
URL: <url-to-devfile-on-supported-git-provider-repo>/devfile.yaml
30+
Token: <repo-personal-access-token>
31+
...
32+
}
33+
```
34+
Note: The url must also be set with a supported git provider repo url.
35+
36+
Minimum token scope required:
37+
1. GitHub: Read access to code
38+
2. GitLab: Read repository
39+
3. Bitbucket: Read repository
40+
41+
Note: To select token scopes for GitHub, a fine-grained token is required.
42+
43+
For more information about personal access tokens:
44+
1. [GitHub docs](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
45+
2. [GitLab docs](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token)
46+
3. [Bitbucket docs](https://support.atlassian.com/bitbucket-cloud/docs/repository-access-tokens/)
47+
48+
[1] Currently, this works under the assumption that the token can authenticate the devfile and the parent devfile; both devfiles are in the same repository.
49+
50+
[2] In this scenario, the token will be used to authenticate the main devfile.
1551

1652
## Usage
1753

@@ -35,7 +71,6 @@ The function documentation can be accessed via [pkg.go.dev](https://pkg.go.dev/g
3571
devfile, variableWarning, err := devfilePkg.ParseDevfileAndValidate(parserArgs)
3672
```
3773

38-
3974
2. To override the HTTP request and response timeouts for a devfile with a parent reference from a registry URL, specify the HTTPTimeout value in the parser arguments
4075
```go
4176
// specify the timeout in seconds
@@ -45,7 +80,6 @@ The function documentation can be accessed via [pkg.go.dev](https://pkg.go.dev/g
4580
}
4681
```
4782

48-
4983
3. To get specific content from devfile
5084
```go
5185
// To get all the components from the devfile
@@ -77,7 +111,7 @@ The function documentation can be accessed via [pkg.go.dev](https://pkg.go.dev/g
77111
},
78112
})
79113
```
80-
114+
81115
4. To get the Kubernetes objects from the devfile, visit [generators.go source file](pkg/devfile/generator/generators.go)
82116
```go
83117
// To get a slice of Kubernetes containers of type corev1.Container from the devfile component containers
@@ -94,7 +128,7 @@ The function documentation can be accessed via [pkg.go.dev](https://pkg.go.dev/g
94128
}
95129
deployment := generator.GetDeployment(deployParams)
96130
```
97-
131+
98132
5. To update devfile content
99133
```go
100134
// To update an existing component in devfile object
@@ -131,20 +165,19 @@ The function documentation can be accessed via [pkg.go.dev](https://pkg.go.dev/g
131165
```go
132166
// If the devfile object has been created with devfile path already set, can simply call WriteYamlDevfile to writes the devfile
133167
err := devfile.WriteYamlDevfile()
134-
135-
168+
136169
// To write to a devfile from scratch
137170
// create a new DevfileData with a specific devfile version
138171
devfileData, err := data.NewDevfileData(devfileVersion)
139172

140173
// set schema version
141174
devfileData.SetSchemaVersion(devfileVersion)
142-
175+
143176
// add devfile content use library APIs
144177
devfileData.AddComponents([]v1.Component{...})
145178
devfileData.AddCommands([]v1.Commands{...})
146179
......
147-
180+
148181
// create a new DevfileCtx
149182
ctx := devfileCtx.NewDevfileCtx(devfilePath)
150183
err = ctx.SetAbsPath()
@@ -154,10 +187,11 @@ The function documentation can be accessed via [pkg.go.dev](https://pkg.go.dev/g
154187
Ctx: ctx,
155188
Data: devfileData,
156189
}
157-
190+
158191
// write to the devfile on disk
159192
err = devfile.WriteYamlDevfile()
160193
```
194+
161195
7. To parse the outerloop Kubernetes/OpenShift component's uri or inline content, call the read and parse functions
162196
```go
163197
// Read the YAML content
@@ -166,6 +200,7 @@ The function documentation can be accessed via [pkg.go.dev](https://pkg.go.dev/g
166200
// Get the Kubernetes resources
167201
resources, err := ParseKubernetesYaml(values)
168202
```
203+
169204
8. By default, the parser will set all unset boolean properties to their spec defined default values. Clients can override this behaviour by specifiying the parser argument `SetBooleanDefaults` to false
170205
```go
171206
setDefaults := false
@@ -174,6 +209,15 @@ The function documentation can be accessed via [pkg.go.dev](https://pkg.go.dev/g
174209
}
175210
```
176211

212+
9. When parsing a devfile that contains a parent reference, if the parent uri is a supported git provider repo url with the correct personal access token, all resources from the parent git repo excluding the parent devfile.yaml will be downloaded to the location of the devfile being parsed. **Note: The URL must point to a devfile.yaml**
213+
```yaml
214+
schemaVersion: 2.2.0
215+
...
216+
parent:
217+
uri: <uri-to-parent-devfile>/devfile.yaml
218+
...
219+
```
220+
177221
## Projects using devfile/library
178222

179223
The following projects are consuming this library as a Golang dependency

‎pkg/devfile/parser/context/content.go

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
//
2-
// Copyright 2022 Red Hat, Inc.
2+
// Copyright 2022-2023 Red Hat, Inc.
33
//
44
// Licensed under the Apache License, Version 2.0 (the "License");
55
// you may not use this file except in compliance with the License.
@@ -69,6 +69,9 @@ func (d *DevfileCtx) SetDevfileContent() error {
6969
if d.url != "" {
7070
// set the client identifier for telemetry
7171
params := util.HTTPRequestParams{URL: d.url, TelemetryClientName: util.TelemetryClientName}
72+
if d.token != "" {
73+
params.Token = d.token
74+
}
7275
data, err = util.DownloadInMemory(params)
7376
if err != nil {
7477
return errors.Wrap(err, "error getting devfile info from url")

‎pkg/devfile/parser/context/context.go

+16-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
//
2-
// Copyright 2022 Red Hat, Inc.
2+
// Copyright 2022-2023 Red Hat, Inc.
33
//
44
// Licensed under the Apache License, Version 2.0 (the "License");
55
// you may not use this file except in compliance with the License.
@@ -45,13 +45,16 @@ type DevfileCtx struct {
4545
// devfile json schema
4646
jsonSchema string
4747

48-
//url path of the devfile
48+
// url path of the devfile
4949
url string
5050

51+
// token is a personal access token used with a private git repo URL
52+
token string
53+
5154
// filesystem for devfile
5255
fs filesystem.Filesystem
5356

54-
// devfile kubernetes components has been coverted from uri to inlined in memory
57+
// devfile kubernetes components has been converted from uri to inlined in memory
5558
convertUriToInlined bool
5659
}
5760

@@ -150,6 +153,16 @@ func (d *DevfileCtx) GetURL() string {
150153
return d.url
151154
}
152155

156+
// GetToken func returns current devfile token
157+
func (d *DevfileCtx) GetToken() string {
158+
return d.token
159+
}
160+
161+
// SetToken sets the token for the devfile
162+
func (d *DevfileCtx) SetToken(token string) {
163+
d.token = token
164+
}
165+
153166
// SetAbsPath sets absolute file path for devfile
154167
func (d *DevfileCtx) SetAbsPath() (err error) {
155168
// Set devfile absolute path

‎pkg/devfile/parser/context/context_test.go

+15-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
//
2-
// Copyright 2022 Red Hat, Inc.
2+
// Copyright 2022-2023 Red Hat, Inc.
33
//
44
// Licensed under the Apache License, Version 2.0 (the "License");
55
// you may not use this file except in compliance with the License.
@@ -83,6 +83,20 @@ func TestPopulateFromInvalidURL(t *testing.T) {
8383
})
8484
}
8585

86+
func TestNewURLDevfileCtx(t *testing.T) {
87+
var (
88+
token = "fake-token"
89+
url = "https://github.com/devfile/registry/blob/main/stacks/go/2.0.0/devfile.yaml"
90+
)
91+
{
92+
d := NewURLDevfileCtx(url)
93+
assert.Equal(t, "https://github.com/devfile/registry/blob/main/stacks/go/2.0.0/devfile.yaml", d.GetURL())
94+
assert.Equal(t, "", d.GetToken())
95+
d.SetToken(token)
96+
assert.Equal(t, "fake-token", d.GetToken())
97+
}
98+
}
99+
86100
func invalidJsonRawContent200() []byte {
87101
return []byte(InvalidDevfileContent)
88102
}

‎pkg/devfile/parser/parse.go

+71-31
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ import (
1919
"context"
2020
"encoding/json"
2121
"fmt"
22+
"github.com/devfile/library/v2/pkg/git"
23+
"github.com/hashicorp/go-multierror"
2224
"io/ioutil"
2325
"net/url"
2426
"os"
@@ -46,6 +48,57 @@ import (
4648
"github.com/pkg/errors"
4749
)
4850

51+
// downloadGitRepoResources is exposed as a global variable for the purpose of running mock tests
52+
var downloadGitRepoResources = func(url string, destDir string, httpTimeout *int, token string) error {
53+
var returnedErr error
54+
55+
gitUrl, err := git.NewGitUrlWithURL(url)
56+
if err != nil {
57+
return err
58+
}
59+
60+
if gitUrl.IsGitProviderRepo() {
61+
if !gitUrl.IsFile || gitUrl.Revision == "" || !strings.Contains(gitUrl.Path, OutputDevfileYamlPath) {
62+
return fmt.Errorf("error getting devfile from url: failed to retrieve %s", url)
63+
}
64+
65+
stackDir, err := os.MkdirTemp("", fmt.Sprintf("git-resources"))
66+
if err != nil {
67+
return fmt.Errorf("failed to create dir: %s, error: %v", stackDir, err)
68+
}
69+
70+
defer func(path string) {
71+
err := os.RemoveAll(path)
72+
if err != nil {
73+
returnedErr = multierror.Append(returnedErr, err)
74+
}
75+
}(stackDir)
76+
77+
if !gitUrl.IsPublic(httpTimeout) {
78+
err = gitUrl.SetToken(token, httpTimeout)
79+
if err != nil {
80+
returnedErr = multierror.Append(returnedErr, err)
81+
return returnedErr
82+
}
83+
}
84+
85+
err = gitUrl.CloneGitRepo(stackDir)
86+
if err != nil {
87+
returnedErr = multierror.Append(returnedErr, err)
88+
return returnedErr
89+
}
90+
91+
dir := path.Dir(path.Join(stackDir, gitUrl.Path))
92+
err = git.CopyAllDirFiles(dir, destDir)
93+
if err != nil {
94+
returnedErr = multierror.Append(returnedErr, err)
95+
return returnedErr
96+
}
97+
}
98+
99+
return nil
100+
}
101+
49102
// ParseDevfile func validates the devfile integrity.
50103
// Creates devfile context and runtime objects
51104
func parseDevfile(d DevfileObj, resolveCtx *resolutionContextTree, tool resolverTools, flattenedDevfile bool) (DevfileObj, error) {
@@ -97,6 +150,8 @@ type ParserArgs struct {
97150
// RegistryURLs is a list of registry hosts which parser should pull parent devfile from.
98151
// If registryUrl is defined in devfile, this list will be ignored.
99152
RegistryURLs []string
153+
// Token is a GitHub, GitLab, or Bitbucket personal access token used with a private git repo URL
154+
Token string
100155
// DefaultNamespace is the default namespace to use
101156
// If namespace is defined under devfile's parent kubernetes object, this namespace will be ignored.
102157
DefaultNamespace string
@@ -129,6 +184,10 @@ func ParseDevfile(args ParserArgs) (d DevfileObj, err error) {
129184
return d, errors.Wrap(err, "the devfile source is not provided")
130185
}
131186

187+
if args.Token != "" {
188+
d.Ctx.SetToken(args.Token)
189+
}
190+
132191
tool := resolverTools{
133192
defaultNamespace: args.DefaultNamespace,
134193
registryURLs: args.RegistryURLs,
@@ -431,17 +490,16 @@ func parseFromURI(importReference v1.ImportReference, curDevfileCtx devfileCtx.D
431490
return DevfileObj{}, fmt.Errorf("failed to resolve parent uri, devfile context is missing absolute url and path to devfile. %s", resolveImportReference(importReference))
432491
}
433492

493+
token := curDevfileCtx.GetToken()
434494
d.Ctx = devfileCtx.NewURLDevfileCtx(newUri)
435-
if strings.Contains(newUri, "raw.githubusercontent.com") {
436-
urlComponents, err := util.GetGitUrlComponentsFromRaw(newUri)
437-
if err != nil {
438-
return DevfileObj{}, err
439-
}
440-
destDir := path.Dir(curDevfileCtx.GetAbsPath())
441-
err = getResourcesFromGit(urlComponents, destDir)
442-
if err != nil {
443-
return DevfileObj{}, err
444-
}
495+
if token != "" {
496+
d.Ctx.SetToken(token)
497+
}
498+
499+
destDir := path.Dir(curDevfileCtx.GetAbsPath())
500+
err = downloadGitRepoResources(newUri, destDir, tool.httpTimeout, token)
501+
if err != nil {
502+
return DevfileObj{}, err
445503
}
446504
}
447505
importReference.Uri = newUri
@@ -450,27 +508,6 @@ func parseFromURI(importReference v1.ImportReference, curDevfileCtx devfileCtx.D
450508
return populateAndParseDevfile(d, newResolveCtx, tool, true)
451509
}
452510

453-
func getResourcesFromGit(gitUrlComponents map[string]string, destDir string) error {
454-
stackDir, err := ioutil.TempDir(os.TempDir(), fmt.Sprintf("git-resources"))
455-
if err != nil {
456-
return fmt.Errorf("failed to create dir: %s, error: %v", stackDir, err)
457-
}
458-
defer os.RemoveAll(stackDir)
459-
460-
err = util.CloneGitRepo(gitUrlComponents, stackDir)
461-
if err != nil {
462-
return err
463-
}
464-
465-
dir := path.Dir(path.Join(stackDir, gitUrlComponents["file"]))
466-
err = util.CopyAllDirFiles(dir, destDir)
467-
if err != nil {
468-
return err
469-
}
470-
471-
return nil
472-
}
473-
474511
func parseFromRegistry(importReference v1.ImportReference, resolveCtx *resolutionContextTree, tool resolverTools) (d DevfileObj, err error) {
475512
id := importReference.Id
476513
registryURL := importReference.RegistryUrl
@@ -839,6 +876,9 @@ func getKubernetesDefinitionFromUri(uri string, d devfileCtx.DevfileCtx) ([]byte
839876
newUri = uri
840877
}
841878
params := util.HTTPRequestParams{URL: newUri}
879+
if d.GetToken() != "" {
880+
params.Token = d.GetToken()
881+
}
842882
data, err = util.DownloadInMemory(params)
843883
if err != nil {
844884
return nil, errors.Wrapf(err, "error getting kubernetes resources definition information")

0 commit comments

Comments
 (0)