Skip to content

Commit 8173cc6

Browse files
resolved upload path issue (#3)
1 parent f4e047e commit 8173cc6

22 files changed

+1181
-964
lines changed

CHANGELOG.md

+5-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# Storj-Zenko Changelog
1+
# Connector-Zenko Changelog
2+
3+
## [1.0.5] - 17-08-2020
4+
### Changelog:
5+
* Resolved upload path issue.
26

37
## [1.0.5] - 22-05-2020
48
### Changelog:

LICENSE

+202-202
Large diffs are not rendered by default.

README.assets/Home.md

+7-7
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ Inside the ```./config``` directory a ```storj_config.json``` file, with Storj n
2020
* satelliteURL - Storj Satellite URL
2121
* encryptionPassphrase - Storj Encryption Passphrase.
2222
* bucket - Name of the bucket to upload data into.
23-
* uploadPath - Path on Storj Bucket to store data (optional) or "/"
23+
* uploadPath - Path on Storj Bucket to store data (optional) or "" or "/" (mandatory)
2424
* serializedAccess - Serialized access shared while uploading data used to access bucket without API Key
2525
* allowDownload - Set true to create serialized access with restricted download
2626
* allowUpload - Set true to create serialized access with restricted upload
@@ -35,29 +35,29 @@ Zenko files are iterated through and streamed in 32KB chunks to the Storj networ
3535
The following flags can be used with the ```store``` command:
3636

3737
* ```accesskey``` - Connects to the Storj network using a serialized access key instead of an API key, satellite url and encryption passphrase .
38-
* ```shared``` - Generates a restricted shareable serialized access with the restrictions specified in the Storj configuration file.
38+
* ```share``` - Generates a restricted shareable serialized access with the restrictions specified in the Storj configuration file.
3939

4040
Once you have built the project you can run the following:
4141

4242
##### Get help
4343
```
44-
$ ./storj-zenko --help
44+
$ ./connector-zenko --help
4545
```
4646
##### Check version
4747
```
48-
$ ./storj-zenko version
48+
$ ./connector-zenko version
4949
```
5050
##### Create backup from Zenko and upload them to Storj
5151
```
52-
$ ./storj-zenko store --zenko <path_to_zenko_config_file> --storj <path_to_storj_config_file>
52+
$ ./connector-zenko store --zenko <path_to_zenko_config_file> --storj <path_to_storj_config_file>
5353
```
5454
##### Create backup files from Zenko and upload them to Storj bucket using Access Key
5555
```
56-
$ ./storj-zenko store --accesskey
56+
$ ./connector-zenko store --accesskey
5757
```
5858
##### Create backup files from Zenko and upload them to Storj and generate a Shareable Access Key based on restrictions in ```storj_config.json```.
5959
```
60-
$ ./storj-zenko store --share
60+
$ ./connector-zenko store --share
6161
```
6262
6363
## Testing

README.md

+10-10
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
# storj-zenko connector (uplink v1.0.5)
1+
# connector-zenko (uplink v1.0.5)
22

3-
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/b76faa16299a4236ad85baa9f39b7294)](https://app.codacy.com/gh/storj-thirdparty/connector-zenko?utm_source=github.com&utm_medium=referral&utm_content=storj-thirdparty/connector-zenko&utm_campaign=Badge_Grade_Dashboard)
3+
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/b76faa16299a4236ad85baa9f39b7294)](https://app.codacy.com/gh/storj-thirdparty/connector-zenko?utm_source=github.com&utm_medium=referral&utm_content=storj-thirdparty/connector-zenko&utm_campaign=Badge_Grade_Dashboard)
44
[![Go Report Card](https://goreportcard.com/badge/github.com/storj-thirdparty/connector-zenko)](https://goreportcard.com/report/github.com/storj-thirdparty/connector-zenko)
55
![Cloud Build](https://storage.googleapis.com/storj-utropic-services-badges/builds/connector-zenko/branches/master.svg)
66

@@ -11,12 +11,12 @@ Command line application (on Windows/Linux/Mac) for taking data backup from Zenk
1111

1212
Zenko is infrastructure software to control Data in Multi-Cloud IT Environments without cloud lock-in and has features such as enabling unified data management from anywhere through a secure cloud portal, providing a single S3 endpoint through which data can be stored, retrieved and searched across any location.
1313

14-
### Features of storj-zenko:
14+
### Features of connector-zenko:
1515
* Connects S3 compatible cloud storages (e.g. Amazon AWS, Azure Blob, Google Cloud Storage, Wasabi) to the Zenko instance for backing up their data to StorJ V3 network.
1616
* Upload any type of data from Zenko to Storj (single or multiple at once) whether it is a folder, document, data file, image, video, etc.
1717
```
1818
Usage:
19-
storj-zenko [command] <flags>
19+
connector-zenko [command] <flags>
2020
2121
Available Commands:
2222
help Help about any command
@@ -37,32 +37,32 @@ To build from scratch, [install the latest Go](https://golang.org/doc/install#in
3737
#### Option #1: clone this repo (most common)
3838
To clone the repo
3939
```
40-
git clone https://github.com/storj-thirdparty/storj-zenko.git
40+
git clone https://github.com/storj-thirdparty/connector-zenko.git
4141
```
4242
Then, build the project using the following:
4343
```
44-
cd storj-zenko
44+
cd connector-zenko
4545
go build
4646
```
4747
#### Option #2: go get into your gopath
4848
To download the project inside your GOPATH use the following command:
4949
```
50-
go get github.com/storj-thirdparty/storj-zenko
50+
go get github.com/storj-thirdparty/connector-zenko
5151
```
5252
## Run (short version)
5353
Once you have built the project run the following commands as per your requirement:
5454

5555
##### Get help
5656
```
57-
$ ./storj-zenko --help
57+
$ ./connector-zenko --help
5858
```
5959
##### Check version
6060
```
61-
$ ./storj-zenko --version
61+
$ ./connector-zenko --version
6262
```
6363
##### Create backup from Zenko and upload to Storj
6464
```
65-
$ ./storj-zenko store
65+
$ ./connector-zenko store
6666
```
6767
## Documentation
6868

cloudbuild.yaml

+29-29
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,29 @@
1-
steps:
2-
# see https://www.npmjs.com/package/editorconfig-checker
3-
- id: eclint
4-
name: "e53e225/editorconfig-checker"
5-
- id: go_version
6-
name: "gcr.io/cloud-builders/go"
7-
args: ["version"]
8-
env: ["GOPATH=."]
9-
- id: go_linter
10-
name: "golangci/golangci-lint"
11-
args: ["golangci-lint","run"]
12-
env: ["GO111MODULE=on", "CGO_ENABLED=0"]
13-
- id: go_security
14-
name: "securego/gosec"
15-
args: [ "./..." ]
16-
- id: go_build
17-
name: "mirror.gcr.io/library/golang"
18-
env: ['GO111MODULE=on']
19-
args: ['go', 'build', './...']
20-
- id: doc_check
21-
name: "ubuntu"
22-
args: ['bash', './test/doc-check.sh']
23-
# name: "gcr.io/cloud-builders/go"
24-
# args: ["build"]
25-
# env: ["GOPATH=.","GO111MODULE=on", "CGO_ENABLED=0"]
26-
# - id: go_test
27-
# name: "gcr.io/cloud-builders/go"
28-
# args: ["test","helloworld"]
29-
# env: ["GOPATH=."]
1+
steps:
2+
# see https://www.npmjs.com/package/editorconfig-checker
3+
- id: eclint
4+
name: "e53e225/editorconfig-checker"
5+
- id: go_version
6+
name: "gcr.io/cloud-builders/go"
7+
args: ["version"]
8+
env: ["GOPATH=."]
9+
- id: go_linter
10+
name: "golangci/golangci-lint"
11+
args: ["golangci-lint","run"]
12+
env: ["GO111MODULE=on", "CGO_ENABLED=0"]
13+
- id: go_security
14+
name: "securego/gosec"
15+
args: [ "./..." ]
16+
- id: go_build
17+
name: "mirror.gcr.io/library/golang"
18+
env: ['GO111MODULE=on']
19+
args: ['go', 'build', './...']
20+
- id: doc_check
21+
name: "ubuntu"
22+
args: ['bash', './test/doc-check.sh']
23+
# name: "gcr.io/cloud-builders/go"
24+
# args: ["build"]
25+
# env: ["GOPATH=.","GO111MODULE=on", "CGO_ENABLED=0"]
26+
# - id: go_test
27+
# name: "gcr.io/cloud-builders/go"
28+
# args: ["test","helloworld"]
29+
# env: ["GOPATH=."]

cmd/root.go

+27-27
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,27 @@
1-
package cmd
2-
3-
import (
4-
"fmt"
5-
"github.com/spf13/cobra"
6-
"os"
7-
)
8-
9-
// rootCmd represents the base command when called without any subcommands
10-
var rootCmd = &cobra.Command{
11-
Use: "connector-zenko",
12-
Short: "Backup Zenko data to the decentralized Storj network.",
13-
Long: `connector-zenko - Backup your Zenko data to the decentralized Storj network.`,
14-
}
15-
16-
// Execute adds all child commands to the root command and sets flags appropriately.
17-
// This is called by main.main(). It only needs to happen once to the rootCmd.
18-
func Execute() {
19-
20-
if err := rootCmd.Execute(); err != nil {
21-
fmt.Println(err)
22-
os.Exit(1)
23-
}
24-
}
25-
26-
func init() {
27-
}
1+
package cmd
2+
3+
import (
4+
"fmt"
5+
"github.com/spf13/cobra"
6+
"os"
7+
)
8+
9+
// rootCmd represents the base command when called without any subcommands
10+
var rootCmd = &cobra.Command{
11+
Use: "connector-zenko",
12+
Short: "Backup Zenko data to the decentralized Storj network.",
13+
Long: `connector-zenko - Backup your Zenko data to the decentralized Storj network.`,
14+
}
15+
16+
// Execute adds all child commands to the root command and sets flags appropriately.
17+
// This is called by main.main(). It only needs to happen once to the rootCmd.
18+
func Execute() {
19+
20+
if err := rootCmd.Execute(); err != nil {
21+
fmt.Println(err)
22+
os.Exit(1)
23+
}
24+
}
25+
26+
func init() {
27+
}

cmd/store.go

+63-63
Original file line numberDiff line numberDiff line change
@@ -1,63 +1,63 @@
1-
package cmd
2-
3-
import (
4-
"fmt"
5-
6-
"github.com/spf13/cobra"
7-
)
8-
9-
// storeCmd represents the store command
10-
var storeCmd = &cobra.Command{
11-
Use: "store",
12-
Short: "Command to upload data to storj V3 network.",
13-
Long: `Command to connect and transfer ALL tables from a desired Zenko instance to given Storj Bucket.`,
14-
Run: zenkoStore,
15-
}
16-
17-
func init() {
18-
19-
// Setup the store command with its flags.
20-
rootCmd.AddCommand(storeCmd)
21-
var defaultZenkoFile string
22-
var defaultStorjFile string
23-
storeCmd.Flags().BoolP("accesskey", "a", false, "Connect to storj using access key(default connection method is by using API Key).")
24-
storeCmd.Flags().BoolP("share", "s", false, "For generating share access of the uploaded backup file.")
25-
storeCmd.Flags().StringVarP(&defaultZenkoFile, "zenko", "z", "./config/zenko_property.json", "full filepath contaning Zenko configuration.")
26-
storeCmd.Flags().StringVarP(&defaultStorjFile, "storj", "u", "./config/storj_config.json", "full filepath contaning storj V3 configuration.")
27-
}
28-
29-
func zenkoStore(cmd *cobra.Command, args []string) {
30-
31-
// Process arguments from the CLI.
32-
zenkoConfigfilePath, _ := cmd.Flags().GetString("zenko")
33-
fullFileNameStorj, _ := cmd.Flags().GetString("storj")
34-
useAccessKey, _ := cmd.Flags().GetBool("accesskey")
35-
useAccessShare, _ := cmd.Flags().GetBool("share")
36-
37-
// Read Zenko instance's configurations from an external file and create an Zenko configuration object.
38-
configZenko := LoadZenkoProperty(zenkoConfigfilePath)
39-
40-
// Read storj network configurations from and external file and create a storj configuration object.
41-
storjConfig := LoadStorjConfiguration(fullFileNameStorj)
42-
43-
// Connect to storj network using the specified credentials.
44-
access, project := ConnectToStorj(fullFileNameStorj, storjConfig, useAccessKey)
45-
46-
// Establish connection with Zenko and get io.Reader implementor.
47-
zenkoReader := ConnectToZenko(configZenko)
48-
49-
// Fetch all file names and their corresponding Reader Object for upload
50-
uploadPathNames, zenkoObjectReaders := GetFileAndObjectNames(zenkoReader)
51-
52-
fmt.Printf("\nInitiating back-up.\n")
53-
// Fetch all backup files from Zenko instance and simultaneously store them into desired Storj bucket.
54-
for i := 0; i < len(uploadPathNames); i++ {
55-
UploadData(project, storjConfig, uploadPathNames[i], zenkoObjectReaders[i])
56-
}
57-
fmt.Printf("\nBack-up complete.\n\n")
58-
59-
// Create restricted shareable serialized access if share is provided as argument.
60-
if useAccessShare {
61-
ShareAccess(access, storjConfig)
62-
}
63-
}
1+
package cmd
2+
3+
import (
4+
"fmt"
5+
6+
"github.com/spf13/cobra"
7+
)
8+
9+
// storeCmd represents the store command
10+
var storeCmd = &cobra.Command{
11+
Use: "store",
12+
Short: "Command to upload data to storj V3 network.",
13+
Long: `Command to connect and transfer ALL tables from a desired Zenko instance to given Storj Bucket.`,
14+
Run: zenkoStore,
15+
}
16+
17+
func init() {
18+
19+
// Setup the store command with its flags.
20+
rootCmd.AddCommand(storeCmd)
21+
var defaultZenkoFile string
22+
var defaultStorjFile string
23+
storeCmd.Flags().BoolP("accesskey", "a", false, "Connect to storj using access key(default connection method is by using API Key).")
24+
storeCmd.Flags().BoolP("share", "s", false, "For generating share access of the uploaded backup file.")
25+
storeCmd.Flags().StringVarP(&defaultZenkoFile, "zenko", "z", "./config/zenko_property.json", "full filepath contaning Zenko configuration.")
26+
storeCmd.Flags().StringVarP(&defaultStorjFile, "storj", "u", "./config/storj_config.json", "full filepath contaning storj V3 configuration.")
27+
}
28+
29+
func zenkoStore(cmd *cobra.Command, args []string) {
30+
31+
// Process arguments from the CLI.
32+
zenkoConfigfilePath, _ := cmd.Flags().GetString("zenko")
33+
fullFileNameStorj, _ := cmd.Flags().GetString("storj")
34+
useAccessKey, _ := cmd.Flags().GetBool("accesskey")
35+
useAccessShare, _ := cmd.Flags().GetBool("share")
36+
37+
// Read Zenko instance's configurations from an external file and create an Zenko configuration object.
38+
configZenko := LoadZenkoProperty(zenkoConfigfilePath)
39+
40+
// Read storj network configurations from and external file and create a storj configuration object.
41+
storjConfig := LoadStorjConfiguration(fullFileNameStorj)
42+
43+
// Connect to storj network using the specified credentials.
44+
access, project := ConnectToStorj(fullFileNameStorj, storjConfig, useAccessKey)
45+
46+
// Establish connection with Zenko and get io.Reader implementor.
47+
zenkoReader := ConnectToZenko(configZenko)
48+
49+
// Fetch all file names and their corresponding Reader Object for upload
50+
uploadPathNames, zenkoObjectReaders := GetFileAndObjectNames(zenkoReader)
51+
52+
fmt.Printf("\nInitiating back-up.\n")
53+
// Fetch all backup files from Zenko instance and simultaneously store them into desired Storj bucket.
54+
for i := 0; i < len(uploadPathNames); i++ {
55+
UploadData(project, storjConfig, uploadPathNames[i], zenkoObjectReaders[i])
56+
}
57+
fmt.Printf("\nBack-up complete.\n\n")
58+
59+
// Create restricted shareable serialized access if share is provided as argument.
60+
if useAccessShare {
61+
ShareAccess(access, storjConfig)
62+
}
63+
}

0 commit comments

Comments
 (0)