Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: hms-dbmi/pic-sure-services
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.2.0
Choose a base ref
...
head repository: hms-dbmi/pic-sure-services
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: release
Choose a head ref

Commits on Mar 13, 2024

  1. [ALS-5571] Add assume role policy to user

    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    7afcd86 View commit details
  2. [ALS-0000] Fix home inst name

    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    0e512d5 View commit details
  3. [ALS-5796] Update logging bucket pems

    - Open up permissions to allow logging service to reach bucket
    - Switch encryption (logging req)
    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    388448b View commit details
  4. [ALS-5796] Create userless CFT

    - Some sites already have a means to authenticate established
    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    7b71116 View commit details
  5. Run tests for PRs

    - Pin mysql dep more aggresively
    - Add GH action to run tests
    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    d36a90f View commit details
  6. [ALS-5832] Upload geno, pheno separately

    - Split upload into two requests
    - Add test coverage to upload service
    - Add a bit of load balancing to uploads so they can't be spammed
    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    cc024e4 View commit details
  7. Copy the full SHA
    5437209 View commit details
  8. [ALS-0000] Release locks on error

    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    d392549 View commit details
  9. Docker tag for 1.2.0 release

    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    2a2ed9a View commit details
  10. Remove build

    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    352bc45 View commit details
  11. Fully enable proxy

    - pull proxy info from system vars
    - use for sdk and hpds client
    - pass system vars to entrypoint
    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    27d499a View commit details
  12. v1.2.1 for yml

    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    0d24146 View commit details
  13. v1.2.3 for yml

    Luke Sikina authored and Luke-Sikina committed Mar 13, 2024
    Copy the full SHA
    b916a35 View commit details

Commits on Jun 11, 2024

  1. [ALS-0000] - Restart containers when ec2s get bonked

    Luke Sikina authored and Luke-Sikina committed Jun 11, 2024
    Copy the full SHA
    5a7c629 View commit details
  2. [ALS-6666] - Fix upload root file system config path

    Luke Sikina authored and Luke-Sikina committed Jun 11, 2024
    Copy the full SHA
    e489b29 View commit details
  3. [ALS-6506] Make GH action to sync release with master

    Luke Sikina authored and Luke-Sikina committed Jun 11, 2024
    Copy the full SHA
    7880192 View commit details

Commits on Jun 12, 2024

  1. [ALS-0000] mv

    Luke Sikina authored and Luke-Sikina committed Jun 12, 2024
    Copy the full SHA
    edac3fd View commit details
  2. This is a small commit going directly into main

    Luke Sikina authored and Luke-Sikina committed Jun 12, 2024
    Copy the full SHA
    3843535 View commit details
  3. specify secret key for action

    Luke Sikina authored and Luke-Sikina committed Jun 12, 2024
    Copy the full SHA
    2071dd2 View commit details
  4. trying to force push

    Luke Sikina authored and Luke-Sikina committed Jun 12, 2024
    Copy the full SHA
    fd945ac View commit details
  5. small change

    Luke Sikina authored and Luke-Sikina committed Jun 12, 2024
    Copy the full SHA
    3d12116 View commit details
  6. Update sync_release.yml

    Luke-Sikina committed Jun 12, 2024
    Copy the full SHA
    7d064ba View commit details
  7. Update README.md

    I added a smiley face
    Luke-Sikina committed Jun 12, 2024
    Copy the full SHA
    0c0653f View commit details

Commits on Mar 28, 2025

  1. [ALS-6074] Fix uploader auth timeouts for instance profile

    - Refresh auth when rebuilding client
    Luke Sikina authored and Luke-Sikina committed Mar 28, 2025
    Copy the full SHA
    74778cf View commit details
  2. [ALS-7538] Make docker-compose standalone for uploader

    Luke Sikina authored and Luke-Sikina committed Mar 28, 2025
    Copy the full SHA
    4228515 View commit details
  3. [ALS-7539] - Get rid of super complex self refreshing client

    Luke Sikina authored and Luke-Sikina committed Mar 28, 2025
    Copy the full SHA
    a442356 View commit details
  4. [ALS-7539] Test HPDS connection

    Luke Sikina authored and Luke-Sikina committed Mar 28, 2025
    Copy the full SHA
    d21ea13 View commit details
  5. [ALS-7200] Upload large files via multipart

    Luke Sikina authored and Luke-Sikina committed Mar 28, 2025
    Copy the full SHA
    f5cb9a6 View commit details
  6. [CHIP-1] Add query.json and patients.txt

    - Add migrations via flyway
    - Change db schema for new fields
    - Add to API to support patient and query upload
    Luke Sikina authored and Luke-Sikina committed Mar 28, 2025
    Copy the full SHA
    9d534ef View commit details
  7. [ALS-8610] Add cumulus upload logic

    Luke Sikina authored and Luke-Sikina committed Mar 28, 2025
    Copy the full SHA
    d7cc028 View commit details
  8. [ALS-7716] - Use AVL role

    Luke Sikina authored and Luke-Sikina committed Mar 28, 2025
    Copy the full SHA
    7e4cb2e View commit details
Showing with 2,106 additions and 855 deletions.
  1. +32 −0 .github/sync_release.yml
  2. +32 −0 .github/workflows/sync_release.yml
  3. +24 −0 .github/workflows/test-runner.yml
  4. +5 −1 README.md
  5. +1 −1 uploader/Dockerfile
  6. +4 −8 uploader/README.md
  7. +0 −318 uploader/aws/just-users.yml
  8. +12 −34 uploader/aws/{s3-bucket.yml → user-based-aws-resources.yml}
  9. +300 −0 uploader/aws/userless-aws-resources.yml
  10. +25 −6 uploader/docker-compose.yml
  11. +4 −0 uploader/env-proto
  12. +8 −0 uploader/flyway/V1__initial.sql
  13. +5 −0 uploader/flyway/V2__add_patient.sql
  14. +12 −2 uploader/pom.xml
  15. +107 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/aws/AWSClientBuilder.java
  16. +19 −46 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/aws/AWSConfiguration.java
  17. +75 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/aws/AWSCredentialsService.java
  18. +21 −10 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/aws/S3StateVerifier.java
  19. +0 −131 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/aws/SelfRefreshingS3Client.java
  20. +20 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/aws/StsClientProvider.java
  21. +45 −20 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/hpds/HPDSClient.java
  22. +56 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/hpds/HPDSConnectionVerifier.java
  23. +63 −3 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/hpds/HttpClientConfig.java
  24. +13 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/hpds/UUIDGenerator.java
  25. +8 −151 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/hpds/hpdsartifactsdonotchange/Query.java
  26. +6 −1 .../src/main/java/edu/harvard/dbmi/avillach/dataupload/hpds/hpdsartifactsdonotchange/ResultType.java
  27. +11 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/site/SiteConfiguration.java
  28. +2 −1 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/status/DataUploadStatuses.java
  29. +5 −3 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/status/DataUploadStatusesMapper.java
  30. +23 −1 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/status/StatusRepository.java
  31. +8 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/status/StatusService.java
  32. +1 −1 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/status/UploadStatus.java
  33. +34 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/upload/DataType.java
  34. +14 −2 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/upload/DataUploadController.java
  35. +169 −35 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/upload/DataUploadService.java
  36. +15 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/upload/UploadConfig.java
  37. +103 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/upload/lambda/CumulusUploadService.java
  38. +97 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/upload/lambda/POSTUrlFetcher.java
  39. +4 −0 uploader/src/main/java/edu/harvard/dbmi/avillach/dataupload/util/Pair.java
  40. +6 −1 uploader/src/main/resources/application.properties
  41. +139 −0 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/aws/AWSClientBuilderTest.java
  42. +0 −38 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/aws/AWSConfigurationTest.java
  43. +47 −0 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/aws/AWSCredentialsServiceTest.java
  44. +51 −15 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/hpds/HPDSClientTest.java
  45. +97 −0 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/hpds/HPDSConnectionVerifierTest.java
  46. +29 −3 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/status/StatusRepositoryTest.java
  47. +24 −2 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/status/StatusServiceTest.java
  48. +20 −16 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/upload/DataUploadControllerTest.java
  49. +274 −0 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/upload/DataUploadServiceTest.java
  50. +18 −0 uploader/src/test/java/edu/harvard/dbmi/avillach/dataupload/upload/UploadConfigTest.java
  51. +14 −3 uploader/src/test/resources/application.properties
  52. +4 −2 uploader/src/test/resources/seed.sql
32 changes: 32 additions & 0 deletions .github/sync_release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Update Test Branch

on:
push:
branches: [main]

permissions: write-all

jobs:
sync:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
ssh-key: ${{secrets.SECRET_KEY}}
fetch-depth: 0

- name: Setup Git User
run: |
git config user.name "Luke Sikina"
git config user.email "lucas.sikina@gmail.com"
- name: Update Test Branch
run: |
git checkout main
git fetch origin
git checkout release
git pull
git rebase origin/main
git push origin release --force
32 changes: 32 additions & 0 deletions .github/workflows/sync_release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Update Test Branch

on:
push:
branches: [main]

permissions: write-all

jobs:
sync:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
ssh-key: ${{secrets.SECRET_KEY}}
fetch-depth: 0

- name: Setup Git User
run: |
git config user.name "Luke Sikina"
git config user.email "lucas.sikina@gmail.com"
- name: Update Test Branch
run: |
git checkout main
git fetch origin
git checkout release
git pull
git rebase origin/main
git push origin release --force
24 changes: 24 additions & 0 deletions .github/workflows/test-runner.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: Java CI with Maven

on:
push:
branches: [ "main", "release" ]
pull_request:
branches: [ "*" ]

jobs:
build:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- name: Set up JDK 21
uses: actions/setup-java@v3
with:
java-version: '21'
distribution: 'corretto'
cache: maven
- name: Build with Maven
run: "for dir in 'uploader' 'info-service'; do cd $dir; mvn verify; cd ..; done"

6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -54,5 +54,9 @@ pull in the artifact by including the following in your pom:
</dependency>
```

:)


When implementing the info endpoint, you can assume your service's resource ID will
be available inside your container via the environment variable `RESOURCE_UUID`
be available inside your container via the environment variable `RESOURCE_UUID`

2 changes: 1 addition & 1 deletion uploader/Dockerfile
Original file line number Diff line number Diff line change
@@ -16,4 +16,4 @@ COPY --from=build target/uploader-*.jar /dataupload.jar
# Time zone
ENV TZ="US/Eastern"

ENTRYPOINT ["java", "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005", "-Xmx8192m", "-jar", "/dataupload.jar"]
ENTRYPOINT java $DEBUG_VARS $PROXY_VARS -Xmx8192m -jar /dataupload.jar
12 changes: 4 additions & 8 deletions uploader/README.md
Original file line number Diff line number Diff line change
@@ -47,7 +47,7 @@ that Amazon provides you.

5. Run

Run `docker run --rm --env-file .env gic-data-uploader`. Here is an example of a
Run `docker compose --profile production up -d `. Here is an example of a
successful output, with logging prefixes omitted:

```
@@ -79,10 +79,6 @@ Verifying delete capabilities
S3 connection verified.
```

TODO:
- Mount this study in service workbench
- Admin UI
- Test all 4 workspace types
- Mount study
- R/W access
-
6. Run Migrations

`docker compose --profile migrate up -d`
Loading