Service config and files

[Thechi2000]

As we seem to go toward a monorepo project, we need to think about how we will:

• Manage the config files (e.g. /etc/nginx) of all services, especially the ones we won't build (just pull the image)
• Manage the config (environment variables, mounts) of all services
• Manage the images of built services
• Manage the secrets
• Integrate all of the above into the CI/CD

Since the goal of this project is to be accessible by all, I think we'll need to put the config files of the services on the repository, but then we need the CD to pull those files and put them in the correct place.

For the services' configs, we can deploy them as stacks, so we'll be able to store the whole config into a docker-compose.yml file, and deploy them very easily. But then: would we put all services in the same stack, or each of them would have a different one ?

The images could be built and pushed to this repository's ghrc.io registry.

The secrets will be managed through docker secret, but this cannot be done on the repository for two reasons:

• Obviously, the secrets must stay secret
• They are only managed through CLI

Building and pushing the images won't be a problem in the CD, but updating the config files and the stacks might be a bit trickier. I think the best solution would be to add a self-hosted runner on the server (it could also relieve us from the limitations of the github-hosted VMs for the actions)

[codeofmochi]

* Manage the config files (e.g. `/etc/nginx`) of all services, especially the ones we won't build (just pull the image)

I think it's fine if we keep one directory per service. I've seen other repos use linux-like tree structures in subfolders such as src/etc/foo/bar

* Manage the config (environment variables, mounts) of all services

For deployment-specific concerns (i.e. dependent on the machine) we may not want to add them to the repo, especially things like secrets which should stay on the end system. At the very least the development docker stack will serve as working example (though we'll need to make it clear which options should not be used in production).

* Manage the images of built services

I suggest we use Github's image registry. We could host them on the server as well but I personally like to keep build-related concerns separate from the production environment

* Manage the secrets

We should consider whether the overhead and complexity of using a secrets manager (such as Hashicorp Vault) is really worth it w.r.t to gained security. It may be useful for things like automatic keys rotation, however it's a rather heavy system to setup and maintain and since we'll have a single node deployment scheme, we probably won't get security that is that improved since we will rely on the filesystem or memory anyway to store the secrets or the decryption key.

For the services' configs, we can deploy them as stacks, so we'll be able to store the whole config into a docker-compose.yml file, and deploy them very easily. But then: would we put all services in the same stack, or each of them would have a different one ?

The entire "web application stack" can be deployed as a single docker-compose file IMO, because it doesn't really makes sense to e.g. run strapi without running the next front-end. However we should be able to connect to other independent services such as Nextcloud, and stop / restart the website without affecting them (and vice-versa).

The images could be built and pushed to this repository's ghrc.io registry.

👍

Building and pushing the images won't be a problem in the CD, but updating the config files and the stacks might be a bit trickier. I think the best solution would be to add a self-hosted runner on the server (it could also relieve us from the limitations of the github-hosted VMs for the actions)

I see what you mean, though I'm not a fan of using the production server to perform CD tasks. If possible, the CI should provide the stack "ready to deploy" to the server, and only things like environment variables should be plugged in from the server side.

[Thechi2000]

I think it's fine if we keep one directory per service. I've seen other repos use linux-like tree structures in subfolders such as src/etc/foo/bar

So we would have, for example, a nginx folder at the root of the repository, and inside a /etc/nginx/ directory containing the configs ?

We should consider whether the overhead and complexity of using a secrets manager (such as Hashicorp Vault)

Docker already has a built-in secret manager (see https://docs.docker.com/engine/swarm/secrets/)

The entire "web application stack" can be deployed as a single docker-compose file IMO, because it doesn't really makes sense to e.g. run strapi without running the next front-end. However we should be able to connect to other independent services such as Nextcloud, and stop / restart the website without affecting them (and vice-versa).

What I meant is that we can have a docker-compose.yaml file for each service (or stack), and use the docker stack command to deploy them. For how I understood you post, this file would not appear on the repo, only on the server

I see what you mean, though I'm not a fan of using the production server to perform CD tasks. If possible, the CI should provide the stack "ready to deploy" to the server, and only things like environment variables should be plugged in from the server side.

I see where I got confused; if we use docker images, we don't need to copy the config files anywhere on the server (e.g. nginx's files don't need to be placed at /etc/nginx), but they could be injected in docker images by the CI. Then we could setup a webhook to automatically redeploy the latest images. Is that correct ?

To sum up my understanding, we would have a repo structure as such:

/app/Dockerfile
/app/src/index.tsx

/strapi/Dockerfile

/nginx/Dockerfile
/nginx/etc/nginx/nginx.conf

The CI would then build one image per service, and the server would pull new images and deploy them accordingly to the stacks' files (stored on the server). This update could be triggered by a secured webhook