Make the docker-compose.yml file manipulatable

[EugenMayer]

This projects seems very useful! Thank you for all the work.

I would like to replace it a pretty complicated script which uses uses several docker compose files and overlays to do something like "load dev enironment with variations A" which means:

docker compose-f production/docker-compose.yml -f development/docker-compose.yml -f variation/a/docker-compose.yml up

So i am using several docker-compose files as overlay, while production/docker-compose.yml is huge and defines all services, while development/docker-compose.yml basically just changes some ENV vars to initiate some the development enironment and rarely using a different image for development.

Any ideas if you would like to either add this "overlay suppport" to session or even, parse the config once, and then let the hash be manipulateble, when running, you write this file temporary into the dir defined, like . and run compuse -f . op

While the reason we should not write to /tmp/ is because compose actuall derives the network-domain from the top-level folder. We could of course simulate that folder in /tmp

What are your ideas?

[xeger]

Hmm... so you want to combine these files in memory, tweak some values, and run the CLI with the result? Seems reasonable. We can write the temp file to the @dir as long as it's named something unique; its content hash is a good choice.

Design wise, I'd probably create a Project object, loadable from YML, with a #to_yaml method and whatever mutation stuff is required. If you pass a Project to the session instead of a file, then boom, you write a temp and run the command.

It might even be possible to extract all filename handling into this class (and retain your array approach, if someone wanted to specify multiple editable compositions ... hmm ... )

I'd probably add a Session#file= writer too, so people can replace the contents entirely.

[EugenMayer]

I think the biggest hurdle here can become the output-parser. You know handled parsing the compose file, now we have to properly write it:

• the same order ( sometimes it matters )
• What about integers defined as strings in yaml, but then written back? ( It will most probably end up being an integer/number ) - does yaml really care about this too much? My experience is, it does not really matter

Use cases for compose file manipulation

a) Open specific ports for Development ( Debugging / Database ports )
b) Add host mount points for code sharing
c) Change ENV variables (like rails env or similar)
d) inject ENV vars based on the current system ENV like the "ip of the host". The user name of the invoker, the date, OS type for development tools

Alternatives?

Using compose file overrides like shown in the pull request, you can handle a-c pretty easy and you can use those docker files without docker-compose.rb, which make them more independent.

A fairly interesting point though is d) which is not solvable right now with the compose-overrides since the values are dynamic. d) is often important for setting up "debugging hosts" on the container so it connects back properly, exposing the host interface for a particular image or apply developer-defauls which are in ~/.myapp/devdefault.yaml - whatever kind of things. One additional integration point could be rsync/unison/nfs share-volume setups for development

E.g. Hodor https://github.com/gansbrest/hodor does offer a .hoderfile which is a compose file, but parsed by ERB. While i like this idea, the file there is not being able to be used in production, its a copy of the production compose file, a separate one. Though one could write a "generate production compose file on build|" CI step to solve this without a copy.

I hope you do not mind that i make this topic a bit bigger. I just want to share some scenarios this kind of things gets used in and actually also the reason i looked for a gem like you have here.

@xeger thanks for making me a contributor. I will continue creating pull-requests nevertheless, so we can have code reviews and things - i guess thats common sense anyway

[xeger]

Use Cases & Alternatives

Hmm. So, there is an obscure feature of docker-compose.yml; it allows host env to be substituted like so:

banana:
  image: busybox
  command: "bash -c echo hello ${TONY}"

It may even support the ${FOO:-default} syntax so you can handle empty host vars.

The docker-compose binary recognizes ${} and subs in the host environment. It has some limitations, e.g. it cannot be used in any context, but it may be enough for (d), and is probably enough for (c).

For (a) and (b), you might be able to handle them with selective overriding -- add additional file if user wants ports, volumes, etc -- but it would get sloppy. Editing the composition in ruby space is more graceful.

[xeger]

YAML Parsing

YAML.load(String) will produce a Hash of nicely-typed Ruby objects, i.e. something that is Integer in the YAML will remain Integer in Ruby-land. YAML.dump(Hash) will likewise preserve types. This is true for Numeric, Boolean, String, Array and nested-Hash -- so we probably don't need to worry about changing the types of objects accidentally. If someone manipulates the Hash and inserts a string in place of an Integer -- well, that might break docker-compose, but it's the programmer's error, not the gem's, and it's easy to fix.

Preserving ordering is a bit more tricky. Ruby's Hash class guarantees preservation of order, so if the YAML parser builds Hashes in a straightforward way, and if it enumerates in order when dumping back to YAML, then order is preserved. Some experiments are in order to find out. There's also the issue that someone could reformat the hash in Ruby space and destroy the order -- once again, though, that's the programmer's bug.

A final consideration is that many things in docker-compose YML that need to be ordered, already are capable of using an Array notation, e.g.

ports:
  - 8080
  - 80
  - 443

[xeger]

Closing (for now) due to lack of interest from contributors.