Add support for interception IO real time

[bilby91]

Hello,

First of all, thanks for this awesome gem! I'm using the gem to build a declarative way of building and running docker & docker-compose based projects.

One feature that I really need (I already have it implemented when doing docker only) is that ability to get the logs of a container real time. So, for example, when I do something like:

@session = Docker::Compose::Session.new(Backticks::Runner.new(buffered: [:stderr], interactive: true),{
      dir: ".",
      file: "docker/docker-compose.yml"
    })

@session.run("my_service", "/bin/long-task")

I would like to get a stream-like with the logs real time. The docker-api gem implementation has some functionality to do this.

After looking at the code, It seems that the shell abstraction used by default is the backticks gem. The gem already provides something for real time IO inspection (Intercepting and Modifying I/O) so I think it could be possible to make it work.

What if Session#run! accepts a block that will yield the log bytes ?

Something like:

@session.run("my_service", "/bin/long-task") do |bytes|
  p bytes

Looking forward on your thoughts @xeger and thanks before hand!

[xeger]

Hmm. As you say, the essential thing you need is built into Backticks already via the Command#tap method. However, we need an elegant way to expose that to you from the Docker::Compose level.

Docker::Compose::Session has a #last_command reader, which is almost perfect since you could tap it. The problem is, all of the Session method are designed to be synchronous, i.e. they call #join on every command before returning, which gives you no chance to tap.

One option is to monkey patch or wrap the Backticks Runner and Command interfaces so that join returns immediately status always returns success. It's a bit of a hack, but it works with no changes to either gem.

The cleaner option is probably to introduce a new Session#asynchronous attribute; if set to true, then run! would not call join and would return nil immediately without checking command status. For your use case you could use an asynchronous session and tap into last_command (then call join on it) to do the monitoring that you want. You would , of course, also need to check the command's status code.

Would that work for you?

[bilby91]

@xeger What if the decision on calling tap would be based if a block is given to run! ? That way we can make the change backward compatible, introducing a real-time log api only by providing a block.

[xeger]

While a "tapping block" would present a superior interface to the user of Session, there are some drawbacks I can think of:

1. Every command of Session would need to gain this tapping block, so there'd be a bunch of code changes.
2. The method signatures for the Session commands are already complex, using *glob and occasionally **glob; adding an optional block would make them even more complex and magical.
3. Backticks::Runner controls some options of how the command output is tapped (e.g. interactive=, buffered=) and the user of Session would still need to configure the underlying Runner directly, even though his tapping blocks would be passed to Session. So, the tapping block is a leaky abstraction (aka poor encapsulation).
4. The interface of the Command#tap block is actually fairly complex; it gets called many times, with a symbolic stream name as well as data; the tapping block of Session#run would inherit that complexity.

Can I ask more specifically about your use case? If you just want to see a realtime log of the container's output and don't mind blocking until run completes, then I don't think you need to mess with tap at all; just construct your own Runner with interactive:true. (And I guess you could use Thread.new { session.run('myservice') } to avoid blocking; Backticks' I/O should be safe to use with Ruby's limited multithreading.)

On the other hand, if you need to intercept the output and do something with it in Ruby, then we do indeed need to make some changes. I still prefer a Session#asynchronous option, plus making the caller do some extra work to tap into the last_command and decide on how/whether to join that command. I know it's a bit more work for your use case, but it's a bit more flexible. For instance, people might benefit from asynchronous even if they are not interested in tapping.

One downside of relying on last_command is that it's possible to accidentally tap or join the wrong command if the user of Session isn't careful. I guess I could modify my proposal, such that when asynchronous is true, then all commands return the Command object itself instead of the String output.

If I did that, then you could write a fairly transparent wrapper around Session that is specialized to do what you need. For instance:

class WelshSession # prints logs in realtime but removes all vowels
  def initialize(session)
    @session = session
    @session.runner.buffered = false
    @session.asynchronous = true
  end
  def method_missing(*args)
    cmd = @session.__send__(*args)
    cmd.tap { |stream, bytes| stream == :stdout && STDOUT.write(bytes.gsub(/[aeiouy]/, '')) }
    cmd.join
    raise 'damn' unless cmd.success?
  end
end

[xeger]

Closing due to inactivity.