Idle timeout and terminate for stdin source

[sgowie]

I found myself relying on the timeout command to prompt Vector into shutting down when used to read from stdin in parallel (about 7000 invocations).

e.g.

find . -type f -name "*.log.gz" | xargs -P 12 -I % sh -c 'zcat % | timeout 15 vector --config /etc/vector/ingest.toml'

The ingest.toml would read from stdin and perform a small set of transforms before sinking to another vector instance where some metrics were extracted. I opted for this model since vector doesn't perform decompression for file type sources. Unfortunately Vector will remain open preventing the xargs loop from proceeding. Using timeout here isn't something I'm necessarily comfortable with, but it did effectively allow me to process nearly 2 years of event logs.

If the stdin source could be configured to trigger the shutdown of vector after some defined idle period (ensuring any batched sinks are flushed), processing historical logs can be done without foreknowledge relating to the processing time per file that vector would presumably take.

[binarylogic]

Hi @sgowie, thanks for reporting. Which version of Vector are you using? I ask because we fixed a related issue where the stdin source would fail to shutdown in #960. Let me know if that resolves your issue. We'll be getting a new release out this week.

[sgowie]

We were using version 0.5.0:
timberio/vector 0.5.0-alpine b4eecb762b28 5 weeks ago 24.5 MB

From what I can tell this is the same version that's available in the nightly build, but was built prior to that pull being merged.

[binarylogic]

Thanks @sgowie. I've confirmed this has been resolved. Please let us know if you continue to experience this.