encoding.timestamp_format=”unix“ store up nanosecond !

[FengZh61]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

.timestamp, err = to_float(.timestamp) can store up to nanosecond resolution!

encoding.timestamp_format="unix" store up to second resolution!

I need to use remap for that!

Sink can’t store up nanosecond

Attempted Solutions

.timestamp, err = to_float(.timestamp) can store up to nanosecond resolution!

Proposal

encoding.timestamp_format="unix" store up to nanosecond resolution!

References

#16206

Version

vector 0.27.0 (x86_64-unknown-linux-musl 5623d1e 2023-01-18)

[jszwedko]

Thanks @FengZh61 ! Can you clarify the output format you want? Do you want an unsigned integer that represents the number of nanoseconds since the Unix epoch? For example: 1676462587000000. Or do you want it to be a float with partial seconds appearing after the . like 1676462587.000000?

Either way I think this will need to be a new encoding timestamp format to preserve compatibility.

[zamazan4ik]

@jszwedko Just an idea. Instead of inventing a new encoding timestamp format with backward compatibility, we can think about a tool that performs config migration from an old to a new format (and back if we are going to support rollbacks). Maybe, it would be easier to use in the long term.

[jszwedko]

@jszwedko Just an idea. Instead of inventing a new encoding timestamp format with backward compatibility, we can think about a tool that performs config migration from an old to a new format (and back if we are going to support rollbacks). Maybe, it would be easier to use in the long term.

Yeah, that is a good idea. I know there are other tools that do that (like terraform).

In this case I think we'd actually want to support both in perpetuity rather than deprecating and removing certain behavior.

[FengZh61]

Thanks @FengZh61 ! Can you clarify the output format you want? Do you want an unsigned integer that represents the number of nanoseconds since the Unix epoch? For example: 1676462587000000. Or do you want it to be a float with partial seconds appearing after the . like 1676462587.000000?

Either way I think this will need to be a new encoding timestamp format to preserve compatibility.

I want it to be a float with partial seconds appearing after the . like 1676462587.000000!

[FengZh61]

@jszwedko Just an idea. Instead of inventing a new encoding timestamp format with backward compatibility, we can think about a tool that performs config migration from an old to a new format (and back if we are going to support rollbacks). Maybe, it would be easier to use in the long term.

Yeah, that is a good idea. I know there are other tools that do that (like terraform).

In this case I think we'd actually want to support both in perpetuity rather than deprecating and removing certain behavior.

Agree with your idea!

[jszwedko]

Closed by #18817