Bignum support

[akheron]

Add bigint/bigfloat types and hooks to allow using external libraries to manage them.

There's an initial patch by Deron Meranda here: http://deron.meranda.us/computing/c/jansson-bignum/

[garymazz]

Bignum support is very much needed !! The CDMI and OCCI cloud standards are using JSON as a data format. Storage devices capacities defined in JSON easily exceed 10 decimal digits. 64bit and possible 128 bit integers are a requirement.

[watsocd]

As a workaround, I converted my big numbers to JSON strings and then converted them back on the server.

[maxtors]

Im using the same solution as watsocd, convering to strings then converting back when the analysis is done. It would be great if some uint64_t, etc, etc, was added.

[sakubara]

I would like to bring this topic up again since I see need for it and also have a solution in mind. I also would like to provide the patch for it, if desired. This would also help to solve #288.

I would like to go a different route though compared to the way it was suggested in the patch linked above or the PR #158. I would recommend to store integers and floating point values internally as string values just as how they were extracted from the Json input. This would retain the original value and help in round trip decoding and encoding. The machine representation of the value can be stored alongside the string representation, if performance is a concern.

Users of jansson can decide to use the string representation of these values and feed them to libraries like GMP in order to process them.

The API provided by jansson can be kept in its current state, the conversion from string to integer and back again would happen on demand and hidden from the user of the API. A new function int json_integer_value(const json_t *integer, json_int_t *out) could be introduced that returns -1 whenever the stored integer overflows a machine word/json_int_t. The function strtoll provides this information, so this can be reused and implemented easily.

The biggest upside would be that this eliminates the need for the extra types JSON_DECIMAL or JSON_BIGINTEGER and JSON_BIGREAL. This would relieve users of jansson from the burden to test for and handle different types of integer values which essentially mean the same thing.

Please let me know what you think @akheron. I would like to start working on this soon if the changes are desired and there is a chance this can be merged.

[akheron]

Sounds good. A few notes:

You would also need to add a function to extract the original string representation for integers and reals.

Storing the string representation should be optional, so that it's only stored if a certain decoding flag is given. This would lower the memory impact for those who don't use the original string.

json_real_t and json_integer_t structs could be expanded by adding a char str[] field to the end, and only store a single \0 if the user didn't opt in for storing the original string. This way, the memory impact for those who don't need the string representation would be minimal.

Looking at the code, json_string_t could also use this method to avoid an extra malloc() call for the string data.

[sakubara]

You would also need to add a function to extract the original string representation for integers and reals.

Of course, the changes would be pointless without access to these values. I'm aware that the internals of the json_t type are handled as an opaque data structure and access to the information held by the objects is granted by the respective functions which form the interface to this library.

Storing the string representation should be optional, so that it's only stored if a certain decoding flag is given. This would lower the memory impact for those who don't use the original string.

This is a very good idea, I'll implement it that way. The functions to retrieve the string representation of the stored numbers would do a string conversion on demand in this case then. Is this a good idea?

json_real_t and json_integer_t structs could be expanded by adding a char str[] field to the end, […]

I'm afraid this is not possible since json_t objects are mutable by design. Changing the value of a json_t object with the json_*_set family of functions may need to reallocate the data structures, if the new value exceeds the current object in size, which in turn would require the API to return a pointer to the (new) json_t object.

[akheron]

I'm afraid this is not possible since json_t objects are mutable by design. Changing the value of a json_t object with the json_*_set family of functions may need to reallocate the data structures, if the new value exceeds the current object in size, which in turn would require the API to return a pointer to the (new) json_t object.

Ah, true.

[cathay4t]

How about we use the yajl way:

yajl_gen_number (yajl_gen hand, const char *num, unsigned int len)

which allow user generate json_t using string of number:

json_t *json_number(const char *num_str).

[Ghazibelaam]

Anything has been done yet to support Unsigned 64 bit numbers?

[lano1106]

I got this error while parsing a JSON string: too big integer near '15726282638479065023'

This number is bigger than LLONG_MAX but is smaller than ULLONG_MAX so it would fit into an unsigned long long...

I could definitely benefit from ull support...

[redsand]

Bump.. this is still needed. Suricata flow ids and session ids in eve.json format exceed formatting specs. If JSON_DECODE_INT_AS_REAL is used, its properly parsed, but all integers are now floats. Be nice to convert this to a float only IF the value size exceeds size of long.

{"timestamp":"2022-06-10T13:00:18.047564+0000","flow_id":24486688685068,"in_iface":"tun1","event_type":"alert","src_ip":"10.100.X.X","src_port":53211,"dest_ip":"10.100.X.X","dest_port":445,"proto":"TCP","tx_id":32,"alert":{"action":"allowed","gid":1,"signature_id":2225006,"rev":1,"signature":"SURICATA SMB file overlap","category":"Generic Protocol Command Decode","severity":3},"smb":{"id":140,"dialect":"3.11","command":"SMB2_COMMAND_READ","session_id":11605213289780938933}}