Add Unassigned, Reserved, and ExperimentalUse options holding opaque binary values #328

zalenskivolt · 2025-03-19T23:45:28Z

Add option classes to hold any option number encountered, with an opaque value.
Decode any unknown options into an Unassigned, Reserved or ExperimentalUse option instead of returning an error.
Format the value in hex in OptionSerializer.

This is to support handling any new or experimental option, for examples new entries in this list
https://www.iana.org/assignments/core-parameters/core-parameters.xhtml#option-numbers
or any custom or experimental one.

The options are classified as:

Reserved for the reserved option numbers: 0, 128, 132, 136, and 140
ExperimentalUse in the range 65000-65535
Unassigned if otherwise unknown

See IANA CoAP Option Numbers

Closes #182

twyatt · 2025-03-21T00:52:07Z

Let me mull on this for a bit.

I'm not eager to implement a "catch-all" type — [assuming not a huge lift] being that CoAP Option Numbers (link you provided) has an exhaustive list of options, it seems it would be better to instead implement those types?

zalenski · 2025-03-21T10:04:33Z

That list also has wide ranges providing for new extensions and experimental use, so I think you would need some way to capture a general option number. Is essence, all numbers could potentially get used

0-255 IETF Review or IESG Approval

256-2047 Specification Required

2048-64999 Expert Review

65000-65535 Experimental use (no operational use)

Of course, it would be great to cover all options already defined with specific implementations. There might possibly be some upgrade issues if first the "Unknown" or general catch-all is added, and later on more specific implementations get added, for anyone using such options. On the other hand, there's no easy way to implement specific option handling before that option is added, and implementations could still pass on all options as is, and things like that. So maybe it is not too bad if a general catch-all is added first, and more specific implementations get added later?

For my personal use case as an example, we are using the No-Response option that is in the list, which is how I noticed the issue. Now we are also about to add a custom option for internal use, and I wish the online KoAP parser to still be useful for all of our CoAP messages!

Please mull over all this. Maybe some other naming is better, and maybe considerations for future upgrades are important. I'm still pushing for any encountered option to be handled in some way! 😄

twyatt · 2025-03-21T21:08:47Z

My thinking is that we simply implement the handful of additional types defined in CoAP Option Numbers (e.g. No-Response, Echo, etc) and then also add:

Unassigned
Reserved
ExperimentalUse

So, I guess Unassigned essentially is Unknown, but it will fall under well defined option numbers. 🙃

codecov · 2025-03-21T21:12:55Z

Codecov Report

Attention: Patch coverage is 56.41026% with 17 lines in your changes missing coverage. Please review.

Project coverage is 69.32%. Comparing base (e426dfd) to head (281259a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
koap-core/src/commonMain/kotlin/Message.kt	51.51%	9 Missing and 7 partials ⚠️
koap-core/src/commonMain/kotlin/Decoder.kt	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #328      +/-   ##
==========================================
- Coverage   69.74%   69.32%   -0.43%     
==========================================
  Files           9        9              
  Lines         790      828      +38     
  Branches      176      215      +39     
==========================================
+ Hits          551      574      +23     
+ Misses        171      168       -3     
- Partials       68       86      +18

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

twyatt · 2025-03-21T21:17:06Z

@zalenskivolt let me know if you want to tackle adding the few remaining options, or if you'd like me to take that on?

zalenski · 2025-03-21T21:36:33Z

Sounds like a plan!

This would make future allocated option numbers parsed as "unassigned", but I guess that makes perfect sense until the code gets updated.

Maybe both Reserved and ExperimentalUse could inherit from Unassigned? They are all unassigned too…

Interesting that the reserved ones are just 5 numbers. The option 0 (zero), and four reserved for "more Location-*" options.

From RFC7252(CoAP) 5.10.7. Location-Path and Location-Query

Beyond Location-Path and Location-Query, more Location-* options may be defined in the future
and have been reserved option numbers 128, 132, 136, and 140.

zalenski · 2025-03-21T21:37:24Z

@zalenskivolt let me know if you want to tackle adding the few remaining options, or if you'd like me to take that on?

I'll see if I can make a start!

twyatt · 2025-03-21T22:45:43Z

Maybe both Reserved and ExperimentalUse could inherit from Unassigned? They are all unassigned too…

If you can make the class hierarchy play nice, then that would be ideal. I'm also fine w/ them living alongside Unassigned if them inheriting causes issues.

Interesting that the reserved ones are just 5 numbers. The option 0 (zero), and four reserved for "more Location-*" options.

From RFC7252(CoAP) 5.10.7. Location-Path and Location-Query

Ya, the scattering of reserved options number is quite strange. I wonder if it has something to do w/ the bitfield? 🤷

Integer value	Hex	Bits
128	`80`	`1000_0000`
132	`84`	`1000_0100`
136	`88`	`1000_1000`
140	`8C`	`1000_1100`

koap-core/src/commonMain/kotlin/Message.kt

zalenski · 2025-03-21T22:58:58Z

If you can make the class hierarchy play nice, then that would be ideal. I'm also fine w/ them living alongside Unassigned if them inheriting causes issues.

from a quick google:

stackoverflow/extend-data-class-in-kotlin

The truth is: data classes do not play too well with inheritance. We are considering prohibiting or severely restricting inheritance of data classes. For example, it's known that there's no way to implement equals() correctly in a hierarchy on non-abstract classes.

So, all I can offer: don't use inheritance with data classes.

twyatt · 2025-03-21T23:11:16Z

Looks like a great start! I'll try to find time this weekend to look it over.

zalenskivolt · 2025-03-22T00:54:08Z

@zalenskivolt let me know if you want to tackle adding the few remaining options, or if you'd like me to take that on?

I'll see if I can make a start!

Started with 258 No-Response in #332, that I had already started on.

Feel free to comment on that one, and I'll try to apply changes to any others…
Maybe they could be added in some blocks, like OSCORE+EDHOC, the different block-wise ones, hop-limit+echo+request-tag?

Missing ones are

#	Option	RFC
9	OSCORE	[RFC8613 OSCORE (using COSE)]
16	Hop-Limit	[RFC8768 Hop-Limit]
19	Q-Block1	[RFC9177 Block-Wise robust]
21	EDHOC	[RFC9668 Ephemeral Diffie-Hellman Over COSE]
23	Block2	[RFC7959 Block-Wise][RFC8323 TLS+][RFC8613 OSCORE]
27	Block1	[RFC7959 Block-Wise][RFC8323 TLS+][RFC8613 OSCORE]
28	Size2	[RFC7959 Block-Wise][RFC8613 OSCORE]
31	Q-Block2	[RFC9177 Block-Wise robust]
252	Echo	[RFC9175 Echo, Request-Tag, and Token Processing]
258	No-Response	[RFC7967 No Server Response][RFC8613 OSCORE]
292	Request-Tag	[RFC9175 Echo, Request-Tag, and Token Processing]

TLS+ = TCP, TLS, and WebSockets

and these are also listed without RFCs:

#	Option	Reference
2049	OCF-Accept-Content-Format-Version	[Michael_Koster]
2053	OCF-Content-Format-Version	[Michael_Koster]
2055	SCP82-Params	[GPC_SPE_207]

zalenskivolt · 2025-03-22T21:20:06Z

@zalenskivolt let me know if you want to tackle adding the few remaining options, or if you'd like me to take that on?

I'll see if I can make a start!

Started with 258 No-Response in #332, that I had already started on.

Feel free to comment on that one, and I'll try to apply changes to any others… Maybe they could be added in some blocks, like OSCORE+EDHOC, the different block-wise ones, hop-limit+echo+request-tag?

PRs for all 11 now:

PR	Options
#332	No-Response
#333	OSCORE, EDHOC
#334	Hop-Limit, Echo, Request-Tag
#335	Size2, Block1, Block2, QBlock1, QBlock2

All of them update the comments in Message.kt, I planned to rebase that away from all but one of them (or add annother PR?).

twyatt · 2025-03-22T21:37:58Z

PRs for all 11 now:

PR Options
#332 No-Response
#333 OSCORE, EDHOC
#334 Hop-Limit, Echo, Request-Tag
#335 Size2, Block1, Block2, QBlock1, QBlock2
All of them update the comments in Message.kt, I planned to rebase that away from all but one of them (or add annother PR?).

Thanks so much for all your work on all this! Really incredible effort! ❤️

I'll follow up on this PR after #329 and #332 are merged.

Apologies for all the rebasing that will likely ensue. 🙃

zalenski · 2025-03-24T22:24:51Z

Apologies for all the rebasing that will likely ensue. 🙃

All your rebase are belong to us! 😄
https://www.youtube.com/watch?v=qItugh-fFgg

twyatt · 2025-03-25T07:30:17Z

Apologies for all the rebasing that will likely ensue. 🙃

All your rebase are belong to us! 😄 https://www.youtube.com/watch?v=qItugh-fFgg

Haha! Classic! 😆

zalenski · 2025-03-31T20:05:46Z

rebased

This reverts commit 4f2e215.

This reverts commit 376050e.

zalenskivolt · 2025-03-31T22:12:23Z

webapp/src/jsTest/kotlin/DecodingTest.kt

-    @Test
-    fun coapWithUnassignedOption() = GlobalScope.promise {


(removed the DecodingTest tests)

zalenskivolt · 2025-03-31T22:13:20Z

webapp/src/jsMain/kotlin/serialization/MessageSerializer.kt

-            is Option.Unassigned -> "Unassigned(number=${value.number}, value=${value.value.hex().trim()})"
-            is Option.Reserved -> "Reserved(number=${value.number}, value=${value.value.hex().trim()})"
-            is Option.ExperimentalUse -> "ExperimentalUse(number=${value.number}, value=${value.value.hex().trim()})"


(moved to toString methods)

twyatt · 2025-04-03T08:29:08Z

Made some minor changes:

Marked Unassigned and Reserved constructors as internal
- This simplifies the error checking (i.e. we can drop the init blocks), and I don't see any reason a library consumer would need to create an Unassigned or Reserved — I'd rather an issue be raised if a new option becomes available rather than people using Unassigned or Reserved for encoding; or Option.Format.opaque could be used if such encoding is needed
Moved the reserved option numbers to a Set so it could be re-used

Let me know if it looks good to you.

zalenskivolt · 2025-04-03T21:03:10Z

Let me know if it looks good to you.

LGTM! 😄

zalenskivolt requested review from twyatt and a team as code owners March 19, 2025 23:45

zalenskivolt requested a review from cedrickcooke March 19, 2025 23:45

zalenskivolt mentioned this pull request Mar 19, 2025

Parse unknown options as opaque instead of failing the whole message #182

Closed

twyatt added the minor Changes that should bump the MINOR version number label Mar 20, 2025

twyatt added a commit that referenced this pull request Mar 21, 2025

Possible changes to #328 [skip ci]

b0244bf

twyatt reviewed Mar 21, 2025

View reviewed changes

koap-core/src/commonMain/kotlin/Message.kt Outdated Show resolved Hide resolved

koap-core/src/commonMain/kotlin/Message.kt Outdated Show resolved Hide resolved

koap-core/src/commonMain/kotlin/Message.kt Outdated Show resolved Hide resolved

zalenskivolt changed the title ~~Add UnknownOption holding an opaque binary value~~ Add Unasigned, Reserved, and ExperimentalUse options holding opaque binary values Mar 21, 2025

zalenskivolt changed the title ~~Add Unasigned, Reserved, and ExperimentalUse options holding opaque binary values~~ Add Unassigned, Reserved, and ExperimentalUse options holding opaque binary values Mar 21, 2025

twyatt self-requested a review March 21, 2025 23:25

zalenskivolt force-pushed the unknown-option branch 3 times, most recently from 3d5270f to f5a15b7 Compare March 31, 2025 20:02

zalenskivolt force-pushed the unknown-option branch from f5a15b7 to 1241d54 Compare March 31, 2025 20:09

zalenskivolt force-pushed the unknown-option branch 2 times, most recently from 8117ff8 to cb639ab Compare March 31, 2025 21:24

zalenskivolt added 7 commits March 31, 2025 23:38

Add UnknownOption holding an opaque binary value

5e45e13

Update to Unassigned, Reserved, ExperimentalUse

d686d74

Add toString methods

ab4aa1e

test toString

70f4a60

MessageSerializer

4f2e215

Revert "MessageSerializer"

fa77caa

This reverts commit 4f2e215.

Add full message decoding tests for unknown options

376050e

zalenskivolt force-pushed the unknown-option branch from ec3fb2c to 376050e Compare March 31, 2025 21:41

Revert "Add full message decoding tests for unknown options"

49ad019

This reverts commit 376050e.

zalenskivolt commented Mar 31, 2025

View reviewed changes

twyatt added 3 commits April 3, 2025 01:23

Make Unassigned constructor internal

4ac6254

Have reserved option numbers defined in a Set

24f7e16

Use EXPERIMENTAL_USE_OPTION_RANGE in when

701244d

twyatt added 5 commits April 3, 2025 01:35

Make Reserved constructor internal

09ee2eb

Move Unassigned in when

5fa2b04

Fix ExperimentalUse test

67bf585

Move Unassigned in when for consistency

32dcf61

./gradlew apiDump

90b2146

twyatt approved these changes Apr 3, 2025

View reviewed changes

twyatt requested a review from zalenski April 3, 2025 08:46

twyatt added 2 commits April 3, 2025 01:47

Merge branch 'main' into unknown-option

46a2928

./gradlew formatKotlin

281259a

cedrickcooke approved these changes Apr 3, 2025

View reviewed changes

twyatt merged commit 2205fba into JuulLabs:main Apr 3, 2025
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Unassigned, Reserved, and ExperimentalUse options holding opaque binary values #328

Add Unassigned, Reserved, and ExperimentalUse options holding opaque binary values #328

zalenskivolt commented Mar 19, 2025 •

edited

Loading

twyatt commented Mar 21, 2025 •

edited

Loading

zalenski commented Mar 21, 2025 •

edited

Loading

twyatt commented Mar 21, 2025 •

edited

Loading

codecov bot commented Mar 21, 2025 •

edited

Loading

twyatt commented Mar 21, 2025

zalenski commented Mar 21, 2025

zalenski commented Mar 21, 2025

twyatt commented Mar 21, 2025 •

edited

Loading

zalenski commented Mar 21, 2025 •

edited

Loading

twyatt commented Mar 21, 2025

zalenskivolt commented Mar 22, 2025 •

edited

Loading

zalenskivolt commented Mar 22, 2025

twyatt commented Mar 22, 2025 •

edited

Loading

zalenski commented Mar 24, 2025

twyatt commented Mar 25, 2025

zalenski commented Mar 31, 2025

zalenskivolt Mar 31, 2025

zalenskivolt Mar 31, 2025

twyatt commented Apr 3, 2025 •

edited

Loading

zalenskivolt commented Apr 3, 2025

Add Unassigned, Reserved, and ExperimentalUse options holding opaque binary values #328

Add Unassigned, Reserved, and ExperimentalUse options holding opaque binary values #328

Conversation

zalenskivolt commented Mar 19, 2025 • edited Loading

twyatt commented Mar 21, 2025 • edited Loading

zalenski commented Mar 21, 2025 • edited Loading

twyatt commented Mar 21, 2025 • edited Loading

codecov bot commented Mar 21, 2025 • edited Loading

Codecov Report

twyatt commented Mar 21, 2025

zalenski commented Mar 21, 2025

zalenski commented Mar 21, 2025

twyatt commented Mar 21, 2025 • edited Loading

zalenski commented Mar 21, 2025 • edited Loading

twyatt commented Mar 21, 2025

zalenskivolt commented Mar 22, 2025 • edited Loading

zalenskivolt commented Mar 22, 2025

twyatt commented Mar 22, 2025 • edited Loading

zalenski commented Mar 24, 2025

twyatt commented Mar 25, 2025

zalenski commented Mar 31, 2025

zalenskivolt Mar 31, 2025

Choose a reason for hiding this comment

zalenskivolt Mar 31, 2025

Choose a reason for hiding this comment

twyatt commented Apr 3, 2025 • edited Loading

zalenskivolt commented Apr 3, 2025

zalenskivolt commented Mar 19, 2025 •

edited

Loading

twyatt commented Mar 21, 2025 •

edited

Loading

zalenski commented Mar 21, 2025 •

edited

Loading

twyatt commented Mar 21, 2025 •

edited

Loading

codecov bot commented Mar 21, 2025 •

edited

Loading

twyatt commented Mar 21, 2025 •

edited

Loading

zalenski commented Mar 21, 2025 •

edited

Loading

zalenskivolt commented Mar 22, 2025 •

edited

Loading

twyatt commented Mar 22, 2025 •

edited

Loading

twyatt commented Apr 3, 2025 •

edited

Loading