feat!: encrypted logs refactor #11400

iAmMichaelConnor · 2025-01-21T21:02:49Z

Description:

Seeking to enable users to specify custom "log assembly strategies".
- Moving some files around, to make it clear that our current AES approach is one "default strategy".
- Separating the code paths of log assembly for: Events, Notes, Partial Notes, because we want to:
  - Completely overhaul partial notes
  - Implement the randomness optimisation for notes.
- I didn't go as far as abstracting the partial note logs - they are still hard-coded as using AES - because they're about to be refactored.
- Note: there are libs like ValueNote which also hard-code the use of the "default strategy".
- Decryption is still happening in typescript, because I know Nico's doing stuff on this front.

Optimisations:

Some byte conversion function constraint optimisations.
The AES log construction was very inefficient, constraint-wise, because it was AES-encrypting all 17 usable fields of a log, even if the note only needed to use 3 or 4 of those fields. I changed the log layout so that we only AES-encrypt those 3 or 4 fields, and the rest is padding. How? I convey the length of the actual ciphertext through the header ciphertext. But isn't the header about to be deleted when we don't need to encrypt the contract address for the PXE to discover which contract a log is for? Yes, but I'm saying it's much more efficient to keep a thing called a "header" around, even if we only convey the ciphertext length in it, because then we avoid AES-encrypting the entire log. Huge saving. This was my main battleground in this PR.

New features:

I threw my Poseidon2 encryption functions in here too, but they're not-yet used by anything.
I realised there's a privacy leakage in the way we convert bytes31's to fields, because the "big-end" 6 bits are always 0. I wrote a function to pad those bits with randomness, but didn't go as far as plugging it into the log strategy, because it felt like too much at once.

AztecBot · 2025-01-22T13:54:22Z

Docs Preview

Hey there! 👋 You can check your preview at https://67954a73b0b4eec3480d8bf2--aztec-docs-dev.netlify.app

…kages into mc/potato-encode

noir-projects/aztec-nr/aztec/src/encrypted_logs/encrypt/poseidon2.nr

noir-projects/aztec-nr/aztec/src/encrypted_logs/encrypt/aes128.nr

docs/docs/migration_notes.md

...cts/aztec-nr/aztec/src/encrypted_logs/log_assembly_strategies/default_aes128/partial_note.nr

noir-projects/aztec-nr/aztec/src/encrypted_logs/log_assembly_strategies/default_aes128/note.nr

noir-projects/aztec-nr/aztec/src/encrypted_logs/payload.nr

noir-projects/aztec-nr/aztec/src/macros/notes/mod.nr

iAmMichaelConnor · 2025-01-22T20:24:56Z

noir-projects/aztec-nr/aztec/src/utils/bytes.nr

-pub fn bytes_to_fields<let N: u32>(input: [u8; N]) -> [Field; (N + 30) / 31] {
-    let mut dst = [0; (N + 30) / 31];
+// Note: ceil(N / 31) = (N + 30) / 31
+pub fn be_bytes_31_to_fields<let N: u32>(bytes: [u8; N]) -> [Field; (N + 30) / 31] {


Optimised vs the previous implementation that did a load of < checks on indices at runtime.

Did you measure this?

Also no need to throw in optimizations on top of further changes 🙃 this could just be its own thing

Did you measure this?

No, but each comparison is something like 15 constraints, so the if byte_index < N for every byte of the old version will have blown up.

Yeah, this is an example of me seeing something and going "I should fix this" instead of "I should open an issue for this and come back to it".

iAmMichaelConnor · 2025-01-22T20:25:51Z

noir-projects/aztec-nr/aztec/src/utils/field.nr

@@ -0,0 +1,124 @@
+global KNOWN_NON_RESIDUE: Field = 5; // This is a non-residue in Noir's native Field.


Ultimately, this should all go in the stdlib Field dir. It's all to get a sqrt function.

yarn-project/archiver/src/archiver/archiver_store_test_suite.ts

github-actions · 2025-01-22T22:13:34Z

Changes to public function bytecode sizes

Generated at commit: 5b4b6107e9a6ce8157c39a585fff7a0438e51d7a, compared to commit: ab2c860c747d3051a1cb85ad6ce5fac2a68867f7

🧾 Summary (100% most significant diffs)

Program	Bytecode size in bytes (+/-)	%
Token::finalize_mint_to_private	+168 ❌	+3.43%
Token::_finalize_mint_to_private_unsafe	+168 ❌	+3.40%
Token::finalize_transfer_to_private	+168 ❌	+3.20%
Token::_finalize_transfer_to_private_unsafe	+168 ❌	+3.17%
Token::complete_refund	+168 ❌	+2.91%
NFT::finalize_transfer_to_private	+118 ❌	+2.65%
NFT::_finalize_transfer_to_private_unsafe	+118 ❌	+2.62%
Token::public_dispatch	+168 ❌	+0.54%
NFT::public_dispatch	+118 ❌	+0.52%

Full diff report 👇

Program	Bytecode size in bytes (+/-)	%
Token::finalize_mint_to_private	5,063 (+168)	+3.43%
Token::_finalize_mint_to_private_unsafe	5,110 (+168)	+3.40%
Token::finalize_transfer_to_private	5,415 (+168)	+3.20%
Token::_finalize_transfer_to_private_unsafe	5,462 (+168)	+3.17%
Token::complete_refund	5,939 (+168)	+2.91%
NFT::finalize_transfer_to_private	4,569 (+118)	+2.65%
NFT::_finalize_transfer_to_private_unsafe	4,616 (+118)	+2.62%
Token::public_dispatch	31,313 (+168)	+0.54%
NFT::public_dispatch	22,850 (+118)	+0.52%

nventuro

I am so sorry about what you had to go through

docs/docs/migration_notes.md

noir-projects/aztec-nr/aztec/src/encrypted_logs/encrypt/aes128.nr

noir-projects/aztec-nr/aztec/src/encrypted_logs/encrypt/poseidon2.nr

noir-projects/aztec-nr/aztec/src/utils/bytes.nr

nventuro · 2025-01-23T18:12:27Z

noir-projects/aztec-nr/aztec/src/macros/notes/mod.nr

@@ -334,6 +330,7 @@ comptime fn generate_multi_scalar_mul(
                aztec::protocol_types::point::Point { x: $generator_x, y: $generator_y, is_infinite: false }
                },
            );
+            // TODO: explain why it's ok to use the unsafe `from_field_unsafe`, here.


Is it ok? Can we not use the fn?

I don't know. I couldn't determine whether it was safe or not, so I planted that todo as a flag.

noir-projects/aztec-nr/aztec/src/encrypted_logs/log_assembly_strategies/default_aes128/note.nr

yarn-project/txe/src/oracle/txe_oracle.ts

iAmMichaelConnor · 2025-01-25T17:15:37Z

I don't know how to diagnose these failed tests. Will need to find help from someone on Monday.
They seem to be an unrelated part of the stack. Maybe timing / polling issues?

iAmMichaelConnor · 2025-01-25T22:31:37Z

Converting this to draft. Superseded by #11503 (which is a superset of this PR), which is ready to me merged.

Addresses the comments from this old PR: #11400 Introduces Poseidon2 encryption, but doesn't plug it into any example contracts. --------- Co-authored-by: Nicolás Venturo <nicolas.venturo@gmail.com>

Addresses the comments from this old PR: AztecProtocol/aztec-packages#11400 Introduces Poseidon2 encryption, but doesn't plug it into any example contracts. --------- Co-authored-by: Nicolás Venturo <nicolas.venturo@gmail.com>

iAmMichaelConnor changed the title ~~WIP: encrypted logs refactor~~ refactor: encrypted logs refactor Jan 21, 2025

refactor encrypted logs stuff

Loading
Loading status checks…

7336241

iAmMichaelConnor force-pushed the mc/potato-encode branch from 2a98d40 to 7336241 Compare January 22, 2025 13:34

fix event log

Loading
Loading status checks…

ac59923

iAmMichaelConnor added 8 commits January 22, 2025 15:38

I found some annoying extra hard-coded stuff in the archiver

Loading
Loading status checks…

e10f5ef

Merge branch 'master' into mc/potato-encode

Verified

This commit was created on github.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

Loading
Loading status checks…

bdb8ee9

fixing that disgusting encoding

9c24ef6

Merge branch 'mc/potato-encode' of github.com:AztecProtocol/aztec-pac…

b1ea549

…kages into mc/potato-encode

merge w/ master

Loading
Loading status checks…

5c73e0a

nargo fmt

Loading
Loading status checks…

0dc91be

rm unused big aes fn

Loading
Loading status checks…

06ebdaf

nargo fmt

Loading
Loading status checks…

07acaa5

iAmMichaelConnor commented Jan 22, 2025

View reviewed changes

iAmMichaelConnor changed the title ~~refactor: encrypted logs refactor~~ feat!: encrypted logs refactor Jan 22, 2025

iAmMichaelConnor added 4 commits January 22, 2025 20:33

reinstate some tests and tidying

aa828ac

nargo fmt

Loading
Loading status checks…

e4fb1e8

Merge branch 'master' into mc/potato-encode

b53fd6c

more hard-coded partial logs stuff in the memory archiver store

Loading
Loading status checks…

82f5748

iAmMichaelConnor added 4 commits January 23, 2025 14:25

a comment

c33e819

merge

db309e5

missing await

Loading
Loading status checks…

8a4a88a

fix missed file txe_oracle

Loading
Loading status checks…

b609f4f

nventuro reviewed Jan 23, 2025

View reviewed changes

iAmMichaelConnor added 5 commits January 23, 2025 19:01

put migration notes in the right place

4b15d9f

address some poseidon2 encryption comments

3328c0c

revert debugging attempt

f323884

bytes.nr

fdf9530

sqrt in field.nr

42c409a

iAmMichaelConnor added 4 commits January 24, 2025 12:19

address review comments

b610bb5

address shared secret derivation comments

Loading
Loading status checks…

7e93c6f

rm poseidon2 encryption from this pr

Loading
Loading status checks…

033bd0e

stopping myself from renaming a thing

Loading
Loading status checks…

bb6d3dc

iAmMichaelConnor mentioned this pull request Jan 24, 2025

Make a PR for the poseidon2 encryption file #11476

Open

iAmMichaelConnor added 5 commits January 24, 2025 12:35

rm debug logs to my mum

Loading
Loading status checks…

f746835

fix tests

f375643

merge w/ master

Loading
Loading status checks…

be4cb2f

zeroed instead of [0; length]

Loading
Loading status checks…

450ddd5

revert that zeroed change, becuase the compiler didn't like it

Loading
Loading status checks…

b5225ef

iAmMichaelConnor mentioned this pull request Jan 25, 2025

refactor!: logs refactor #11503

Merged

Merge branch 'master' into mc/potato-encode

Verified

This commit was created on github.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

Loading
Loading status checks…

99e5d04

Merge branch 'master' into mc/potato-encode

Verified

This commit was created on github.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

Loading
Loading status checks…

548c629

iAmMichaelConnor marked this pull request as draft January 25, 2025 22:31

nventuro closed this Jan 27, 2025

nventuro deleted the mc/potato-encode branch January 27, 2025 17:48

iAmMichaelConnor mentioned this pull request Feb 11, 2025

feat: poseidon2 encryption #11919

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: encrypted logs refactor #11400

feat!: encrypted logs refactor #11400

iAmMichaelConnor commented Jan 21, 2025 •

edited

Loading

AztecBot commented Jan 22, 2025 •

edited

Loading

iAmMichaelConnor Jan 22, 2025

nventuro Jan 23, 2025

nventuro Jan 23, 2025

iAmMichaelConnor Jan 23, 2025

iAmMichaelConnor Jan 23, 2025

iAmMichaelConnor Jan 22, 2025

github-actions bot commented Jan 22, 2025 •

edited

Loading

nventuro left a comment

nventuro Jan 23, 2025

iAmMichaelConnor Jan 23, 2025

iAmMichaelConnor commented Jan 25, 2025 •

edited

Loading

iAmMichaelConnor commented Jan 25, 2025

		@@ -0,0 +1,124 @@
		global KNOWN_NON_RESIDUE: Field = 5; // This is a non-residue in Noir's native Field.

feat!: encrypted logs refactor #11400

feat!: encrypted logs refactor #11400

Conversation

iAmMichaelConnor commented Jan 21, 2025 • edited Loading

AztecBot commented Jan 22, 2025 • edited Loading

Docs Preview

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 22, 2025 • edited Loading

Changes to public function bytecode sizes

🧾 Summary (100% most significant diffs)

nventuro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iAmMichaelConnor commented Jan 25, 2025 • edited Loading

iAmMichaelConnor commented Jan 25, 2025

iAmMichaelConnor commented Jan 21, 2025 •

edited

Loading

AztecBot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

iAmMichaelConnor commented Jan 25, 2025 •

edited

Loading