fix: towards patterns and stores

Author: erights

packages/ERTP/src/interfaces.js

@@ -0,0 +1,45 @@
+/* eslint-disable no-use-before-define */
+/* global foo */
+const M = foo();
+
+export const MatchDisplayInfo = M.rest(
+  {
+    assetKind: M.or('nat', 'set'),
+  }.optionals({
+    decimalPlaces: M.number(),
+  }),
+);
+
+export const BrandI = M.interface({
+  isMyIssuer: M.callWhen(M.await(IssuerI)).returns(M.boolean()),
+  getAllegedName: M.call().returns(M.string()),
+  getDisplayInfo: M.call().returns(MatchDisplayInfo),
+});
+
+export const MatchAmount = {
+  brand: BrandI,
+  value: M.or(M.bigint(), M.array()),
+};
+
+export const IssuerI = M.interface({
+  getBrand: M.call().returns(BrandI),
+  getAllegedName: M.call().returns(M.string()),
+  getAssetKind: M.call().returns(M.or('nat', 'set')),
+  getDisplayInfo: M.call().returns(MatchDisplayInfo),
+  makeEmptyPurse: M.call().returns(PurseI),
+
+  isLive: M.callWhen(M.await(PaymentI)).returns(M.boolean()),
+  getAmountOf: M.callWhen(M.await(PaymentI)).returns(MatchAmount),
+});
+
+export const PaymentI = M.interface({
+  getAllegedBrand: M.call().returns(BrandI),
+});
+
+export const PurseI = M.interface({
+  getAllegedBrand: M.call().returns(BrandI),
+  deposit: M.apply(M.rest([PaymentI]).optionals([MatchAmount])).returns(
+    MatchAmount,
+  ),
+  withdraw: M.call(MatchAmount).returns(PaymentI),
+});

packages/marshal/docs/marshal-vs-store-level.md

@@ -0,0 +1,98 @@
+# Passables: Store  *vs* Marshal levels of abstraction
+
+We have three very distinct abstraction levels in our system in which to describe the passable data types and the operations on them. On the left is the higher *Store* level, containing the passable data types and operations of concern to the normal application programmer. A document intended for those application programmers would explain the Store level in a self contained manner. This is not that document.
+
+In the middle is the lower *Marshal* level of abstraction, which defines the semantics of the language-independent protocol. The Marshal level provides the data types and operations used to implement the Store level. The Marshal layer provides interoperability both between endpoints of different languages, and between endpoints defining higher layers instead of or in addition to the Store layer.
+
+On the right is the *JS* level, explaining how these map onto JavaScript language. This mapping determines how JavaScript values round trip or not through the protocol. Only hardened JavaScript values can be passable. The mapping of protocol concepts to JavaScript should serve as an example of how to map the protocol onto the concepts of other languages.
+
+|                | Store Level        | Marshal level                       | JS                                                    |
+| -------------- | ------------------ | ----------------------------------- | ----------------------------------------------------- |
+| Classification | `keyKindOf(p)`    | `passStyleOf(p)`<br>`getInterfaceOf(j)` | `typeof j`                    |
+| Type Testing   | `isKey(p)`         | `isOnlyData(p)`                     | `isPrimitive(j)`<br>`Array.isArray(j)`                |
+| Equivalence    | `sameKey(k,k)`     |                                     | `j === j`<br>`Object.is(j,j)`<br>`sameValueZero(j,j)` |
+| Ordering       | `compareKeys(k,k)` | `compareRank(p,p)`                  | `j <= j`<br>`[...js].sort(compare(j,j))`              |
+
+Where the parameter
+   * `j` is for any JavaScript value.
+   * `p` is for any `Passable`, a subset of JavaScript values.
+   * `k` is for any `Key`, a subset of passables.
+
+
+|            | `keyKindOf`                                    | `passStyleOf`                            | `typeof`           |
+| ---------- | ----------------------------------------------- | ---------------------------------------- | ------------------ |
+| Primitives | js primitives + `null`  | js primitives + `null` | `undefined` `boolean` `bigint`<br>`number` `string` `symbol` |
+| Containers | `copyArray` `copyRecord`<br>`copySet` `copyMap` | `copyArray` `copyRecord`<br>`tagged` | `object`           |
+| Behavior   | `remotable`                                     | `remotable`                              | `function`         |
+| Others     | No keys. <i>patternNodes</i><br>&nbsp;          | `tagged`<br>`error` `promise`        |                    |
+
+
+## The Marshal level organized by `passStyleOf`
+
+We organize according to the classification from<br>
+**`passStyleOf(p: Passable) => PassStyle`**<br>
+where the returned `PassStyle` is one of a fixed enumeration of strings, similar to the strings returned by JavaScript's `typeof`.
+
+`PassStyle` groupings:
+- **Primitive styles**: `"undefined"`, `"null"`, `"boolean"`, `"bigint"`, `"number"`, `"string"`, `"symbol"`
+- **Container styles**: `"copyArray"`, `"copyRecord"`, `"tagged"`
+- **Behavior style**: `"remotable"`
+- **Other styles**: `"error"`, `"promise"`
+
+Full `PassStyle` taxonomy:
+   * **The `PassStyle` styles**: The structure styles + the other styles
+     * **The structure styles**: The data styles + the behavior style
+         * **The data styles**: The primitive styles + the container styles
+           * **The primitive styles**: `"undefined"`, `"null"`, `"boolean"`, `"bigint"`, `"number"`, `"string"`, `"symbol"`
+           * **The container styles**: `"copyArray"`, `"copyRecord"`, `"tagged"`
+         * **The behavior style**: `"remotable"`
+     * **The other styles**: `"error"`, `"promise"`
+
+
+The **primitive styles** identify the *passable primitives*. The passable primitives have the expected mapping down to the JavaScript primitives. However, the passable primitives do not include JavaScript's `-0` or anonymous symbols. The included JavaScript primitives will all round-trip through the protocol.
+
+The **data styles** tag data-like containers. If a data-like container contains only data, then it is only data, passing the `isOnlyData` type test.
+  * A *copyArray* is a simple JavaScript array containing only passables.
+  * A *copyRecord* is a simple JavaScript record-like object, with only string-named own data properties containing only passables.
+  * A *tagged* is the extensibility point used by the Store level to implement *copySets*, *copyMaps*, and *patternNodes*.
+
+The **structure styles** introduce the *remotable*, which is a potentially remote behavioral object with a unique unforgeable identity and a primary location. At that primary location is the remotable's *Primary Remotable*, often referred to as a *far object*. At all other locations, the remotable is represented by a *remote*, implemented as a JavaScript handled-promise *presence*. Any message sent to a remotable is delivered to its primary remotable, invoking one of its methods. A remotable is a *structure*. A data-like container containing only structures, i.e., data and remotables, is itself a structure.
+
+Each remaining **passable** is either
+  * An *error* carrying diagnostic information.
+  * A *promise*, providing the eventual distributed access that JavaScript's promises were designed to provide.
+
+**`compareRank(left: Passable, right :Passable) => -1 | 0 | 1`**<br>
+implements a full order over *all* passables, suitable for sorting with `Array.prototype.sort`. However, it is a very imprecise full order. For example, for any two remotables `r` and `q`, `compareRank(r,q) === 0`. All remotables are therefore equivalent in this order. Likewise for all errors and all promises. Thus, `comparePassable`'s zero answer provides an equivalence class over all passables, but an imprecise one.
+
+ ## The Store level organized by `keyKindOf`
+
+**`keyKindOf(p: Passable) => KeyKind`**<br>
+ where `KeyKind` is an enumeration quite similar to `PassStyle`
+
+`KeyKind`:
+   * The Store's **primitive styles** are identical to Marshal's.
+   * The Store's **data styles** styles preserves the `"copyArray"` and `"copyRecord"` classification from Marshal's data styles, omits `"tagged"`, but uses it to implement the `"copySet"`, `"copyMap"`, and the various patternNodes that it introduces. The store level must also recognize whether a tagged that claims to be a copySet or copyMap has a valid representation. A tagged that does not validate as a copySet or copyMap is not a key. patternNodes are not keys regardless.
+   * Store's **"remotable"** is identical to Marshal's. Every remotable is a key. On two remotables `p` and `q`, `sameKey(p,q)` compares using the remotable's unique unforgeable identity. Thus, for two remotables, `sameKey(p,q)` iff `p === q` iff `Object.is(p,q)` iff `sameValueZero(p,q)`.
+   * The remaining passables: errors and promises, are not keys.
+
+Primitives and remotables are keys. copyArrays, copyRecords, copySets, and copyMaps are key-like containers. If a key-like container contains only keys, then it is a key.
+  * A *copySet* can only contain keys, and so each copySet is necessarily a key.
+  * A *copyMap* entry can use only use a key as a key, but can use any passable as a value including unrecognized taggeds. But a copyMap is a key only of all of its values are keys.
+
+**`sameKey(k1 :Key, k2: Key) => boolean`**<br>
+defines an equivalence class of keys. For example, a copySet is encoded into a tagged in sorted order according to `compareRank`. Because remotables are distinct but cannot be canonically ordered, two copySets `s` and `t` representing the same logical set may be encoded into different tagged representations. In that case, `sameKey(s,t)` would still be true.
+
+**`compareKeys(left :Key, right: Key) => -1 | 0 | 1 | undefined`**<br>
+defines a partial order over all keys, comparing magnitudes. `compareKeys` returns `undefined` to indicate that the two keys are incommensurate, that their magnitudes cannot meaningfully be compared. Some example of how this differs from `compareRank`:
+   * `compareRank`, in order to sort all passables, compares passables of different pass styles according to the order of those
+     pass styles listed above. `compareKeys`, applied to keys of different key styles will always return `undefined`.
+   * `compareRank` orders `NaN` after all other numbers. Which is an arbitrary choice, but some choice was necessary. In order to preserve reflexivity, in violation of IEEE floating point recommendations both `compareRank(NaN,NaN) === 0` and `compareKeys(NaN,NaN) === 0`. But comparing `NaN` and anything else using `compareKeys` will produce `undefined`.
+   * Applied to two copySets `s` and `t`, `compareRank(s,t)` will do a lexicographic comparison of their tagged representation. `compareKeys(s,t)` will check if either is a subset of the other. If each contains elements the other lacks, then `compareKeys` will return `undefined`.
+   * Given two distinct remotables `r` and `q`, `compareRank(r,q) === 0` but `compareKeys(r,q)` === undefined.
+
+## The copySet example
+
+The best example to understand this layering is how the Store layer implements its copySets out of the tagged provided by the Marshal layer. As a set, it must store each element only once. This _only once_ requirement means we must limit sets to contain only objects we can compare according to some equivalence. It would be appealing to be able compare the sorted encoding into a tagged directly. Due to the constraints of permissionless loosely-coupled decentralized systems, remotables _cannot_ be canonically ordered.
+
+Thus, a copySet containing remotables cannot be canonically sorted. Instead, a copySet is implemented as a tagged containing a copyArray sorted according to `compareRank`. Because remotables cannot be canonically ordered, `sameKey(r,s)` of two remotables `r` and `s` will always return `0`, meaning that they sort as equivalent. Thus two copySets that contain the same set of remotables might represent their elements with two different orders among those remotable. `sameKey` will do an order independent comparison of the copySets, but still taking advantage of the fact that the `tagged` representation is still as sorted as it can be according to `compareRank` applied to the copySet's elements.

packages/store/docs/store-taxonomy.md

@@ -0,0 +1,67 @@
+# A Taxonomy of Stores
+
+Some dimensions on which our Stores differ. For each of these dimensions, in the headings below we indicate the dominant default choice with an asterisk (*). A given Store maker should document which non-default points in this design space it supports. The `makeScalarMapStore` function makes stores that default to all the default choices below, allowing only the non-default option to specify a schema, to create a schematized store.
+
+All stores are Far objects (a kind of remotable) that may be snapshotted to (or restored from) a corresponding pass-by-copy objects. MapStores snapshot into CopyMaps. SetStores snapshot into CopySets.
+
+# API Differences
+
+## SetStores vs MapStores*
+
+* SetStores store only keys (of type Key), where each key is key-unique, i.e., it is not keyEQ to any other key in the set.
+* MapStores store key value associations, where the keys are key-unique Keys, as with sets, and the associated values are Passable.
+
+## Weak vs Strong*
+
+* A weak store does not enable its contents to be iterated. Only with the key of a given entry will a weak store reveal anything about the entry. This sometimes enables gc of entries that the collector knows can never be mentioned. But, unlike JavaScript WeakSets and WeakMaps, a weak store *may* allow other values as keys, such as numbers, that can be synthesized at will. Entries indexed by such synthesizable keys can never be collected.
+* A strong store does enable iteration of its contents, and so must retain those contents as long as the strong store itself exists.
+
+# Storage and Representation Differences
+
+## Scalar Keys Only* vs Composite Keys Allowed
+
+* Scalars are primitives or remotables, those keys that can be compared without looking inside objects. This directly reflects the semantics of JavaScript's Maps and Sets, which only index by scalar keys.
+* Composite keys are things like copyArrays, copyRecords, copySets, or copyMaps, that compare by structural key equality.
+
+## Unstructured* vs Schematized
+
+* An unstructued store is restricted only by the representation limitations listed here. They keys must be Keys. The values must be Passable. Scalar stores allow only scalar keys. Persistent stores only allow contents that are well defined after vat death.
+* A schematized store is further restricted by a pattern provided when it was created to serve as its schema. A schematized store will only allow contents that match its schema pattern.
+
+## Long* vs Short Expected Lifetime (WeakStores only)
+
+The expected lifetime of a weak store helps us optimize its representation.
+* If we expect a weak store to outlive most of its keys, we may internally use a JavaScript WeakMap.
+* If we expect the keys in a weak store to outlive the store, we may internally use a JavaScript Map, to avoid the costs often associated indexing a key for JavaScript WeakMaps that are no longer alive. Though keep in mind that this representation denies the garbage collector the opportinity to collect those keys while these expected-to-be-short-lived stores are still alive.
+
+## Heap* vs Virtual
+
+* Like JavaScript Maps and Sets, heap stores live in the JavaScript language heap, limited by the size of heap memory and occupying room in vat snapshots.
+* Virtual collections potentially live outside the heap. A virtual store serializes (marshals) its contents to store it, and unserializes (unmarshals) its contents to retrieve it. Virtual objects (a kind of far object) that are accessible *only* from virtual stores can completely disappear from the heap, to be restored if any serialized reference to it is restored. A virtual store is itself a virtual object.
+
+## Ephemeral* vs Persistent (Heap is always Ephemeral)
+
+Stores only exist in one vat at a time, giving us free transactional integrity. All heap stores are ephemeral. Virtual stores can be ephemeral or persistent.
+* Ephemeral stores only exist in the vat that initially created them, dying when that vat dies. The content of virtual ephemeral stores, can contain references to ephemeral non-virtual remotables or promises.
+* A persistent store can be part of a vat's "estate", to be transfered to that vat's heir/successor on the death of its current vat. A persistent store can only contain keys and values that are still meaningful across such traumas, such as
+    * persistent objects (a kind of virtual object)
+    * Remotes, a kind of remotable designating a far object in another vat
+    * Promises, perhaps. These are tempting because they can be restored into a well-defined severed state, the rejected promise.
+
+    A persistent store may not store a reference to a local non-persistent far object. A persistent store is itself a persistent object.
+
+# Iteration Differences
+
+## Rank Order* vs Altered Rank Order Iteration
+
+* Our stores iterate in rank-sorted order of keys. Although rank-order is normally a lower-level implementation concept, the invariants between key-order (a meaningful partial order) and rank-order (an implementation-oriented full order) make this potentially useful, even for code that is only aware of key-order:
+    * If X key== Y, then X rank== Y.
+    * If X key< Y, then X rank<= Y.
+
+* Altered Rank Order is not yet implemented, and may not be for a while. This is a placeholder for a declarative description of a bidirectional 1-to-1 transform of schematized store content from the form seen through the API (the API form) vs the store content as it is actually stored (the storage form).
+
+  In altered rank order, the contents will be organized and iterated according to the rank order of the storage form, but retrieved as transformed back to the API form. In order to preserve the internal rankCover optimization, for indexing into rankStores to find pattern matches, the patterns themselves need to be transformed through the bijection. Fortunately, this pattern transformation need not be bidirectional. The purpose of altered rank order is more to enable better rankCover optimizations for certain patterns, rather than any interest in alternate iteration orders.
+
+  For example, if we want to organize keys like `{foo,bar}` so that they are sorted with `bar` as more significant, we may transform these to `makeTagged('{bar,foo}',[bar,foo])`. Tagged objects of the same tag sort according to the rank order of their payload, which is lexicographic for arrays. If we reserve such tag names for this purpose, we might be able to universally untransform the storage form back into the original API form. If we coalesce the storage for identical tag names, then this resembles a "hidden class" representational technique.
+
+  We expect the expression of bijections to be intimately related to the expression of schema.