diff --git a/FIPS/fip-0032.md b/FIPS/fip-0032.md
index 8ba93b0dc..dc2a6388d 100644
--- a/FIPS/fip-0032.md
+++ b/FIPS/fip-0032.md
@@ -1,7 +1,7 @@
---
fip: "0032"
-title: Gas model adjustment for non-programmable FVM
-author: Raúl Kripalani (@raulkr), Steven Allen (@stebalien)
+title: Gas accounting model adjustment for non-programmable FVM
+author: Raúl Kripalani (@raulkr), Steven Allen (@stebalien), Jakub Sztandera (@Kubuxu)
discussions-to: https://github.com/filecoin-project/FIPs/discussions/316
status: Draft
type: Technical
@@ -15,296 +15,518 @@ requires:
replaces: N/A
---
-# FIP-0032: Gas model adjustment for non-programmable FVM
+# FIP-0032: Gas accounting model adjustment for non-programmable FVM
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Simple Summary](#simple-summary)
+- [Abstract](#abstract)
+- [Change Motivation](#change-motivation)
+- [Specification](#specification)
+ - [Execution gas](#execution-gas)
+ - [Syscall gas](#syscall-gas)
+ - [Extern gas](#extern-gas)
+ - [IPLD state management gas](#ipld-state-management-gas)
+ - [Milligas precision and under/overflow handling](#milligas-precision-and-underoverflow-handling)
+- [Design Rationale](#design-rationale)
+ - [Gas sizing baseline](#gas-sizing-baseline)
+ - [Benchmark methodology](#benchmark-methodology)
+- [Backwards Compatibility](#backwards-compatibility)
+- [Test Cases](#test-cases)
+- [Security Considerations](#security-considerations)
+- [Incentive Considerations](#incentive-considerations)
+- [Product Considerations](#product-considerations)
+- [Implementation](#implementation)
+- [Assets](#assets)
+- [Annex A: Table of gas utilization increase per message type](#annex-a-table-of-gas-utilization-increase-per-message-type)
+- [Annex B: Table of total final gas used per message type](#annex-b-table-of-total-final-gas-used-per-message-type)
+- [Acknowledgements](#acknowledgements)
+- [Copyright](#copyright)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## Simple Summary
-This FIP is part of a two-FIP series that adjust the gas model of the Filecoin
-blockchain in preparation of user-programmable actors. This FIP focuses on
-attaining high-fidelity gas accounting under the Wasm runtime that built-in
-system actors will migrate to as part of FIP-0030 and FIP-0031. It does not
-cover upcoming system features like actor deployment, actor loading, memory
-management, and more. Those will be addressed in a subsequent FIP that focuses
-on gas model changes to support user-programmability.
+This FIP is part of a two-FIP series that adjust the gas accounting model of the
+Filecoin blockchain in preparation of user-programmable actors. This FIP focuses
+on achieving high-fidelity gas accounting under the new Wasm runtime that
+built-in system actors will migrate to as part of [FIP-0030] and [FIP-0031]. It
+does not cover upcoming system features like actor deployment, actor dispatch,
+actor loading, memory management, and more. Those will be covered in a future
+FIP.
## Abstract
-With the arrival of the Filecoin Virtual Machine, the Filecoin network will gain
-the capability to run untrusted code in the form of user-defined actors.
-However, the current gas model is designed strictly for predefined, trusted
-code. In order to maintain network security guarantees, the gas model needs to
-evolve to accurately charge for computational instructions and resource usage of
-actor logic.
-
-This FIP expands the current gas model by introducing a new gas category to
-account for the real compute cost of the logic performed by every actor in the
-system. It also rearchitects IPLD state management gas, adjusts the actor call
-fee to account for Wasm invocation container mechanics, and revises the fees for
-syscalls that traverse the extern boundary (FVM <> Node).
+The current gas accounting model was designed to cover the execution of
+predefined, trusted code developed by the Filecoin Core Devs.
-This is the first FIP out of a two-FIP series to revise the gas model as a
-result of the introduction of the Filecoin Virtual Machine.
+With the arrival of the Filecoin Virtual Machine, the Filecoin network will gain
+the capability to run untrusted code in the form of user-defined actors. Thus,
+the gas accounting model needs to evolve to price arbitrary code.
-This FIP is designed to accompany FIP-0030 and FIP-0031 under the same network
-upgrade. Taken together, these three FIPs represent the theory and process of
-change for introducing an FVM with non-programmable smart contracts.
+This FIP evolves the current gas accounting model by introducing the concepts of
+_execution gas_, _syscall gas_, and _extern gas_. Furthermore, this FIP revises
+the IPLD state management fees to adapt to the new design.
-The second gas change FIP will be proposed and considered at a later date in
-order to support the introduction of user-programmable smart contracts during
-phase 2 of FVM roll-out.
+This FIP is designed to accompany [FIP-0030] and [FIP-0031] under the same network
+upgrade (Milestone 1 of the FVM roadmap).
## Change Motivation
Gas is an essential mechanism to (a) compensate the network for the computation
-performed during transaction execution, and to (b) restrict the compute capacity
-of blocks to achieve a designated epoch time (30 seconds).
-
-Today, Filecoin determines the amount of gas a transaction consumes by
-intercepting specific operations and actions, and applying gas explicit gas
-charges on them. Concretely:
-
-- On including a chain message, variable on the length of the message.
-- On returning a value, variable on the length of returned payload.
-- On performing a call, dependent on the type of call (simple value transfer, or
- other).
-- On performing an IPLD store get, fixed amount.
-- On performing an IPLD store put, variable on block payload size.
-- On actor creation, fixed amount.
-- On actor deletion, fixed amount.
+performed during the execution of transactions, (b) allocate chain bandwidth,
+and to (c) restrict the compute capacity of blocks so that the designated epoch
+time (30 seconds) can be sustained with contemporary hardware.
+
+Today, transactions on the Filecoin network are charged gas on these operations:
+
+- On including a chain message (variable gas units, based on length).
+- On returning a value (variable gas units, based on length of returned
+ payload).
+- On performing a call (variable gas units, depending on whether it's a simple
+ value transfer, or an invocation to actor logic).
+- On performing an IPLD store get (fixed gas units).
+- On performing an IPLD store put (variable gas units, based on block payload
+ size).
+- On actor creation (fixed gas units).
+- On actor deletion (fixed gas units).
- On cryptographic operations:
- - On signature verification, variable on signature type and plaintext
- length.
- - On hashing, fixed price.
- - On computing the CID of an unsealed sector, variable on proof type and
- piece characteristics.
- - On verifying a seal, variable on the seal characteristics.
- - On verifying a window PoSt, variable on the PoSt characteristics.
+ - On signature verification (variable gas units, depending on signature type
+ and plaintext length).
+ - On hashing (fixed gas units).
+ - On computing the CID of an unsealed sector (variable gas units, based on
+ proof type and piece properties).
+ - On verifying a seal (variable gas units, based on seal properties).
+ - On verifying a window PoSt (variable gas units, based on PoSt properties).
- On verifying a consensus fault.
-None of these gas charges accounts for the costs of the execution of the actor’s
-compute logic itself. This results in an incomplete resource accounting model.
-So far, the network has coped without it because:
+None of these gas charges accounts for the actual costs of the execution of the
+actor's compute logic itself. This results in an imprecise resource accounting
+system. So far, the network has coped without it because:
-1. The cost of built-in actor logic is negligible when compared to the cost of
- the above operations, most of which are triggered through syscalls.
+1. With the exception of some methods, the cost of most built-in actor logic is
+ proportional to the cost of the above operations, most of which are triggered
+ through syscalls.
2. Only pre-determined, trusted logic runs on chain in the form of built-in
actors.
-3. Technical limitations. Prior to the FVM (FIP-0030, FIP-0031), there was no
+3. Technical limitations. Prior to the FVM ([FIP-0030], [FIP-0031]), there was no
portable actor bytecode whose execution could be metered identically across
all implementations.
With the ability to deploy user-defined actors on the FVM, these assumptions no
-longer stand and it becomes important to start tracking real execution costs
-with the highest fidelity possible, to preserve security guarantees, and to
-compensate the network for the logic it executes.
+longer stand and it's vital to start tracking real execution costs with the
+highest fidelity possible in order to preserve security guarantees.
## Specification
-Here’s an overview of proposed changes to the existing gas model. We explain
-each one in detail in the following sections.
+Here's an overview of proposed changes to the existing gas accounting model. We
+explain each one in detail in the following sections.
-- Introduce Wasm bytecode-metered execution gas.
-- Adjust the *actor call* fee, to account for new call dispatch overhead.
-- Rearchitect the IPLD state IO fees.
-- Revise fees for extern-traversing syscalls.
+- Introduce execution gas, metered by Wasm bytecode.
+- Introduce syscall gas.
+- Introduce extern gas.
+- Redesign the IPLD state management fees.
-### Wasm bytecode-metered execution gas
+### Execution gas
-We introduce a new category of gas. Execution gas is charged per Wasm
-instruction. Every Wasm instruction has a predefined amount of execution units
-associated with it.
+Execution gas is charged per [Wasm instruction][wasm-instructions]. Every Wasm
+instruction expends a number of execution units, which can be static or dynamic
+based on its parameters.
-Every Wasm instruction expends a flat 1 execution unit, except the following
-structural Wasm instructions: `nop`, `drop`, `block`, `loop`, `unreachable`,
-`return`, `else` and `end` . These instructions expend 0 execution units
-because either they do not translate into machine instructions by themselves, or
-they are control flow instructions with no logic to evaluate.
+As part of this FIP, we assign a flat value of 1 execution unit for every Wasm
+instruction, with the exception of structural instructions. These instructions
+are: `nop`, `drop`, `block`, `loop`, `unreachable`, `return`, `else` and `end`.
+The exemption is due to the fact that these Wasm instructions do not translate
+into machine instructions by themselves, or they are control flow instructions
+with no associated logic.
**Pricing formula**
-The conversion of Wasm execution units to Filecoin gas units is at a predefined
-ratio of `<<TODO: currently benchmarking>>`. This equivalence has been
-calculated by performing extensive benchmarks, with the goal of honouring the
-invariant baseline target of 10 gas units/ns of wall clock time[^1].
+The conversion of Wasm execution units to Filecoin gas units is at a fixed ratio
+of 4gas per execution unit.
-Both the pricing policy for Wasm instructions and the conversion rate need to be
-assessed and revised continuously as technology improves (e.g. Wasm compilers
-and processors). The ongoing goal is to sustain high execution fidelity.
+This ratio is consistent with the average instruction throughput observed in
+benchmarks, and tracks the baseline target of 10 gas units/ns of wall clock
+time.
+
+```
+p_gas_per_exec_unit := 4
+```
-### Call gas fee
+**Future considerations**
-In Filecoin network version 15 and earlier, the call gas fee is higher for value
-transfers than it is for actor code invocations. This may sound
-counterintuitive, but it follows the fact that the call fee has to cover the
-full validation and state update cost of the transaction, because these
-transactions don’t invoke actor code. Conversely, the call fee for an actor
-invocation represents only the dispatch effort, hence why it’s lower.
+1. Our benchmarks revealed that memory instructions perform differently to pure
+ CPU-level instructions. They also exhibit high variance depending on the
+ memory access pattern of the actor, CPU caching dynamics, and other factors.
+ At this stage, we have opted to simplify the model by setting a fixed average
+ rate for all instructions. But in the future, we may tabulate execution gas
+ per instruction, or per instruction family.
+2. The pricing policy for Wasm instructions and the conversion rate need to be
+ assessed and revised continuously as Wasm compilers and hardware evolve. The
+ ongoing goal is to sustain high execution fidelity.
-With FVM actors, the call cost structure changes. For actor code invocations, a
-Wasm-based invocation container needs to be instantiated, and parameters need to
-be serialized and inserted into the IPLD blockstore. In other words, there is
-more work to do that than previously, and therefore the fee is adjusted upward.
+### Syscall gas
+
+Syscalls are host-provided functions exposed to actor code. They allow the actor
+to traverse the sandbox in a controlled manner to:
+
+- access data (e.g. state data, chain data, randomness).
+- perform side effects (e.g. write to state).
+- invoke operations that are more efficient to implement in native space than in
+ Wasm (e.g. cryptograhic primitives).
+
+Calling a syscall triggers a context switch out, and back in when execution
+returns to Wasm. The syscall gas merely covers this overhead. It does not price
+the actual work performed by the syscall; that work is charged for separately,
+and is specific to every syscall.
**Pricing formula**
+We have quantified the average context switch overhead to be equivalent to 3500
+execution units. At the rate of 4 gas per execution unit, we set the syscall gas
+price to be 14000 gas.
+
```
-TODO: benchmarking
+p_syscall_gas := 14000
```
-### IPLD state management fees
-
-Currently, IPLD block read incur a fixed fee, and IPLD block writes incur a
-variable fee, dependent on the number of bytes written.
-
-With the FVM, IPLD state I/O is managed through five syscalls, with the
-specified overhead and pricing. We introduce a `block_memcpy_per_byte` cost for those
-operations that involve a memcpy. The overhead and pricing are as follows:
-
-**Overhead**
-
-- `block_open`: looks up the supplied CID, loads the block from the state
- blockstore into the FVM kernel’s block cache, reports its size and codec. It
- returns a numeric block handle.
- - Overhead: This syscall traverses the FVM<>client boundary via an extern in
- order to query the state blockstore. This leads to disk IO with associated
- read amplification. The bytes returned through the extern are copied into
- the FVM’s memory so that the buffer can be reclaimed by the client side;
- thus there is an alloc and memcpy cost. The bytes are retained in memory
- until the actor call finishes and the invocation container is dropped, and
- thus the actor is charged for this memory usage.
-- `block_read`: takes a block handle and an output buffer, and copies the
- block’s data from the FVM kernel’s block cache into the actor’s memory.
- - Overhead: This syscall resolves inside the FVM and doesn’t traverse the
- FVM<>node boundary. The overhead is that of a memcpy operation.
-- `block_write`: takes some block data and a codec, and merely copies it to the
- FVM kernel’s block cache. It returns a numeric block handle.
- - Overhead: This syscall resolves inside the FVM and doesn’t traverse the
- FVM<>node boundary. It involves a memcpy to a buffer that is retained
- until the actor call finishes. The bytes are retained in memory until the
- actor call finishes and the invocation container is dropped, and thus the
- actor is charged for this memory usage.
-- `block_link`: computes the CID of a block, and commits the block to the state
- blockstore.
- - Overhead: This syscall traverses the FVM<>client boundary via an extern in
- order to write to the state blockstore (although this is deferred until
- the machine flush). This leads to disk IO with associated write
- amplification. The bytes are copied over to the client’s memory, thus
- there is an alloc and memcpy cost. Computing the CID involves a hashing
- operation (akin to the hash syscall, but without the syscall overhead).
- Because the CID is returned to the actor via an output buffer, there is
- another memcpy involved.
-- `block_stat`: returns the length and the codec of a block.
- - Overhead: This syscall resolves inside the FVM and doesn’t traverse the
- FVM<>node boundary. It performs a map lookup and returns an integer tuple.
-
-**Pricing formulae**
+**Future considerations**
-- `block_open` (extern-traversing): `block_open_base + block_open_per_byte * len`
-- `block_read`: `block_read_base + block_memcpy_per_byte * len`
-- `block_write`: `block_create_base + block_memcpy_per_byte * len`
-- `block_link` (extern-traversing): `block_link_base + block_link_per_byte * storage_gas_multiplire * len`
-- `block_stat`: `block_stat` (fixed cost)
+The cost of syscall-driven context switches varies depending on the syscall, its
+parameters, the amount of work it performs (thus invalidating CPU caches and
+instruction pipeline), and other factors. In the future, we may tabulate syscall
+gas depending on the complexity of the syscall, its arguments, its impact on CPU
+optimizations, and more.
-### Extern-traversing syscall fee revision
+### Extern gas
-Some syscalls cannot be resolved entirely within FVM space. Such syscalls
-need to traverse the "extern" (FVM<>node) boundary to access client
-functionality.
+Some syscalls cannot be resolved entirely within FVM space. Such syscalls need
+to traverse the _extern_ (FVM <> Node) boundary to access client functionality.
Depending on the languages of the client and the FVM implementation, traversing
-this boundary may involve foreign-function interface (FFI) mechanics.
-
-That is indeed the case with the reference client (Lotus) and the reference FVM
-(ref-fvm). Not only is this the worst case scenario, but it's the most common
-scenario today given the client distribution of the network.
-
-In accordance to the design principle of high-execution fidelity, we propose
-that the gas fees account for the cost of the FFI boundary.
+this boundary may involve foreign-function interface (FFI) mechanics. That is
+indeed the case with the reference client (Lotus) and the reference FVM
+(ref-fvm). This is the most common scenario today given the client distribution
+of the network.
-The associated costs are:
+The associated overheads are:
-- data structure translation between representations, including serialization
- costs.
+- data structure translation between representations, including potential
+ serialization overheads.
- context switch.
- the memory overhead on both sides of the interface.
-- the amortized (opportunity) cost of dedicating OS threads.
+- the amortized opportunity cost of dedicating OS threads to handle FFI calls.
- the amortized garbage collection or deallocation costs once the extern
concludes.
- function lookup and dispatch.
**Pricing formulae**
-The pricing formulae for extern-traversing syscalls are:
+Extern gas is priced at 1.5x syscall gas. It is equivalent to a flat 5250
+execution units, and therefore it costs 21000 gas. Extern gas is additive to
+syscall gas.
+
+```
+p_extern_gas := 21000
+```
+
+The following syscalls are extern-traversing syscalls, and therefore are subject
+to extern gas:
+
+- `rand::get_chain_randomness`.
+- `rand::get_beacon_randomness`.
+- `crypto::verify_consensus_fault`.
+- `ipld::block_open`.
+
+**Future considerations**
+
+It may be possible to drop the extern traversal on randomness fetches, by
+passing in a reference to a ring-buffer-like data structure (maintained by the
+Node) on Machine construction. This data structure would hold 2*finality worth
+of VRF proofs & beacons, for the FVM kernel to index into directly, thus
+eliminating the extern access.
+
+- The estimated size is 405Kb + data structure overhead (225 bytes of data per
+ epoch x 1800 epochs).
+- This future improvement is tracked under [filecoin-project/ref-fvm#525](https://github.com/filecoin-project/ref-fvm/issues/525)
+ for the reference FVM implementation.
-- `get_chain_randomness`: `<<TODO>>`
-- `get_beacon_randomness`: `<<TODO>>`
-- `verify_consensus_fault`: `<<TODO>>`. This is a fixed cost that covers all blockstore operations,
- signature verifications, and other logic needed to resolve whether a consensus fault has occured.
- Note that differs from network v15 and earlier, where the cost is variable depending on the exact
- blockstore operations performed.
-- Blockstore operations: already absorbed in the fees specified above ("IPLD
- state management fees").
+### IPLD state management gas
-### Under/overflow handling
+Currently, IPLD state block operations incur these fees:
-Operations that have the potential of either underflowing or overflowing should be performed
-using [saturation arithmetic](https://en.wikipedia.org/wiki/Saturation_arithmetic). The valid
-range for any fuel operation, including total fuel consumed, is [0, 2<sup>64</sup>-1].
+- Reads incur a fixed fee.
+- Writes incur a variable fee, based on the number of bytes written.
-The valid range for any gas operation, including gas available and total gas used, is
-[0, 2<sup>63</sup>-1]. Gas operations only go up to 2^63-1 because they are performed using
-signed arithmetic. This is because the Filecoin protocol has some gas costs that are negative as
-of nv15.
+With the introduction of the FVM, the state management operations have evolved
+from a two-method design to five syscalls:
+
+- `ipld::block_open`
+- `ipld::block_read`
+- `ipld::block_write`
+- `ipld::block_link`
+- `ipld::block_stat`
+
+For more details, refer to `ipld` namespace syscall catalogue in
+[FIP-0030](https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0030.md#namespace-ipld).
+
+**memcpy per byte gas charge**
+
+We introduce a gas charge for memcpy'ing bytes across system boundaries
+(`p_memcpy_gas_per_byte`). The fee is 500 milligas per byte, resulting from
+statistics collected on average memcpy latencies, adapted to the Filecoin gas
+sizing baseline (10gas per ns).
+
+```
+p_memcpy_gas_per_byte := 0.5
+```
+
+**Memory retention per byte gas charge**
+
+Some operations retain buffers in FVM Kernel and Machine space. We introduce a
+gas charge to cover memory retention, modelled after a 1GB limit of memory usage
+per transaction (worst case scenario, contemporary block validator hardware).
+
+```
+p_memret_gas_per_byte := 10
+```
+
+**Pricing for `ipld::block_open`**
+
+_Overhead:_ This syscall traverses the FVM <> Node boundary via an extern to
+access the state blockstore. The bytes returned through the extern are copied
+into FVM kernel buffers; thus there is an memcpy and alloc cost (the latter is
+absorbed into the former). The kernel buffers are retained until the actor call
+finishes.
+
+The pricing formula is:
+
+```
+p_blockopen_base_gas := 114617 # extant fee
+op_ipld_blockopen := p_syscall_gas
+ + p_extern_gas
+ + p_blockopen_base_gas
+ + (p_memcpy_gas_per_byte * block_length)
+ + (p_memret_gas_per_byte * block_length)
+```
+
+**Pricing for `ipld::block_read`**
+
+_Overhead:_ This syscall performs an memcpy from the FVM kernel's buffers to
+Wasm memory.
+
+The pricing formula is:
+
+```
+op_ipld_blockread := p_syscall_gas + (p_memcpy_gas_per_byte * byte_length)
+```
+
+**Pricing for `ipld::block_write`**
+
+_Overhead:_ This syscall performs an memcpy from Wasm memory into the FVM
+kernel's buffers, with a corresponding alloc (absorbed into the memcpy cost).
+The kernel buffers are retained until the actor call finishes.
+
+The pricing formula is:
+
+```
+op_ipld_blockwrite := p_syscall_gas
+ + (p_memcpy_gas_per_byte * block_length)
+ + (p_memret_gas_per_byte * block_length)
+```
+
+Future considerations:
+
+- This call will likely become more expensive when user programmability is
+ introduced as `ipld::block_write` will need to reason about data
+ availability/reachability.
+
+**Pricing for `ipld::block_link`**
+
+_Overhead:_ This syscall incurs a hashing operation to compute the CID of the
+block. It then copies it over to the buffered blockstore, where it is retained
+until the Machine is flushed and the blocks are written to the Node's blockstore
+via an extern. The extern is amortized across all the blocks written throughout
+the Machine's lifetime, and the cost is included in the storage multiplier.
+
+The pricing formula for compute gas is:
+
+```
+p_blocklink_base_gas := 353640 # extant fee
+p_storage_gas_multiplier := 1300 # extant fee
+op_ipld_blocklink_compute := p_syscall_gas
+ + p_blocklink_base_gas
+ + (p_memcpy_gas_per_byte * block_length * 2)
+op_ipld_blocklink_storage := p_storage_gas_multiplier * block_length
+op_ipld_blocklink := op_ipld_blocklink_compute + op_ipld_blocklink_storage
+```
+
+Note: we don't charge a memory retention cost here because it's already factored
+into the base operation gas (this operation is equivalent in overhead to the
+pre-FVM `StorePut`).
+
+**Pricing for `ipld::block_stat`**
+
+No relevant overhead.
+
+The pricing formula is:
+
+```
+op_ipld_blockstat := p_syscall_gas
+```
+
+### Milligas precision and under/overflow handling
+
+The unit of internal gas accounting is milligas (1e3 precision, increments of
+1/1000). When message execution completes, or gas is queried by an actor, the
+FVM rounds the _gas used_ up, and the _gas available_ down, to the nearest whole
+gas.
+
+Operations that have the potential of either underflowing or overflowing should
+be performed using [saturation arithmetic](https://en.wikipedia.org/wiki/Saturation_arithmetic).
+The valid range for any execution units operation, including total execution units
+consumed, is [0, 2<sup>64</sup>-1].
+
+The valid range for any gas operation, including gas available and total gas
+used, is [0, (2<sup>63</sup>-1)/1000]. Internally, during message execution, gas
+used may ephimerally turn negative as of Filecoin nv15 (gas refunds). Upon
+concluding message execution, gas used is clamped to 0 on the lower bound, thus
+effectively setting a floor of 0.
## Design Rationale
-To simplify community discussion and align with the FVM deployment roadmap, all
-changes related to new system features related to user-programmable actors are
-kept out of scope of this FIP. This set of changes covers only the technological
-transition to a Wasm runtime for built-in actors. This FIP will be followed by
-another FIP introducing the former.
+### Gas sizing baseline
+
+We use a baseline of 10 gas/nanosecond of wall-clock time, or 1 milligas/10
+picosecond. This baseline follows from the need to sustain a 30s epoch by
+validating and applying unique messages in a tipset, with a block count
+following a Poisson distribution (win rate), with a 10B block gas limit,
+assuming 6s of block propagation time.
+
+### Benchmark methodology
+
+The parameters for this FIP were calculated through extensive benchmarking using
+three profiling techniques to obtain a performance data from various vantage
+points:
+
+1. Fine-grained tracing of 2.5MM mainnet message applications.
+2. Aggregated execution statistics of 2.5MM mainnet messages.
+3. Raw Wasmtime micro-benchmarks.
+
+The machines had the following specs: 1 x AMD EPYC 7402P, 24 cores @ 2.8 GHz, 2
+x 240GiB SSD (boot), 2 x 480GiB SSD (storage), 64GiB DDR4 RAM, running Debian
+4.19.208-1.
+
+The code assets for (1) and (2) are available at
+[filecoin-project/ref-fvm#465](https://github.com/filecoin-project/ref-fvm/pull/465).
-Concerning the design of each individual change, the specification section
-elaborates on overhead that is being accounted for and the variables that make
-up every pricing formula.
+The raw Wasmtime micro-benchmarker is available at
+[filecoin-project/ref-fvm#502](https://github.com/filecoin-project/ref-fvm/pull/502).
+
+Benchmarking overheads were measured and corrected for in the data.
## Backwards Compatibility
-The data model by which gas allowance and usage is represented in external
-interfaces (gas limit, gas used, etc.) remains unchanged. However, the same
-transaction made before and after this FIP will, with very high likelihood,
-result in different values for *gas used*. Transactions sitting in the mempool
-at the activation epoch of this FIP will need to be re-estimated and replaced.
+External-facing gas representation (gas limit, gas used, etc.) remains
+unchanged.
-## Test Cases
+However, the gas usage of most messages will increase upwards, so systems using
+static gas limits should be adapted accordingly.
+
+Clients sending messages near the activation epoch of this FIP should estimate
+gas under the new pricing model to guarantee inclusion.
-Testing for this FIP involves two efforts:
+- If the message is included pre-activation, the message will result in a higher
+ overestimation burn.
+- If the message is included post-activation, the message will be executed
+ normally.
+
+## Test Cases
-1. An updated corpus of FVM-compatible [test
- vectors](https://github.com/filecoin-project/fvm-test-vectors) with the new
- gas model.
-2. The testnet plan laid out in FIP-0031.
+Refer to test plan laid out in [FIP-0031].
## Security Considerations
-TODO draft.
+Metering and pricing execution enables more precise gas accounting, which
+translates into higher network security.
+
+Note that this FIP alone is insufficient to secure the network for
+user-programmed actors; it is merely a stepping stone. A future FIP will
+introduce further the final gas accounting model changes before enabling user
+programmability.
## Incentive Considerations
-With the introduction of execution gas to account for real execution cost, actor
-compute logic will no longer be exempted from gas charges, thus resulting in a
-net increase in gas expenditure in the network.
+The gas utilization of all messages will increase, except for value transfers
+which are unaffected.
+
+In order to quantify the magnitude of change, we have backtested this model
+against 2.5MM mainnet messages.
+
+- Annex A includes a detailed breakdown of net utilization increase per message
+ type (relative to original gas).
+- Annex B includes the final total gas per message type.
+
+On average, most message types increase their utilization between 1.5-3x. The
+gas performance of specific messages types is subject to various factors. The
+reader is advised to study Annex A to understand the variability involved.
+
+In our backtest, the four most important messages for storage provider
+operations were impacted in this manner:
+
+- SubmitWindowPoSt (miner actor): gas usage increases an average of 3x, with a
+ median of 2.79x, and a 90% percentile score of 4.47x. Some executions spiked
+ as much as 15.24x. These are likely window PoSt submissions that recovered
+ power, as the resulting work has a significant computational cost that was not
+ previously accounted for.
+- PreCommitSector (miner actor): gas usage increases an average of 3.12x, with a
+ median of 2.49x, and a 90% percentile score of 5.09x. Some executions spiked
+ as much as 9.69x.
+- ProveCommitSector (miner actor): gas usage increases an average of 1.25x, with
+ an identical median, and a 90% percentile score of 1.31x. Some executions
+ spiked as much as 3.02x.
+- PublishStorageDeals (market actor): gas usage increases an average of 1.80x,
+ with a median of 1.84x, and a 90% percentile score of 2.03x. Some executions
+ spiked as much as 2.17x.
+
+These two message types deserve special attention.
+
+- DeclareFaultsRecovered (miner actor): highest variability between executions
+ and highest multiplier; on average, gas cost increases 8.67x, with some
+ instances spiking as much as 33.3x in the backtest. All executions were within
+ the block gas limit.
+- ExtendSectorExpiration (miner actor): infrequent; was already an expensive
+ message. Some executions lead to gas usage beyoned the block limit, rendering
+ them inviable (up to 2.3e10 gas). However, this is a batched message, so
+ clients should size batches within gas limits.
+
+On average, we observe a global increase of 1.95x in gas utilization, if the
+current chain activity were to remain unchanged. However, the HyperDrive upgrade
+introduced power onboarding batching mechanisms in [FIP-0008] and [FIP-0013]
+that have stayed underutilized. We expect these to kick in an adaptive response
+to lower usage and increase chain bandwidth again.
+
+At last, we expect upwards pressure on the base fee, due to two factors:
+
+- higher chain bandwidth utilization.
+- message executions more frequently surpassing the base fee target (50% of
+ block gas limit, i.e. 5B).
## Product Considerations
-The cost every actor method changes upwards as a result of this FIP.
-Furthermore, by charging for actor logic itself, methods that are more
-computationally intensive may have their price structure changed significantly.
-These two implications may transpire as changes in onboarding, power maintenance,
-deal-making, and other features that are noteworthy from a product perspective.
-
-`<<TODO: More data will be added to this section once benchmarking has been
-completed>>.`
+1. Messages get more expensive. See Incentive Considerations section for a
+ synthesis, and Annex A and Annex B for raw data.
+2. Potential for higher community motivation to optimize built-in actors logic,
+ as doing so directly translates into cost reductions.
+3. Messages estimated pre-FVM will likely not be executed post-FVM. See
+ Backwards Compatibility section for ways to mitigate.
+4. Expect higher usage of aggregated forms of PreCommitSector and
+ ProveCommitSector, as they become more cost-effective under the new gas
+ scenario.
+5. Clients may need to re-evaluate thresholds for batched operations, so as to
+ avoid generating inviable messages due to exceeding the block gas limit.
## Implementation
@@ -313,21 +535,95 @@ For clients relying on the reference FVM implementation
implementation of this FIP will be transparent, as it is self-contained inside
the FVM.
+## Assets
+
+Raw data assets, as well as synthesised backtest results, can be found at
+[raulk/fil-fip-0032-data/](https://github.com/raulk/fil-fip-0032-data/).
+
+## Annex A: Table of gas utilization increase per message type
+
+| | | gas\_utilization\_increase | | | | | | | | | | | |
+| ---------------------- | ------ | -------------------------- | ----------- | -------------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
+| | | count | mean | std | min | 50% | 60% | 70% | 80% | 90% | 95% | 99% | max |
+| code | method | | | | | | | | | | | | |
+| fil/7/account | 0 | 45871 | 1.00597202 | 0.02439399865 | 1 | 1 | 1 | 1 | 1 | 1 | 1.105295017 | 1.105620324 | 1.116646265 |
+| fil/7/init | 2 | 6 | 1.492804871 | 0.004038268814 | 1.48830639 | 1.491762603 | 1.493232453 | 1.494468613 | 1.495704774 | 1.497400912 | 1.498248981 | 1.498927436 | 1.49909705 |
+| fil/7/multisig | 0 | 111 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
+| fil/7/multisig | 2 | 1837 | 1.525822767 | 0.1538518354 | 1.475017101 | 1.49094471 | 1.490971227 | 1.491292227 | 1.492171101 | 1.503233732 | 1.628553559 | 2.395083768 | 2.559610339 |
+| fil/7/multisig | 3 | 467 | 2.092939348 | 0.5178361861 | 1.419330832 | 2.004487358 | 2.04664811 | 2.402342227 | 2.681836119 | 2.943685447 | 3.024047067 | 3.13563372 | 3.282831103 |
+| fil/7/multisig | 4 | 1 | 1.568009198 | | 1.568009198 | 1.568009198 | 1.568009198 | 1.568009198 | 1.568009198 | 1.568009198 | 1.568009198 | 1.568009198 | 1.568009198 |
+| fil/7/paymentchannel | 0 | 1 | 1 | | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
+| fil/7/storagemarket | 2 | 2586 | 1.767349037 | 0.0179153187 | 1.705184746 | 1.764432273 | 1.771243992 | 1.777229832 | 1.781349799 | 1.795999314 | 1.79848267 | 1.799319127 | 1.80223568 |
+| fil/7/storagemarket | 3 | 1 | 1.947764405 | | 1.947764405 | 1.947764405 | 1.947764405 | 1.947764405 | 1.947764405 | 1.947764405 | 1.947764405 | 1.947764405 | 1.947764405 |
+| fil/7/storagemarket | 4 | 5265 | 1.800550738 | 0.1892337028 | 1.421170992 | 1.836807517 | 1.846956112 | 1.8859387 | 1.965049598 | 2.032417936 | 2.058474275 | 2.098231267 | 2.166245941 |
+| fil/7/storageminer | 0 | 22 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
+| fil/7/storageminer | 3 | 33 | 1.581185587 | 0.01756393579 | 1.55043385 | 1.576662766 | 1.587495788 | 1.588772509 | 1.594206699 | 1.609600058 | 1.60997889 | 1.615531646 | 1.618144708 |
+| fil/7/storageminer | 4 | 56 | 1.437003934 | 0.01053285152 | 1.412468752 | 1.440333482 | 1.440578616 | 1.443058913 | 1.445157793 | 1.448293134 | 1.448357005 | 1.457685961 | 1.458410689 |
+| fil/7/storageminer | 5 | 217040 | 3.062597507 | 1.284676077 | 1.084260869 | 2.794653576 | 3.170948509 | 3.551019009 | 3.954638061 | 4.47275546 | 5.219714821 | 7.854538141 | 15.24397403 |
+| fil/7/storageminer | 6 | 1185718 | 3.122825812 | 1.479978666 | 1 | 2.489184964 | 2.66072048 | 3.119869193 | 4.095753137 | 5.087751869 | 6.484390167 | 8.728160893 | 9.688604136 |
+| fil/7/storageminer | 7 | 1053752 | 1.254059457 | 0.04338576394 | 1 | 1.251206444 | 1.262072669 | 1.274505314 | 1.289751641 | 1.310614092 | 1.326975995 | 1.363865829 | 3.020495782 |
+| fil/7/storageminer | 8 | 30 | 2.561482948 | 0.8795361903 | 1.659726975 | 2.193520195 | 2.353174497 | 2.878519824 | 3.252169093 | 4.1322215 | 4.158195316 | 4.196685589 | 4.208294346 |
+| fil/7/storageminer | 9 | 104 | 2.773999785 | 0.7078355684 | 1.924300767 | 2.470266911 | 2.830244664 | 3.240411111 | 3.318940856 | 3.485240501 | 3.720053128 | 5.585537848 | 5.639734092 |
+| fil/7/storageminer | 11 | 1615 | 8.670368309 | 8.066618708 | 1.711216051 | 3.91247307 | 4.886204415 | 10.96614305 | 17.87576882 | 23.55604103 | 24.2148853 | 27.84382546 | 33.30957868 |
+| fil/7/storageminer | 16 | 1310 | 2.162632701 | 0.05734323166 | 1.582717534 | 2.161534147 | 2.165197244 | 2.169072728 | 2.170384968 | 2.171488192 | 2.173423405 | 2.355466711 | 2.394978809 |
+| fil/7/storageminer | 18 | 18 | 1.445781244 | 0.01207120396 | 1.419464401 | 1.447819167 | 1.454712965 | 1.454744278 | 1.454813087 | 1.454813087 | 1.456660407 | 1.465034926 | 1.467128555 |
+| fil/7/storageminer | 22 | 2 | 1.578751057 | 0.000634365616 | 1.578302493 | 1.578751057 | 1.57884077 | 1.578930483 | 1.579020196 | 1.579109909 | 1.579154765 | 1.57919065 | 1.579199622 |
+| fil/7/storageminer | 23 | 37 | 1.44732633 | 0.006346257808 | 1.429727021 | 1.44896509 | 1.449177713 | 1.449769555 | 1.451859339 | 1.455148039 | 1.455649445 | 1.456810884 | 1.457432851 |
+| fil/7/storageminer | 25 | 7482 | 2.846175784 | 0.8397517462 | 1 | 2.509378787 | 2.641833976 | 2.899623107 | 3.117726086 | 3.476581572 | 4.980988959 | 6.008160965 | 8.725638329 |
+| fil/7/storageminer | 26 | 5810 | 1.914048048 | 0.428505585 | 1 | 1.909877692 | 2.135478681 | 2.278302186 | 2.332170825 | 2.468622524 | 2.556291786 | 2.719278562 | 3.132584833 |
+| fil/7/storageminer | 27 | 81 | 1.736822205 | 0.1415775839 | 1.65961248 | 1.698638407 | 1.702767421 | 1.706075911 | 1.719693881 | 1.825980876 | 1.926526938 | 2.186172291 | 2.823086016 |
+| fil/7/storagepower | 2 | 34 | 1.479426147 | 0.007010768301 | 1.466525427 | 1.479850457 | 1.481154027 | 1.482644207 | 1.485159657 | 1.487302302 | 1.490627682 | 1.495758666 | 1.496217996 |
+| fil/7/verifiedregistry | 4 | 11 | 1.804133732 | 0.03922232077 | 1.692439068 | 1.813042981 | 1.825646137 | 1.825646137 | 1.82587245 | 1.82587245 | 1.828249976 | 1.830151998 | 1.830627503 |
+
+## Annex B: Table of total final gas used per message type
+
+| | | final\_total\_gas | | | | | | | | | | | |
+| ---------------------- | ------ | ----------------- | ----------- | ----------- | ---------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
+| | | count | mean | std | min | 50% | 60% | 70% | 80% | 90% | 95% | 99% | max |
+| code | method | | | | | | | | | | | | |
+| fil/7/account | 0 | 45871 | 579530.8664 | 429997.1536 | 278696 | 491868 | 491868 | 491868 | 493168 | 495768 | 2326134.5 | 2332634.5 | 2415834.5 |
+| fil/7/init | 2 | 6 | 18638578.58 | 729881.099 | 17420380 | 18954972.25 | 19060170.5 | 19129952.75 | 19199735 | 19208524.75 | 19212919.63 | 19216435.53 | 19217314.5 |
+| fil/7/multisig | 0 | 111 | 443475.2072 | 23048.51686 | 415168 | 441168 | 441168 | 442468 | 443768 | 493168 | 493168 | 493168 | 494468 |
+| fil/7/multisig | 2 | 1837 | 5837582.178 | 2411162.29 | 3558017.5 | 5629077 | 5629866 | 5634561 | 5638611.5 | 5643135.5 | 5682706.9 | 18325446.76 | 30094244 |
+| fil/7/multisig | 3 | 467 | 13862511.4 | 12066758.5 | 991698.5 | 8122883 | 12734041 | 14655463.8 | 25220149 | 35496028 | 41887381.8 | 46613173.3 | 50468854.5 |
+| fil/7/multisig | 4 | 1 | 1997154.5 | | 1997154.5 | 1997154.5 | 1997154.5 | 1997154.5 | 1997154.5 | 1997154.5 | 1997154.5 | 1997154.5 | 1997154.5 |
+| fil/7/paymentchannel | 0 | 1 | 486668 | | 486668 | 486668 | 486668 | 486668 | 486668 | 486668 | 486668 | 486668 | 486668 |
+| fil/7/storagemarket | 2 | 2586 | 19653863.1 | 479936.3183 | 18433677 | 19536613.5 | 19608916 | 19958224 | 20155859.5 | 20396658.5 | 20397347 | 20653887 | 21137437.5 |
+| fil/7/storagemarket | 3 | 1 | 22612491 | | 22612491 | 22612491 | 22612491 | 22612491 | 22612491 | 22612491 | 22612491 | 22612491 | 22612491 |
+| fil/7/storagemarket | 4 | 5265 | 431978034.6 | 476844103.7 | 65922953 | 288570649 | 339742341.4 | 404653981.3 | 843094881.7 | 888783311.4 | 929103138.6 | 2511917888 | 2630317760 |
+| fil/7/storageminer | 0 | 22 | 440636.1818 | 33021.16762 | 417768 | 419718 | 420368 | 454038 | 488228 | 493168 | 495638 | 495768 | 495768 |
+| fil/7/storageminer | 3 | 33 | 3041236.333 | 156295.4373 | 2911417.5 | 2984946 | 3001525.2 | 3021368.7 | 3051762.3 | 3367924.5 | 3369224.5 | 3442895.7 | 3477564.5 |
+| fil/7/storageminer | 4 | 56 | 2523479.821 | 48041.34968 | 2410819.5 | 2528831.5 | 2534763.5 | 2556110.5 | 2559944.5 | 2580522.5 | 2580616.5 | 2612800.1 | 2621395.5 |
+| fil/7/storageminer | 5 | 217040 | 50567157.53 | 129615886.8 | 1277248 | 29928389.5 | 38593441 | 49359013.65 | 64280830.6 | 87800703.9 | 112949545.8 | 278422165.1 | 7276968168 |
+| fil/7/storageminer | 6 | 1185718 | 57884701.53 | 64060736.44 | 335919 | 33593786.5 | 37303189 | 46857982.6 | 72455034.6 | 111511861.5 | 179449550.5 | 364239625 | 476861537.5 |
+| fil/7/storageminer | 7 | 1053752 | 65322144.86 | 10709175.02 | 2759119 | 62072363 | 64138457.6 | 66994027.95 | 71603156.3 | 80459562.8 | 89151232.75 | 102002577.8 | 124322877.5 |
+| fil/7/storageminer | 8 | 30 | 6142153792 | 8813636083 | 31233192 | 57607162.25 | 117330392.8 | 15159834743 | 16307686048 | 19076238418 | 21880538294 | 22785383969 | 23059819584 |
+| fil/7/storageminer | 9 | 104 | 146564002.1 | 186092472.6 | 27323860.5 | 82113225.25 | 115345332.9 | 146638929.4 | 186883609.9 | 294737331.4 | 336147597.8 | 1108397934 | 1497431427 |
+| fil/7/storageminer | 11 | 1615 | 223766675.3 | 417849908 | 10456724.5 | 49380451.5 | 75358129.9 | 157136408.2 | 407604727.1 | 536195865.4 | 909680332.4 | 2094319347 | 4050112282 |
+| fil/7/storageminer | 16 | 1310 | 20528854.52 | 2019509.742 | 2314035 | 20569943.25 | 20687486.7 | 20908518.25 | 21065727.2 | 21164879.5 | 21244818.55 | 25779334.27 | 26217311 |
+| fil/7/storageminer | 18 | 18 | 2488603.722 | 57093.71661 | 2374243 | 2502224.5 | 2531594 | 2533234.7 | 2533605 | 2533605 | 2540255.25 | 2570403.05 | 2577940 |
+| fil/7/storageminer | 22 | 2 | 1942800 | 780.6458864 | 1942248 | 1942800 | 1942910.4 | 1943020.8 | 1943131.2 | 1943241.6 | 1943296.8 | 1943340.96 | 1943352 |
+| fil/7/storageminer | 23 | 37 | 2466446.095 | 39663.66082 | 2364863.5 | 2457840.5 | 2471928.1 | 2478669.7 | 2495597.9 | 2528554.9 | 2534383.5 | 2535670.54 | 2536394.5 |
+| fil/7/storageminer | 25 | 7482 | 217542474.4 | 259599842.7 | 341119 | 137725560.5 | 183224106.5 | 230833197.3 | 282139260.3 | 566387010.5 | 868583081.1 | 1192473146 | 1276876882 |
+| fil/7/storageminer | 26 | 5810 | 842389295.1 | 707401414.1 | 30303519 | 525095349.8 | 790506876.3 | 1021712113 | 1521512231 | 1685691298 | 2230385687 | 3368285319 | 4734149631 |
+| fil/7/storageminer | 27 | 81 | 137359349.3 | 18291408.57 | 124208174 | 133304884.5 | 134924673 | 136404981.5 | 139511612 | 150077218.5 | 158469749.5 | 208944161.6 | 263801274 |
+| fil/7/storagepower | 2 | 34 | 41528419.24 | 605565.076 | 40013457 | 41455588.5 | 41645275.8 | 41789855.7 | 42007558.7 | 42477707.6 | 42543546 | 42586868.5 | 42586868.5 |
+| fil/7/verifiedregistry | 4 | 11 | 13307839.5 | 787667.1698 | 12461938 | 12928372.5 | 12929241 | 14163140 | 14163140 | 14190283 | 14383126 | 14537400.4 | 14575969 |
+
+## Acknowledgements
+
+Credits to:
+
+- [@magik6k](https://github.com/magik6k) for inputs throughout the design process.
+- [@arajasek](https://github.com/arajasek) for the initial implementation of FIP-0032 in ref-fvm.
+
## Copyright
Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
-## Footnotes
-
-[^1]: The baseline target itself determined by the need to sustain a 30 second
- epoch frequency, with a 6 second block propagation time, under these
- circumstances, which represent the absolute worst possible scenario:
- - Block gas limit of 15B gas units.
- - Attaining the maximum winrate at an epoch (11).
- - No duplicated messages across blocks in a tipset.
- - 100% saturation of every block with compute.
-
- The choice of 10 gas units/ns as a baseline implies a block validation time
- of 1.5s and a fully saturated tipset validation time of 16.5s (given that
- message and block execution cannot be parallelized).
+[wasm-instructions]: https://webassembly.github.io/spec/core/syntax/instructions.html
+[FIP-0008]: https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0008.md
+[FIP-0013]: https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0013.md
+[FIP-0030]: https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0030.md
+[FIP-0031]: https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0031.md
\ No newline at end of file