fix(UTXO): calc of txfee with change, min relay fee #2316

dimxy · 2025-01-09T14:50:12Z

Fixes for utxo tx fee calculation in UtxoTxBuilder::build() fn:

include change output in txfee
fix min relay tx fee calc (to multiply by tx_size).

UtxoTxBuilder::build() code was refactored to implement fixed txfee calc: made it similar to daemon code (with a loop), also broke it into functions.
Existing tests updated for the fixed txfee.

Fixes issue: #2313 (a test added to validate txfee from the issue).
Should also fix #1567 issue.

TODO: fix input size for segwit

@cipig

fix use gas_fee in build(); fix min_relay_tx_fee calc; refactor break build() into functions rename fn get_tx_fee to get_fee_per_kb; fix utxos tests for recalculated tx fee

* dev: feat(tendermint): validators RPC (#2310) chore(CI): validate Cargo lock file (#2309) test(P2P): add test for peer time sync validation (#2304)

fix refactored UtxoTxBuilder::build(): return only txfee (w/o gas fee) with tx as it used to be

mariocynicys · 2025-01-09T17:09:37Z

Is this ready for review?

dimxy · 2025-01-09T17:18:38Z

Is this ready for review?

I am doing final checks and will change the status for ready after that

borngraced · 2025-01-10T01:44:57Z

Thank you for this PR. Covers most of what I was working on here #2083.

I will close mine when this is approved.

dimxy · 2025-01-10T09:13:41Z

Thank you for this PR. Covers most of what I was working on here #2083.

I will close mine when this is approved.

I thought #2083 was about fee priority (although this PR and #2083 definitely have some interception in txfee calc fixes)

onur-ozkan · 2025-01-22T07:53:03Z

mm2src/coins/utxo/utxo_common.rs

            let total_fee = if tx.outputs.len() == outputs_count {
                // take into account the change output
-                data.fee_amount + (dynamic_fee * P2PKH_OUTPUT_LEN) / KILO_BYTE
+                data.fee_amount + actual_tx_fee.get_tx_fee_for_change(None)


Suggested change

data.fee_amount + actual_tx_fee.get_tx_fee_for_change(None)

data.fee_amount + actual_tx_fee.get_tx_fee_for_change(0)

onur-ozkan · 2025-01-22T07:53:31Z

mm2src/coins/utxo/utxo_common.rs

-                    data.fee_amount
-                }
+                // take into account the change output
+                data.fee_amount + fee_per_kb.get_tx_fee_for_change(Some(tx_bytes.len() as u64))


Suggested change

data.fee_amount + fee_per_kb.get_tx_fee_for_change(Some(tx_bytes.len() as u64))

data.fee_amount + fee_per_kb.get_tx_fee_for_change(tx_bytes.len() as u64)

onur-ozkan · 2025-01-22T07:54:15Z

mm2src/coins/utxo.rs

+    }
+
+    /// Return extra tx fee for the change output as p2pkh
+    fn get_tx_fee_for_change(&self, tx_size: Option<u64>) -> u64 {


I see that tx_size can be used as u64 (see the other suggestions)

removed Option from get_tx_fee_for_change in 3f709fc

onur-ozkan · 2025-01-22T08:15:47Z

mm2src/coins/utxo/utxo_common.rs

+        for utxo in self.required_inputs.clone() {
+            self.tx.inputs.push(UnsignedTransactionInput {
+                previous_output: utxo.outpoint,
+                prev_script: utxo.script,
+                sequence: SEQUENCE_FINAL,
+                amount: utxo.value,
+            });
+            total += utxo.value;
+        }
+
+        for utxo in self.available_inputs.clone() {
+            if total >= amount {
+                break;
+            }
+            self.tx.inputs.push(UnsignedTransactionInput {
+                previous_output: utxo.outpoint,
+                prev_script: utxo.script,
+                sequence: SEQUENCE_FINAL,
+                amount: utxo.value,
+            });
+            total += utxo.value;
+        }


This should be slightly cheaper:

Suggested change

for utxo in self.required_inputs.clone() {

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script,

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

for utxo in self.available_inputs.clone() {

if total >= amount {

break;

}

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script,

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

for utxo in &self.required_inputs {

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script.clone(),

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

for utxo in &self.available_inputs {

if total >= amount {

break;

}

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script.clone(),

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

done in 50a38f8

onur-ozkan · 2025-01-22T08:18:58Z

mm2src/coins/utxo/utxo_common.rs

+            if total >= amount {
+                break;
+            }


We should be able to add this check above the loop, so we don't start iterating it for no reason if it's already true.

fixed in 50a38f8

mariocynicys

Thanks! 1st iteration.
went over all the changes but skipped most of my comments on the tx builder stuff since i couldn't fully comprehend it and don't wanna cause confusion.
will need a couple more iters as this part of the code is a lil risky and confusing.

mariocynicys · 2025-01-22T11:12:14Z

mm2src/coins/utxo.rs

@@ -279,11 +279,43 @@ pub enum TxFee {
 pub enum ActualTxFee {
    /// fee amount per Kbyte received from coin RPC
    Dynamic(u64),
-    /// Use specified amount per each 1 kb of transaction and also per each output less than amount.
+    /// Use specified fee amount per each 1 kb of transaction and also per each output less than the fee amount.


could you explain more about this fee scheme. the per each output part isn't really clear (understood it as 1 fee_rate per 1 output, which i doubt is a correct understanding).

Hmm, actually I have the same question:
this fee calc is not fully clear to me. The comment refers to DOGE, I took a look at its daemon code but did not find where this logic is.

mariocynicys · 2025-01-22T11:14:32Z

mm2src/coins/utxo.rs

+impl ActualTxFee {
+    fn get_tx_fee(&self, tx_size: u64) -> u64 {
+        match self {
+            ActualTxFee::Dynamic(fee_per_kb) => (fee_per_kb * tx_size) / KILO_BYTE,


couldn't this wrongfully floor to zero?

I guess it could.
If it goes too low we would use min relay fee (if this is set though)

mm2src/coins/utxo.rs

mm2src/coins/utxo/utxo_common.rs

mariocynicys · 2025-01-22T16:24:26Z

mm2src/coins/utxo/utxo_common.rs

+            if self.sum_inputs >= self.sum_outputs + self.total_tx_fee() {
+                break;
+            }


shouldn't this be exactly equal (or equal + unused change)?

I believe, not.
At this step tx_fee has been updated by the tx_size * fee_rate formula (not like difference of inputs - outputs). So if we have added sufficient inputs, we are finished

and the change will have been adjusted to inputs - outputs - fees, which brings the equation above to equality.

I think this is not necessarily so.
Within an iteration, we are getting next input set and change for the total outputs + previous-txfee.
But due to the input selection algo, we may get this input set with the length less than it was at the previous step (rare case, ofc). So the updated txfee may be less than the previous-txfee (but we want to quit now as we have added enough inputs to cover the txfee. As I can see, kmd works like that: it quits the loop when nFeeRet >= nFeeNeeded)

So the updated txfee may be less than the previous-txfee (but we want to quit now as we have added enough inputs to cover the txfee. As I can see, kmd works like that: it quits the loop when nFeeRet >= nFeeNeeded)

I am not following here. Do you mean that the new tx fee is less than the previous iteration but we willingly and intentionally don't update/reduce it? why?

or we can make this argument simpler: if sum_inputs > sum_outputs + fees + unused_change where do these wasted sats (extra fees?) go? and what is the higher bound on the amount of these wasted sats?

sets from previous loop iteration and we hope that the newly created inputs and outputs would be sufficient to cover the updated txfee

but this can:

underpay fees -> we do another iteration.

pay exact fees -> perfect, we should also have the equality holding in this case.

overpay fees -> why overcharge? we should should subtract the overpaid fees and reward them to our change output (or the output paying the fees in general). (ref. https://github.com/bitcoin/bitcoin/blob/15717f0ef3960969ee550a4a41741987b86684dc/src/wallet/spend.cpp#L1268-L1276)

I am OK with this as long as we can have a check bounding the paid_fee - needed_fee to some maximum. Even though I think it's better to have a deterministic algo with a stricter/more-expected output.

lmk please if this statement is correct:
the algo here in it's core should terminate when the fees (after update at the end of the loop) at iteration x is the same as fees at iteration x+1. that is, the inputs will stay the same, the outputs will stay the same, whether there is a change or not will stay the same, but the fees changed a little bit to account for the true transaction size (that's the fee update at iteration x. iteration x+1 didn't need a fee update). in such case, we should have calculated the fees correctly already without overpaying?

we should should subtract the overpaid fees and reward them to our change output (or the output paying the fees in general). (ref.

bitcoin has additional fee optimisations indeed. I guess, this correction of the change in BTC (you refer to) works when the updated txfee becomes lower (due to the smaller reselected input set).
In our code we we need more inputs we add them in sequence (deterministically) so I think we won't have this case.
I suggest adding this as a TODO for now as well as implementation of an input selection algo like 'Branch and Bound'.

lmk please if this statement is correct: ... at iteration x is the same as fees at iteration x+1.

Almost. We break if next fee equals or less than the previous fee. It could be less if the change turns out to be dust (so the tx size becomes lower) so some overpay may appear but I guess this is okay.

we can have a check bounding the paid_fee - needed_fee to some maximum

We may add check for 'absurd fee' but this is not to prevent overpay

I think it's better to have a deterministic algo with a stricter/more-expected output.

Is this algo not deterministic enough? I believe most cli wallets use it and it fixes the existing bug.
It may produce slight overpay but I believe this is acceptable (and unavoidable)

I added a proptest https://github.com/dimxy/komodo-defi-framework/blob/69789daed7f204abdbea79bbb906853d54110f3f/mm2src/coins/utxo/utxo_tests.rs#L1228 with randomised inputs outputs. It catches when overpay occurs due to the change not added.

bitcoin has additional fee optimisations indeed. I guess, this correction of the change in BTC (you refer to) works when the updated txfee becomes lower (due to the smaller reselected input set).

ah maybe. im not super sure tho since the thing isn't in a loop, so maybe not.

It could be less if the change turns out to be dust (so the tx size becomes lower) so some overpay may appear but I guess this is okay.

yup. that much overpay is perfectly justified for the dust case. and this is why i am saying there is a bound on how much we spend and such a bound is dust (actually a tiny bit greater due to how the change output mechanism bumps the fee).

Is this algo not deterministic enough? I believe most cli wallets use it and it fixes the existing bug.

the nondeterministic-ness here comes from the fact that we rely on the past state of the loop (tx fees computation) to solve-for/construct the tx in the current loop. it's not like a simple "these are the problem inputs and they give u this output/tx", but rather u need to reason about what was the last loop the tx solver gone through to know what the output tx would be.
tbh though, that's a play over a very small margin, it's not much of a big deal now since we don't have some sophisticated input selection algo. so not like one loop we will have a lot of small utxo inputs and next we have one large single input utxo (this case overpays in fees a lot).

i think i now have much confidence in this tx constructor algo :). will try to calculate the bound on the max overpay in fees though and discuss if we could add a check for it as that would be safer.

I think overpay may consist of dust and the fee for the removed change output.

(I improved the utxo builder test 9126c49: made inputs/outputs randomised and catch and print the overpay)

mm2src/coins/utxo/utxo_common.rs

borngraced

Really great work! Some minor notes from my side

mm2src/coins/utxo/utxo_common.rs

refactor (borngraced): add_change simplified, remove tx_fee mut, use checked_sub

…true

onur-ozkan

Thanks for the previous fixes.

It LGTM.

smk762 · 2025-02-10T07:51:03Z

Does this PR require and docs updates? I've created a docs PR at KomodoPlatform/komodo-docs-mdx#416 for the related KF PR #2083 please advise in the docs PR if anything else is needed.

laruh · 2025-02-21T13:28:42Z

@dimxy pr started to have conflicts

mm2src/coins/utxo/utxo_common.rs

… update txfee

dimxy · 2025-02-24T18:07:33Z

Does this PR require and docs updates? I've created a docs PR at KomodoPlatform/komodo-docs-mdx#416 for the related KF PR #2083 please advise in the docs PR if anything else is needed.

I don't thing so.
It's a bug fix for tx fee calculation. No changes in any rpcs introduced.

* dev: (24 commits) fix(eth-tpu): remove state from funding validation (#2334) improvement(rpc-server): rpc server dynamic port allocation (#2342) fix(tests): fix or ignore unstable tests (#2365) fix(fs): make `filter_files_by_extension` return only files (#2364) fix(derive_key_from_path): check length of current_key_material (#2356) chore(release): bump mm2 version to 2.4.0-beta (#2346) fix(tests): add additional testnet sepolia nodes to test code (#2358) fix(swaps): maintain legacy compatibility for negotiation messages (#2353) refactor(SwapOps): impl defaults for protocol specific swapops fns (#2354) feat(tpu-v2): provide swap protocol versioning (#2324) feat(wallet): add change mnemonic password rpc (#2317) fix(tpu-v2): fix tpu-v2 wait for payment spend and extract secret (#2261) feat(tendermint): unstaking/undelegation (#2330) fix(utxo-withdraw): get hw ctx only when `PrivKeyPolicy` is trezor (#2333) feat(event-streaming): API-driven subscription management (#2172) fix(hash-types): remove panic, enforce fixed-size arrays (#2279) fix(ARRR): store unconfirmed change output (#2276) feat(tendermint): staking/delegation (#2322) chore(deps): `timed-map` migration (#2247) fix(mem-leak): `running_swap` never shrinks (#2301) ...

…sable txfee helpers

mariocynicys

reviewed the latest commits. looks good. i like the new interest calculations. that made it much easier to understand what's going on.

single nit/optimization for now.
will do one last pass after discussions in the open threads are done.

mariocynicys · 2025-03-04T20:08:22Z

mm2src/coins/utxo/utxo_common.rs

+    unsigned: &mut TransactionInputSigner,
+) -> UtxoRpcResult<u64> {
+    if !coin.is_kmd() {
+        return Ok(0);
    }
    unsigned.lock_time = coin.get_current_mtp().await?;
    let mut interest = 0;


we should replace get_verbose_transaction with get_verbose_transactions_from_cache_or_rpc since we run this inside the tx solver loop.

sorry can't annotate the specific line (L#814 = 6 lines below) cuz it's not changed.

mariocynicys

very neat test. thanks.
ran it many times and never failed :)

mariocynicys · 2025-03-06T11:51:15Z

mm2src/coins/utxo/utxo_tests.rs

+        const CHANGE_OUTPUT_SIZE: u64 = 1 + 25 + 8;
+        let max_overpay = dust + fee_rate * CHANGE_OUTPUT_SIZE / 1000; // could be slight overpay due to dust change removed from tx


thanks!
could we include this bound in the tx builder code as well.

dimxy added 4 commits January 8, 2025 23:52

fix UtxoTxBuilder::build() tx fee calculation to include change output;

cea1b2f

fix use gas_fee in build(); fix min_relay_tx_fee calc; refactor break build() into functions rename fn get_tx_fee to get_fee_per_kb; fix utxos tests for recalculated tx fee

Merge branch 'dev' into fix-utxo-tx-fee-for-change

455403c

* dev: feat(tendermint): validators RPC (#2310) chore(CI): validate Cargo lock file (#2309) test(P2P): add test for peer time sync validation (#2304)

fix qrc20 gas fee calc in trade preimage

8d2fe79

revert qtum gas_fee calc in trade preimage,

bc69a42

fix refactored UtxoTxBuilder::build(): return only txfee (w/o gas fee) with tx as it used to be

dimxy added status: in progress priority: medium Moderately important tasks that should be completed but are not urgent. bug: API labels Jan 9, 2025

dimxy added status: pending review and removed status: in progress labels Jan 13, 2025

dimxy requested review from borngraced and mariocynicys January 21, 2025 17:13

onur-ozkan reviewed Jan 22, 2025

View reviewed changes

mariocynicys reviewed Jan 22, 2025

View reviewed changes

borngraced requested changes Jan 23, 2025

View reviewed changes

dimxy added 4 commits January 23, 2025 14:26

fix: err if change < dust (was <=)

40cfbda

refactor (borngraced): add_change simplified, remove tx_fee mut, use checked_sub

refactor (onur-ozkan): use refs instead of clone in add_tx_inputs

50a38f8

refactor (mariocynicys): fix return_err_if macro to check if cond is …

615ff3e

…true

refactor: rename vars to use always 'fee rate' root word

3f709fc

onur-ozkan previously approved these changes Feb 5, 2025

View reviewed changes

shamardy added the 2.5.0-beta label Feb 17, 2025

shamardy requested a review from mariocynicys February 17, 2025 10:07

mariocynicys reviewed Feb 21, 2025

View reviewed changes

mm2src/coins/utxo/utxo_common.rs Outdated Show resolved Hide resolved

mm2src/coins/utxo/utxo_common.rs Outdated Show resolved Hide resolved

fix (mariocynicys): recalc kmd interest when tx inputs reselected and…

459a379

… update txfee

dimxy dismissed onur-ozkan’s stale review via 459a379 February 24, 2025 17:51

dimxy added 5 commits February 24, 2025 23:07

fix after merge

ad0699e

refactor (mariocynicys) removed unnecessary checks

1b2c538

fix fee_amount in AdditionalTxData, get rid of unused_change, add reu…

110a236

…sable txfee helpers

add extra test to check tx builder calc with many inputs

b5994e6

shamardy requested a review from mariocynicys March 3, 2025 11:52

remove unused_change in utxo tx builder (var not used any more)

ae8c3ee

mariocynicys reviewed Mar 4, 2025

View reviewed changes

fix utxo tx builder test: add randomizer, catch overpay

9126c49

mariocynicys approved these changes Mar 6, 2025

View reviewed changes

	data.fee_amount + actual_tx_fee.get_tx_fee_for_change(None)
	data.fee_amount + actual_tx_fee.get_tx_fee_for_change(0)

	data.fee_amount + fee_per_kb.get_tx_fee_for_change(Some(tx_bytes.len() as u64))
	data.fee_amount + fee_per_kb.get_tx_fee_for_change(tx_bytes.len() as u64)

		const CHANGE_OUTPUT_SIZE: u64 = 1 + 25 + 8;
		let max_overpay = dust + fee_rate * CHANGE_OUTPUT_SIZE / 1000; // could be slight overpay due to dust change removed from tx

fix(UTXO): calc of txfee with change, min relay fee #2316

Are you sure you want to change the base?

fix(UTXO): calc of txfee with change, min relay fee #2316

Conversation

dimxy commented Jan 9, 2025 • edited Loading

mariocynicys commented Jan 9, 2025

dimxy commented Jan 9, 2025

borngraced commented Jan 10, 2025

dimxy commented Jan 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariocynicys left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimxy Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

borngraced left a comment

Choose a reason for hiding this comment

onur-ozkan left a comment

Choose a reason for hiding this comment

smk762 commented Feb 10, 2025

laruh commented Feb 21, 2025

dimxy commented Feb 24, 2025

mariocynicys left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariocynicys left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimxy commented Jan 9, 2025 •

edited

Loading

dimxy Mar 5, 2025 •

edited

Loading