[Feature Request] Potentially the chain could be stalled due to too many listings

[leo-plugdefi]

observation:
This pallet uses an on_initialize hook to close ended listings, in a loop, which is dangerous and costly, as we can have a number of listings, and I think it is already noted in the code comment.

proposal:
move this function out of hook, use an automated bot to send extrinsic to maintain the status of the listings, and always check listing end block in related extrinsic.

[markopoloparadox]

By automated bot I assume you mean an external program that is run by us. This would be even more dangerous, expensive and less reliable.

If we are not already, we should limit the amount of operations that we do per block. This will help us make sure that we don't go over a critical threshold.

[leo-plugdefi]

While we have weight system in place, using on_initialized hook could take up all the weight. Even if we set a limit on the total number of open listings and ensure they can be processed in time, the allocated time for the on_initialize phase would still be limited. Unfortunately, I do not yet have a concrete number for this limit, but it is likely to be quite low. This could lead to an issue where someone could spam the marketplace by sending tons of listings, preventing anyone else from listing anything.

I understand that the external bot is not an ideal solution in terms of security and cost, however it is a pattern that has been used in the crypto space. The maintenance extrinsic can only do simple and restricted tasks, so the worst case scenario would be if the bot fails to call this maintenance extrinsic occasionally, it could be easily picked up and retried, even manually. Alternatively, if you were to lose your private key for the bot, then the maximum you would lose is a small amount of gas fee sent to it. After all, if we continue with the existing on-chain approach, then if something were to go wrong there would be no more blocks finalized and a fork &and restart might not be avoidable - which I believe is far more dangerous and costly.

[markopoloparadox]

According to you link there is nothing wrong about on_initialize and it only suggests to move all logic from on_finalize to on_initialize.

Let me respond to some of your comments:
1) Someone can spam a marketplace or whatever tx
They can only spam it if the tx is free, if not then there is nothing wrong with it. If they want to burn their money on spamming a tx then they are free to do it. This applies to all our txs

2) Not ideal in terms of security and cost
Security and cost are the driving factors in what we do. Crypto space is filled with garbage networks, software, tools or whatever. Companies deploy contracts and bridge relayers just for them to be hacked. So yeah, our top priority should and is security and cost.

3) maintenance extrinsic can only do simple and restricted tasks
Basically what you are suggesting is to have the same code but also deploy a bot that will call that same code instead of it being called automatically. If your concern is that our code might halt the chain, then you have done nothing to mitigate that and instead bring more complexity and problems.

Three more things:

1. Deploying a bot to do essential work is problematic because this means that all our networks need to have that bot running. This means that I cannot anymore deploy a chain without deploying a bot that will do that essential work. So yeah, not gonna happen.
2. Any code that we write in substrate can break our whole chain. Writing error-prone storage mitigations is one quick way to competently break our networks so I think that we are all aware that we need to test our code as much as possible.
3. Last but not least, one of the core Futuverse values are transparency and openness. Having all the essential work inside our substrate code makes sure that there is nothing that we are hiding and anyone who wants to use or participate in our Mainnet can always check our code and be sure that is up the highest standard (once we make our repo open source ofc).

[leo-plugdefi]

Appreciate your prompt reply, very well said.

I just verified that I used a different link than what I intended , but you can understand why the issue of on_initialize is a concern. Everybody want to reduce the gas cost for users, the listing number limit could be very low, so the total cost of spamming the entire area could be much lower than expected. Spamming with many other transactions will not prevent this market from functioning, but reaching the listing limit could. Therefore, from the marketplace product standpoint, I would be grateful if the listing number limit could be avoided.

Basically what you are suggesting is to have the same code but also deploy a bot that will call that same code instead of it being called automatically

this is not the case though, as we have weight limit in 1 block, so we have an limit on how many listings we can loop for in one block, The reason I am in favor of an external bot is that you can adjust the timing, chunk size, pivot etc flexibly, split the big batch into smaller chunks per block, and avoid setting a limit on open listings.

I acknowledge the reasons that you rule out external bot. Then in this case, I think it is still worth considering solving this. Any hint could be helpful. In the mean time, I can spend sometime to find out how many open listings will actually brick the chain.