When I first started playing with the Flashbots’ infrastructure there was a lot of onion peeling on my part just to figure out why things weren’t working. My MEV bundles were just sitting there like a sack of potatoes and kept getting ignored by the MEV friendly miners.
So I originally started drafting this brain dump for my own benefit as a troubleshooting checklist of sorts, but decided to share it on medium in case it can save others from a few hours of hair pulling.
Overview
The HOW and the WHY of the Flashbots implementation and MEV in general are documented quite extensively by the Flashbots team so I won’t rehash them, other than to suggest reading up on the following two articles as a primer, in order to properly understand this infrastructure, its purpose and how to get hands-on experience with it.
- https://medium.com/flashbots/frontrunning-the-mev-crisis-40629a613752
- https://ethresear.ch/t/flashbots-frontrunning-the-mev-crisis/8251
At a high level, Flashbots runs on a modified version of Geth (MEV-Geth) which is an open-sourced sealed-bid block space auction mechanism for communicating transaction order preference. As illustrated below, MEV-Geth essentially delegates the job of finding MEV opportunities to ‘Searchers’ (i.e. this is where you come in), whether it’d be finding arbitrage, liquidation, whitehat rescues (or frontrunning, backrunning, sandwiching, french toasting…etc).
During the searching process, you’re competing with other Searchers to find the most profitable MEV opportunities, which you then submit in the form of a “Bundle” to miners’ RPC endpoints exposed through MEV-Geth and is bid against other Searchers in the form of miner bribes extracted from the MEV for inclusion in the next block. The participating mining pools then evaluate these Bundles based on their profitability logic and chooses accordingly.
At the time of writing, UUPool and the recently onboarded Spider Pool are the only pools actively mining with MEV-Geth, however there are other pools testing the system before committing. UUPool is mainly active on Weibo (Chinese Twitter) but you’ll need a valid Wechat/Weibo account and basic proficiency in Chinese. I don’t know much about Spider Pool but their website’s here. There are also some… interesting… discussions about UUPool in the Flashbots discord, so I’ll leave it to you to DYOR.
Troubleshooting
So the exciting day comes around when you finally receive your shiny new Flashbots API key and you rush to your machine to clone the Flashbots’ Ethers Provider Bundle repo to start firing off MEV bundles. The repo successfully compiles and the demo script executes with no errors, resulting in the submission of your Bundle. You sit back and wait.
…but it never gets picked up by the miners.
Your frustration starts to build with each passing block and you start to imagine the MEV miners sitting atop their diamond encrusted ivory towers judging you with such disdain. Like Roman senators spectating a gladiatorial contest, you look up at the miners and wonder, “are you not entertained?’.
And this is where the onion peeling begins…
Below are some of the potential causes preventing your MEV bundle from being picked up by miners and what you can do about it.
1. Noncompetitive Gwei Price (Bribe)
Cause: MEV-Geth’s default profitability logic will only mine a flashbots MEV bundle for that block if it will result in higher miner revenue compared to the the least profitable non-MEV transaction(s) at the bottom of a block.
Mitigation: Head to the Flashbots discord’s Bundle-Alerts channel which shows the transaction bundles successfully picked up by MEV miners as well as the associated bribes. Use this as an indicator on how much bribe you’ll need to offer the miners in order for your MEV bundle to get picked up. Simply put, the miner’s bribe should roughly be your chosen bribe price in gwei multiplied by the total gas usage of your transaction.
2. Incorrect Gas Estimates
Cause: Even if you’re using the right bribe price in gwei, miscalculating the total gas usage of your MEV bundle will still result in your bribe being noncompetitive by several orders of magnitude.
Mitigation: The Simulate() function has been activated in the latest version of ethers-provider-flashbots-bundle. Calling simulate() like this will return a json object which will estimate the gas used across all your transactions within the MEV bundle. Ensure the miner bribe is the total sum of all gas used multiplied by your chosen miner bribe price in gwei. i.e. in the 2 tx bundle example below the bribe would be X Gwei * (21000+33283)
3. UUPool/SpiderPool did not mine a block in your specified block range
Cause: Not every block will be mined by a Flashbots miner, so if your bundle is targeting a range of blocks mined by other mining pools then nothing will happen.
Mitigation: At the time of writing since UUPool and Spider Pool are the only pools servicing Flashbots it works out to be about one in every fifteen blocks. So factor that strike rate into your bot’s logic until more mining pools are onboarded.
4. You got pushed out of the way
Cause: Currently only 1 Flashbots bundle will make it into each block mined by UUPool or SpiderPool, so you’re essentially competing for prime real estate with other MEV Searchers, which can make it challenging when money printing Sandwich Bots are making it rain on MEV miners like a Kendrick Lamar music video.
Mitigation: The Flashbots team is working on functionality that would enable multiple Searchers to combine their respective bundles for submission to miner RPC endpoints. So either wait for that, or look for ways to game the system e.g. by populating your bundle with transactions that inflate your implied gasPrice…etc.
5. Your transaction is failing
Cause: Your transaction itself is no good, i.e. reverting or otherwise. If you implemented your logic correctly your bot should only release the miner bribe via block.coinbase.transfer() if the original transaction (e.g. arb, liquidation, frontrun, sandwich) was successful. So by that logic, if your original transaction reverted then no payment is made, which translates into getting ignored by MEV miners.
Mitigation: Always simulate your transactions via mainnet forks first, followed by ethers-provider-flashbots-bundle’s simulate() function before submitting to MEV miners. I recommend using hardhat for mainnet forking as it’s pretty easy to setup.
6. You went overboard
Cause: Your frustrations led you to spam bundle submissions to increase your chances of getting picked up by MEV miners, so now you’re rate-limited by the Flashbots API.
Mitigation: The alpha release limit is 60 bundle submissions per minute so keep your bot’s enthusiasm in check and go beg Tina to undo your rate limit.
7. You‘re gassy AF
[Updated: 3rd Mar 2021] — this is no longer an issue.
8. Issue with the transaction nonce
Cause: One of the nonces inside the transaction is too low, either before or on the target block.
Mitigation: Check your account’s minimum nonce and increment accordingly for the next transaction.
Key considerations
The following aren’t troubleshoot-related but are worth keeping in mind as you develop your MEV strategies.
Only pay when you win
One of the key features of the Flashbots infra is the ability to only pay the miner bribe if your original transaction was successful. Which is music to the ears of beginner frontrunners whose bots have been failing and racking up gas expenses. Therefore to take advantage of this you should code your bots to move away from the traditional transaction fee payment concept and adopt coinbase payments. i.e. submiting 0-gas-price transactions for the initial transaction (arb, liquidation, frontrun…etc) followed by on-chain logic to release the bribe to the miner via block.coinbase.transfer() if the initial transaction was successful.
Transaction Ordering
MEV-Geth merely uses the order of transactions that was sent via ethers-provider-flashbots-bundle’s sendBundle function, but the Searcher who sent the bundle can pick any order they like within this bundle. This creates a relatively risk free environment to build sandwiches and other strategies.
Trusted Intermediary
In the alpha release of the FlashBots you’re trusting FlashBots and UUPool/SpiderPool miners not to frontrun or sandwich you. They can also see the content of the bundles and have the option to potentially censor or steal them i.e. privileged liquidation.
Whitehat Rescues
Flashbots and other private relay networks like Tai Chi will keep your transactions hidden from frontrunners until after it gets mined. For mission critical whitehat rescues that only involve a single transaction however, the Tai Chi network is a better choice at this point in time purely based on the fact that they have more hash power behind it, so your rescue transaction will get mined faster that way. If the rescue op involves firing off multiple transactions then the Flashbots setup may be better suited.
Other MEV Players
If you’re curious on what other MEV groups are doing in this space, look up Rook, ArcherDAO, Tai Chi and StakeDAO.
So that’s all the stuff in my head at the moment but I will keep adding to this over time. If you have additional lessons learnt that can be useful to new Searchers, then please reach out to me on Twitter and I’ll add them here.
Have fun and please MEV responsibly.
