Hello and welcome to issue number nine of WTF News, our biweekly status update.
Over the last two weeks, we ran a handful of wallets through their paces, and today we can reveal the results! Spoiler alert, but there were more issues in our tests than in the best scoring wallets.
First, the regular status update!
What good is a testing framework if it isn't used? Get in touch if you want to integrate WTF into your wallet's testing pipeline. We'll even build the automation ourselves!
We want to expand our tests for WTF, to catch issues with the framework itself. To that end, if you work on a wallet that works offline (aside from the RPC provider) and want to try out WTF, get in touch with us!
BigIntto hex in
jsonrpcin responses to the wallet.
|Wallet||Version||Connection||Passing Tests||Pass %||Platforms||Browser||Notes|
|Brave Wallet||EIP-3085||61||100%||Browser||Brv 1.57.42|
|Taho||0.46.1||EIP-3085||61||100%||Extension||Chr 115.0.5790.170||Had to impersonate chainlist.org|
|Frame||0.6.7||EIP-3085||60||98%||Desktop||Ff 102.14.0esr (64-bit)||Missing signTransaction|
|Enkrypt||1.24.0||Manual||59||97%||Extension||Ff 102.14.0esr (64-bit)||Attempted EIP-3085, but popup doesn’t show RPC url|
|Coinbase||3.30.2||EIP-3085||48||79%||Extension||Chr 115.0.5790.170||Missing hex prefix in logs|
|Metamask||10.34.0||EIP-3085||48||79%||Extension||Ff 102.14.0esr (64-bit)|
|OKX||2.58.1||EIP-3085||18||30%||Extension||Chr 115.0.5790.170||Tests stopped running somewhere around getFilterChanges|
|TokenPocket||1.1.19||EIP-3085||15||25%||Extension||Chr 115.0.5790.170||Tests stopped running somewhere around getFilterChanges|
|Argent X||5.7.2||Can’t Connect||0||0%||Extension||Chr 115.0.5790.170||No window.ethereum|
|Bitkeep||1.4.8||Can’t Connect||0||0%||Extension||Chr 115.0.5790.170||Attempted EIP-3085, and Manual, but got system error|
|Sequence||2.0.4||Can’t Connect||0||0%||Extension||Chr 115.0.5790.170||Cannot add custom chains|
|Exodus||23.8.14||Can’t Connect||0||0%||Extension||Chr 115.0.5790.170||Cannot add custom chains|
The astute reader may notice that there are in fact 62 tests in the repository
at that commit. Our
eth_getProof test is broken, and we are excluding it from
Our intent is not to claim that certain wallets are better than others, and a low score here is as likely to indicate that our tests have problems as it is to indicate bugs in a wallet. We're very interested in improving scores across the board, so we're happy to accept contributions to fix tests and just as happy to help debug.
Very little of this report is scientific/rigorous, and the wallet selection process was no exception. We chose a few wallets we're familiar with, and a small sampling of random wallets from the Find an Ethereum Wallet list.
Because Wallet Test Framework only supports
window.ethereum so far, we're
limited to wallets that provide it; that pretty much excludes mobile wallets for
the time being. Our architecture requires connecting to a custom chain, so
wallets that don't support custom chains cannot be tested (labelled with
Can't Connect in the table.)
For future reports, get in touch if you're interested in having your results published (or if you'd prefer we didn't mention you.)
This is a contentious endpoint that we've discussed previously.
Very few wallets support
eth_signTransaction, mostly by choice.
Interestingly, this was Frame's only failure: instead of signing in the wallet, Frame passes these requests through to the RPC provider.
There are only two hard things in Computer Science: cache invalidation and naming things. Phil Karlton
Caching is hard. We've been bitten by it. In a space where you have to pay for every API request, but your users expect a free wallet experience, caching is a necessary evil.
We suspect, but haven't verified for all wallets, that many of the test failures are caused by stale values cached by wallets.
In the case of
eth_call, we have verified that MetaMask specifies an old block
number, causing a call into a freshly deployed contract to fail. This is
unlikely to cause major issues on a real network with many seconds between
blocks, but the tests mine blocks much more frequently, triggering the issue.
Regardless, we're following up with their team to see what we can do.
Some of the newer endpoints like
eth_maxPriorityFeePerGas aren't yet supported
in all wallets. Expect these tests to start passing as they are implemented.
We haven't debugged why the tests stalled. It could be as simple as adding a timeout to WTF, or something more involved. We'll be looking into this before the next report.
We'll continue adding tests, and we'll try to run through this report somewhat regularly. That said, this is entirely a manual process. Once wallets start running WTF as part of their continuous integration processes, we can make some pretty graphs, include more wallets, and make reports more often.
In the meantime, join us on Discord!