WTF News (Week 19)

Hello and welcome to issue number nine of WTF News, our biweekly status update.

Over the last two weeks, we ran a handful of wallets through their paces, and today we can reveal the results! Spoiler alert, but there were more issues in our tests than in the best scoring wallets.

First, the regular status update!

Call to Action🔗

CI/Build Pipelines🔗

What good is a testing framework if it isn't used? Get in touch if you want to integrate WTF into your wallet's testing pipeline. We'll even build the automation ourselves!

Testing the Tests🔗

We want to expand our tests for WTF, to catch issues with the framework itself. To that end, if you work on a wallet that works offline (aside from the RPC provider) and want to try out WTF, get in touch with us!

Code Changes🔗

• Upgraded to viem 1.6!

Bug Fixes🔗

• Use smaller integers to make ganache happy.
• Send WebSocket pings to keep the connection alive.
• Use from instead of account in eth_signTransaction.
• Use hex instead of decimal in eth_signTransaction.
• Convert BigInt to hex in eth_getProof.
• Actually put jsonrpc in responses to the wallet.
• Handle race condition when falling back to adding a chain manually.

Upstream Issues🔗

• eth_call unclear when Block parameter is omitted.

Test Report🔗

Summary🔗

The astute reader may notice that there are in fact 62 tests in the repository at that commit. Our eth_getProof test is broken, and we are excluding it from this report.

Disclaimer🔗

Our intent is not to claim that certain wallets are better than others, and a low score here is as likely to indicate that our tests have problems as it is to indicate bugs in a wallet. We're very interested in improving scores across the board, so we're happy to accept contributions to fix tests and just as happy to help debug.

Wallet Selection🔗

Very little of this report is scientific/rigorous, and the wallet selection process was no exception. We chose a few wallets we're familiar with, and a small sampling of random wallets from the Find an Ethereum Wallet list.

Because Wallet Test Framework only supports window.ethereum so far, we're limited to wallets that provide it; that pretty much excludes mobile wallets for the time being. Our architecture requires connecting to a custom chain, so wallets that don't support custom chains cannot be tested (labelled with Can't Connect in the table.)

For future reports, get in touch if you're interested in having your results published (or if you'd prefer we didn't mention you.)

Discussion🔗

Congratulations to Brave and Taho for getting perfect scores! Considering this was our first time trying them out with the Framework (we've been using Frame to develop), this was quite surprising.

eth_signTransaction🔗

This is a contentious endpoint that we've discussed previously. Very few wallets support eth_signTransaction, mostly by choice.

Interestingly, this was Frame's only failure: instead of signing in the wallet, Frame passes these requests through to the RPC provider.

Caching🔗

There are only two hard things in Computer Science: cache invalidation and naming things. Phil Karlton

Caching is hard. We've been bitten by it. In a space where you have to pay for every API request, but your users expect a free wallet experience, caching is a necessary evil.

We suspect, but haven't verified for all wallets, that many of the test failures are caused by stale values cached by wallets.

In the case of eth_call, we have verified that MetaMask specifies an old block number, causing a call into a freshly deployed contract to fail. This is unlikely to cause major issues on a real network with many seconds between blocks, but the tests mine blocks much more frequently, triggering the issue. Regardless, we're following up with their team to see what we can do.

Post-merge Endpoints🔗

Some of the newer endpoints like eth_maxPriorityFeePerGas aren't yet supported in all wallets. Expect these tests to start passing as they are implemented.

Stalled Tests🔗

OKX and TokenPocket both failed somewhere around the eth_getFilterChanges suites, and didn't continue to the rest of the tests. It's likely that they would've passed more tests had they continued.

We haven't debugged why the tests stalled. It could be as simple as adding a timeout to WTF, or something more involved. We'll be looking into this before the next report.

Future Plans🔗

We'll continue adding tests, and we'll try to run through this report somewhat regularly. That said, this is entirely a manual process. Once wallets start running WTF as part of their continuous integration processes, we can make some pretty graphs, include more wallets, and make reports more often.

In the meantime, join us on Discord!