Test Report #2

Test Report🔗

Summary🔗

Disclaimer🔗

Our intent is not to claim that certain wallets are better than others, and a low score here is as likely to indicate that our tests have problems as it is to indicate bugs in a wallet. We're very interested in improving scores across the board, so we're happy to accept contributions to fix tests and just as happy to help debug.

Wallet Selection🔗

Very little of this report is scientific/rigorous, and the wallet selection process was no exception. We chose a few wallets we're familiar with, and a small sampling of random wallets from the Find an Ethereum Wallet list.

Because Wallet Test Framework only supports window.ethereum so far, we're limited to wallets that provide it; that pretty much excludes mobile wallets for the time being. Our architecture requires connecting to a custom chain, so wallets that don't support custom chains cannot be tested (omitted from this table.)

For future reports, get in touch if you're interested in having your results published (or if you'd prefer we didn't mention you.)

Discussion🔗

Automation🔗

The most exciting part of this Test Report is the new "Automated" column. Wallets with a "✔️ " (currently Taho and Coinbase) use the newest feature of WTF, custom glue. Both of these wallets use [Selenium WebDriver]-based automation to interact with their user interface to respond to prompts, like connecting or signing transactions.

Automating wallets is the first step in our plan—removing humans from the process means the tests can run unattended and regularly. Next, we plan to build a reporting platform to track results over time and highlight regressions.

Here's what Taho looks like running the tests:

Common Issues🔗

There are several issues that are hurting scores that we see across wallets: incomplete information, unsupported features, minor deviations from the JSON-RPC specification, and out-of-date information.

Far too many wallets (like Enkrypt, Coin98, etc.) do not display the source or destination address when preparing a transaction. Commonly wallets will display an abbreviated address (0xabcd...1234) and provide no way to view the full value. This is a security issue, since it is relatively easy to mine addresses that collide on only four bytes. Always provide a way to access the full value, like in a tooltip or a "copy" button.

Perhaps following MetaMask's lead, many wallets do not support the eth_signTransaction endpoint. Some other wallets do not support subscriptions. While adding support for these features is nontrivial, it is important to provide the same base level of functionality.

In other cases, wallets do provide a feature but do not follow the specification. Coinbase, for example, omits the 0x prefix on hexadecimal log results and xdefi implements a very different eth_signTransaction. Having to work around these differences increases the burden on dapp developers.

Finally, there is a whole class of bugs related to out-of-date information being presented to the user. For example, Opera Wallet does not refresh balance information while authorizing a transaction, preventing the user from completing a transaction that would otherwise succeed.

Conclusion🔗

That's pretty much it for this Test Report. In the meantime, join us on Discord!