Test Report #2

Test ReportπŸ”—


Wallet Test Framework – 94f9943cea81ccb8a3456ec031ad1bd00ee9a1f6

Not a statement of a wallet's quality or correctness.

Wallet Automated Version Passing Tests Pass % Platform Notes
Brave Wallet ❌ 61 100% Brave v1.63.16
Frame ❌ 0.6.8 60 98% Chromium 120.0.6099.129 signTransaction unsupported
Taho βœ”οΈ 0.57+ 60 98% Chrome 122.0.6261.94 Changed definition of sign test to exclude hex
Enkrypt ❌ 1.33.0 59 97% Chrome 122.0.6261.94 signTransaction unsupported, doesn't show destination address
Rainbow ❌ 1.4.8 56 92% Chrome 122.0.6261.94 signTransaction opened, but was blank
Backpack ❌ 0.10.37 56 92% Chrome 122.0.6261.94 sendTransaction invalid length, sign just closed
Opera Wallet ❌ 56 92% Opera 107.0.5045.36 Doesn't update balance before sending
Coin98 ❌ 9.1.6 55 90% Chrome 122.0.6261.94 Doesn't display full addresses
Aurox ❌ 2.0.0 53 87% Chrome 122.0.6261.94
Metamask ❌ 11.10.1 48 79% Chromium 120.0.6099.129 signTransaction unsupported, logging/storage seems to lag behind chain
Coinbase βœ”οΈ 3.57.0 47 77% Chromium 120.0.6099.129 Doesn't show full addresses, can't differentiate sign/send transaction
TokenPocket ❌ 1.2.4 46 75% Chrome 122.0.6261.94 signTransaction unsupported
xdefi ❌ <28.3.6 45 74% Chrome 122.0.6261.94 Notably different signTransaction behaviour
Trust ❌ 2.7.2 43 70% Chrome 122.0.6261.94 Doesn't support filters/subscriptions
BlockWallet ❌ 1.2.4 40 66% Chrome 122.0.6261.94
OKX ❌ 2.85.0 22 36% Chromium 120.0.6099.129 As likely to be a bug in WTF as a wallet problem


Our intent is not to claim that certain wallets are better than others, and a low score here is as likely to indicate that our tests have problems as it is to indicate bugs in a wallet. We're very interested in improving scores across the board, so we're happy to accept contributions to fix tests and just as happy to help debug.

Wallet SelectionπŸ”—

Very little of this report is scientific/rigorous, and the wallet selection process was no exception. We chose a few wallets we're familiar with, and a small sampling of random wallets from the Find an Ethereum Wallet list.

Because Wallet Test Framework only supports window.ethereum so far, we're limited to wallets that provide it; that pretty much excludes mobile wallets for the time being. Our architecture requires connecting to a custom chain, so wallets that don't support custom chains cannot be tested (omitted from this table.)

For future reports, get in touch if you're interested in having your results published (or if you'd prefer we didn't mention you.)



The most exciting part of this Test Report is the new "Automated" column. Wallets with a "βœ”οΈ " (currently Taho and Coinbase) use the newest feature of WTF, custom glue. Both of these wallets use [Selenium WebDriver]-based automation to interact with their user interface to respond to prompts, like connecting or signing transactions.

Automating wallets is the first step in our planβ€”removing humans from the process means the tests can run unattended and regularly. Next, we plan to build a reporting platform to track results over time and highlight regressions.

Here's what Taho looks like running the tests:

Common IssuesπŸ”—

There are several issues that are hurting scores that we see across wallets: incomplete information, unsupported features, minor deviations from the JSON-RPC specification, and out-of-date information.

Far too many wallets (like Enkrypt, Coin98, etc.) do not display the source or destination address when preparing a transaction. Commonly wallets will display an abbreviated address (0xabcd...1234) and provide no way to view the full value. This is a security issue, since it is relatively easy to mine addresses that collide on only four bytes. Always provide a way to access the full value, like in a tooltip or a "copy" button.

Perhaps following MetaMask's lead, many wallets do not support the eth_signTransaction endpoint. Some other wallets do not support subscriptions. While adding support for these features is nontrivial, it is important to provide the same base level of functionality.

In other cases, wallets do provide a feature but do not follow the specification. Coinbase, for example, omits the 0x prefix on hexadecimal log results and xdefi implements a very different eth_signTransaction. Having to work around these differences increases the burden on dapp developers.

Finally, there is a whole class of bugs related to out-of-date information being presented to the user. For example, Opera Wallet does not refresh balance information while authorizing a transaction, preventing the user from completing a transaction that would otherwise succeed.


That's pretty much it for this Test Report. In the meantime, join us on Discord!