How to balance requirements for Bitcoin key storage and signing
Bitcoin private key custody is a complex and varied operation that requires a delicate balance between trust and risk management. For different use-cases there are different trade-offs, such that the timeless wisdom for most individuals (not your keys, not your UTXOs) does not strictly translate to the needs of a business.
But what exactly does a Bitcoin custodian need to do? The considerations behind securing private keys are a trade-off with risk and trust balanced against the conveniences that various models offer for your needs. The guiding principle is to absolutely minimize risk and trust, even structurally off-setting them where possible by engaging insured partners, while still being able to operate.
Risk and trust must be measured against a variety of activities during which one can expect to use the private keys in question to exercise control over UTXOs. As thoroughly documented in the Knox Custody Risk Management document, these activities can be grouped into four categories. More specifically, here we assess these processes through the lens of a private key lifecycle.
Private key lifecycle
A model for linking core activities as they relate to private keys is to map out how they relate to each other over time. The private key lifecycle might be self-explanatory to some: the idea is to consider the entire existence of a particular private key and the activities it must be capable of exercising, contextualizing everything from its generation to eventual retirement. Conceptualizing a private key lifecycle has long been a central practice in managing the cryptographic keys of any cryptosystem, say for SSH keys allowing a team of developers to access critical infrastructure, so it is helpful to borrow this methodology for a new type of cryptosystem: Bitcoin.
The activities documented in the Private Key Lifecycle help us to define and enforce key management policies. The end result is simple: each individual key requires its own usage policy defining which device(s) and applications can employ it, and precisely what actions (in the case of Bitcoin, signing transactions) can be performed by them. Thus we can piece together not only the life of a private key, but also the processes needed to measure and mitigate risk, and sometimes transfer it. We have been outlining some of the specific ways that Knox covers these activities in our Security Series.
For many of the stages detailed throughout, it’s important to note that it is not possible to properly protect the cryptographic key lifecycle with a network-connected personal computer. At the Bitcoin consumer level, this manifests in specialized key signing devices known as “hardware wallets”, such as the COLDCARD Wallet. Using these devices has become a best practice even for modest holdings. But this is only just good enough for some individual users–at any greater scale, proper key life cycle management is available only by utilizing a specialized Hardware Security Module (HSM), true random entropy generation (using fair dice rolls, for example), and a dedicated key life cycle management system.
A private key is just a very large random number that can’t reasonably be cracked by brute force, even at the hands of an extremely well-resourced adversary. It follows that for this key to be strong it must be truly random, an assurance that is difficult to attain given the deterministic nature of computers. Key generation can be graded on how entropy is generated–say, by a number of dice rolls in combination with a random number generator.
Private keys must also be unique: that is, “wallets” must be segregated at the root entropy level with new keys generated for every new wallet. At Knox, this is taken to an extreme: each wallet uses a distinct multisig 3-of-4 scheme, with each of these root keys generated in a distinct location for every new account.
Critically, many companies in the business of private key storage make use of omnibus accounts where root entropy is shared across clients. This should absolutely not be used to safeguard multiple clients' keys, as it concentrates risk and should be a red flag for those who need these services.
It’s important to note that each individual key is generated, archived and distributed independently, and Knox is very strict about how these activities are sequenced. Each zone is complete before the next begins, which means there's no moment in time when keys are in their most dangerous form at the same time as any other. Thanks to the multisig accounts, this means there is effectively no moment in time when signing authority is even theoretically attainable with the material in the open. Thus, consider the next two stages (Archiving and Distribution) as a distinct series of activities relating to a single key and fully isolated from any other.
Key Archiving, Backup and Storage
The obvious objective in private key storage is to keep the private key safe from a wide range of potential compromises, from collusion to external breaches. Cold storage using an air-gapped HSM is the most basic requirement for storing private keys such that the secrets can not be retrieved. But even here there is a range of considerations to keep in mind: how can the key never be exposed to a network connection, but still remain functional? What sort of security assessments are needed for the HSM? How can sensitive material be backed up such that it doesn’t introduce new attack vectors?
At Knox, keys begin to be processed for archiving immediately after generation. For archiving, each key engaged in the multisig account is itself split into four shards using Shamir’s Secret Sharing (SSS) and archived such that each shard is held in a distinct vault with a third-party security provider. This act of archiving can be seen as our backup of key material.
Secure transport and storage of key shares is provided by a globally-recognized secure logistics firm. These shares are legally owned by clients such that Knox is excluded from any claim on these artefacts. Furthermore, they are derived in a 2-of-4 scheme, meaning destruction of backups at any 2 facilities will not result in key loss, nor can the complete compromise of all backup material in any one city result in disclosure of complete keys.
At its most extreme, even in the event of the total destruction of all archival facilities, customers would be capable of authorizing a sweep into a new account due to the presence of the collusion-resistant signing facilities.
As we have documented before, Knox has gone so far as to lock itself out to mitigate internal collusion, a key guarantee of our insurance coverage.
Key Distribution and Installation
So we have our private key generated and safely backed up, but how do we securely provide access to the right participants?
Concurrent with key archiving, a full workable seed of a key comes to exist in an HSM with specific guards (e.g. requiring a customer’s explicit sign-off before it is used to sign). These HSMs are securely transported by our security and logistics provider back to our three geographically distinct processing facilities for storage and use. This distribution and installation of HSMs is what we refer to as arming a facility with an account.
Ultimately, some sort of access authority must be delivered to the end client, verified and further protected. For our present purposes we are particularly interested in the activities of the Knox facilities themselves, so we will cover the client-side in a future piece. In the meantime, there are brief summaries available on the Knox Custody Security page.
Each full key we have been talking about thus far, whether it is sharded into 4 pieces for archiving, or flashed to an HSM for general storage and use, is one part of a 3-of-4 multisignature wallet. Without getting too technical, this simply means that three distinct private keys are always needed to sign a transaction (e.g. “spend”) from a client account.
But if we are working with several distinct keys that effectively collaborate to produce certain outputs, how do we send transactions to them such that this protocol is enforced? For our purposes, key registration is the act of pulling together the xpubs (public keys) from each of these keys such that multisig addresses can be derived and enforce our desired rules. These public keys are sensitive in that exposing them can result in a loss of transaction privacy, but they are not a security risk in terms of fund loss.
Thus, the generated public keys can be registered to a specific user and stored safely in operational systems. This registration process means the key is ready for use and integrated into operational processes, and includes audits that the key performs as needed. It should be noted that, so far as the key itself is concerned, it has no reference to the existence of any other key in its quorum, so there's no private information that needs to be laced throughout the system to make it all work.
The seemingly impossible task: how can a client use a private key held under custody while maintaining its security assurances? In Bitcoin private key vaulting, the use of a private key is typically for signing transactions, an activity that carries potentially disastrous financial costs if mis-managed. Furthermore, the customer must be assured that only they can initialize a transaction.
At Knox, transaction signing happens 100% offline, and due to the 3-of-4 multisig script, no single physical location has signing authority over a customer account. Knox has no power to sign a transaction with any wallet key unless the customer has sent an authorizing signature on that transaction.
In most cryptosystems, it is often unappreciated that using a private key can give off some critical scraps of information to potential attackers. Given enough use, an amalgamation of this data can be used by incredibly talented operators to potentially expose the full private key. As such, key rotation is a critical component of many cryptosystems, with best practices covering a maximum number of uses or a certain amount of time before a key must be re-generated.
Frankly, this can be a pain to do and is often neglected. Consider a large corporation using distinct private keys on each of their devices to access the company VPN. Rotating all these keys at the same time is a huge coordination challenge, and demonstrates the operational expertise that private key vaulting services can provide beyond their obvious security services.
This is partly mitigated by the nature of Bitcoin itself, as UTXOs are not encumbered by the primary public-private key pair. Instead, each address has its own unique keypair derived from the master keys, and it is best practice to only use each such address once for both privacy and security considerations. At Knox, thanks to our segregated entropy generation, a customer can have any number of sub-accounts, each containing any number of wallets with keys derived from the customer’s root keys using BIP32. Thus, “key rotation” in the context of a Bitcoin key vault is achieved by way of using distinct addresses: unless you were doing regular sweeps of wallets, you wouldn’t want to rotate root keys as this may add new security considerations.
Any risk of losing private key data is a deal-breaker for custodians. Losing it would certainly put the custodian out of business and might even pose a risk to the client’s business considering the scale that typically justifies such a relationship. So key recovery is important, but how to do this while maintaining our previously discussed assurances?
In critical scenarios where archived shards are required from the backup process described earlier, Knox does not simply reconstitute full keys and sweep them to a new account. Rather, Knox first ships new access keys to clients, and once access is granted, Knox visits archive sites with the corresponding public keys. New HSMs are created that are still associated to the exact same account (i.e. public keys, addresses and UTXOs remain the same) but to a different set of access keys (held by the client). Notably, it is important that access keys are shipped prior to key material being accessed to ensure that they are never in the same location and under a single point of control. The replacement HSMs are shipped to our transaction processing facilities by the third-party security and logistics provider where they are securely armed once more in an offline vault.
Key Revocation and Suspension
Key revocation and suspension is another area where the Bitcoin private key lifecycle takes a slightly different form to other cryptosystems. Typically, this stage has to do with the scenario whereby a private key simply expires, or is somehow compromised by theft, loss or other threats.
As we discussed under Key Rotation, root private keys should not have to be rotated, nor should they need to be revoked. Rather, in Bitcoin systems we can imagine key revocation and suspension as the systems in place to ensure that new addresses are derived such that they only ever hold a single unique UTXO in their entire lifecycle. When that UTXO is used as an input to a new transaction, the address is permanently retired in backend systems.
Key destruction in the ultimate sense is full destruction of the physical hardware after revocation.
There will come a time that private keys must be destroyed if they have expired, been revoked, or were found to have been compromised. But even this isn’t easy. You might know that there is no guaranteed way to destroy data from a device that you wish to keep. Data tends to persist, and this is why forensics labs might be able to save your vacation photos from that damaged laptop: data must be explicitly overwritten to be removed. Obviously, this is a serious concern for sensitive private key data, and must be handled thoroughly.
In the case of Knox, physical key destruction might entail incinerating or chemically dissolving key-storing devices. However, while this is a good practice, it should be noted that private keys to an empty Bitcoin wallet aren’t particularly sensitive compared with those in other cryptosystems.
Bringing it all together
Managing the private key lifecycle and all the processes involved at each stage is the operational overhead that makes a private key custodian much more than an expensive hardware wallet. Everyday holders can learn a lot from these processes, but at a certain scale it’s a practical necessity to engage specialized operators: not only for their practical expertise, but for offsetting some staggering risks from their own books and operational oversight.
If you need to manage Bitcoin private keys and want to offset that risk with insurance, check out Knox Custody to help secure your share of Bitcoin’s information space. More information about our own processes behind the private key lifecycle can be found on our Security page.