Skip to main content
Version: 0.6

Storage Interface

Introduction​

The high-level account API takes care of publishing updates to an identity and storing secrets securely. It does the latter by using an implementation of the Storage interface. In this section, we will go into more depth of the interface, and how to implement that interface.

The key idea behind the interface is strongly inspired by the architecture of key management systems (KMS) or secure enclaves: once private keys are entered into the system, they can never be retrieved again. Instead, all operations using the key will have to go through that system. This approach is what allows Storage implementations to be architected more securely than simply storing and loading private keys from a regular database. Of course, the security is directly dependent on the concrete implementation, which is why we provide one such implementation using Stronghold, and strongly recommend using it. However, there are cases where one cannot use Stronghold or may want to integrate key management of identities into their own KMS or similar, which is why the Storage interface is an abstraction over such systems. Any implementation of that interface can then be used by the Account.

The storage interface has three major categories of functions. A brief overview of those functions:

  • DID Operations: Management of identities.
    • did_create: Based on a private key, or a generated one, creates a new DID.
    • did_list: List all DIDs in this Storage.
    • did_exists: Returns whether the given DID exists in this Storage.
    • did_purge: Wipes all data related to the given DID.
  • Key Operations: Various functionality to managing cryptographic keys.
    • key_generate: Generates a new key for the given DID.
    • key_insert: Inserts a pre-existing private key for the given DID location.
    • key_public: Calculates and returns the public key for the given location to a private key.
    • key_delete: Removes the key at the given location.
    • key_sign: Signs the given data with the key at the given location.
    • key_exists: Returns whether the key at the given location exists.
  • Data Operations: Used for keeping state persistent. Storages only need to serialize and store the data.
    • chain_state_get: Returns the ChainState data structure for the given DID.
    • chain_state_set: Sets the ChainState data structure for the given DID.
    • document_get: Returns the DID document for the given DID.
    • document_set: Sets the DID document for the given DID.

Storage Layout​

Identifiers​

There are two types of identifiers in the interface, DIDs and key locations. A DID identifies an identity, while a key location identifies a key. An implementation recommendation is to use the DID as a partition key. Everything related to a DID can be stored in a partition identified by that DID. Importantly, the location of a key is only guaranteed to be unique within the DID partition it belongs to. If no partitioning is used, then DID and key location should be combined (e.g. concatenated) to produce a single, globally unique (i.e. across all identities) identifier for a key in storage.

Representations​

A KeyLocation is a compound identifier based on the fragment of a verification method and the hash of a public key. The motivation for this design is that a KeyLocation can be derived given a DID document and one of its verification methods. Thus, no additional state is necessary.

Canonical string representations of the IotaDID and KeyLocation type can be obtained using the string representation of a DID and the canonical method on KeyLocation respectively. These representations are intended to be kept stable as much as possible.

Example layout​

This illustrates the recommended approach for partitioning the storage layout (where location -> key is a mapping from location to key):

  • did:iota:Ft3wA8Tv2nF25hij3aegR54Wvqju7t5zqW9xnCB5L3Wu
    • sign-0:16843234495045965331 -> 0xc6f0dbacd56156ff4c383d549ac61ada87f8aa69454f3bfae99f5fa9e093a5c3
    • kex-0:7560300328640998700 -> 0xe494e36164e0a760140f3a9ab7dfdad38edac698f93d5239655dbd7499194760
  • did:iota:DSvXWs7FUch9MQcaUKmrRFZyHYcHwt3t3pbjvKsQBfep
    • sign-0:16843234495045965331 -> 0xc6f0dbacd56156ff4c383d549ac61ada87f8aa69454f3bfae99f5fa9e093a5c3
    • kex-0:16546298247591944074 -> 0x8e1d037cd343f84276ab737b638da9095bcb6052f7fd9628d21d20f434f9959a
    • key:8559754420653090937 -> 0x4ef484a54aa16503878aa1ecaa6d73cb8254aefa3f80a569ed33ca685289d01e

Note how fragments (such as kex-0) can appear more than once, but the hash of the public key - calculated from the stored private key - makes the location unique in general.

caution

Although unlikely in practice, even the same private key can be used across different DIDs, which produces the same key location (here sign-0:16843234495045965331) and it's important that these are stored independently, so that deleting one does not accidentally delete the other. Hence why a key's full identifier in storage needs to be based on the DID and the key location.

That said, the following flattened structure also satisfies the requirements:

  • did:iota:Ft3wA8Tv2nF25hij3aegR54Wvqju7t5zqW9xnCB5L3Wu:sign-0:16843234495045965331 -> 0xc6f0dbacd56156ff4c383d549ac61ada87f8aa69454f3bfae99f5fa9e093a5c3
  • did:iota:Ft3wA8Tv2nF25hij3aegR54Wvqju7t5zqW9xnCB5L3Wu:kex-0:7560300328640998700 -> 0xe494e36164e0a760140f3a9ab7dfdad38edac698f93d5239655dbd7499194760
  • did:iota:DSvXWs7FUch9MQcaUKmrRFZyHYcHwt3t3pbjvKsQBfep:sign-0:16843234495045965331 -> 0xc6f0dbacd56156ff4c383d549ac61ada87f8aa69454f3bfae99f5fa9e093a5c3
  • did:iota:DSvXWs7FUch9MQcaUKmrRFZyHYcHwt3t3pbjvKsQBfep:kex-0:16546298247591944074 -> 0x8e1d037cd343f84276ab737b638da9095bcb6052f7fd9628d21d20f434f9959a
  • did:iota:DSvXWs7FUch9MQcaUKmrRFZyHYcHwt3t3pbjvKsQBfep:key:8559754420653090937 -> 0x4ef484a54aa16503878aa1ecaa6d73cb8254aefa3f80a569ed33ca685289d01e

The primary advantage of the partitioning is that it simplifies the implementation of the did_purge operation, which wipes all data belonging to a given DID. With partitioning, this operation can simply wipe the partition whereas a storage with a flattened layout will have to do more work.

Indexing​

The interface has two methods called did_list and did_exists. These return the list of stored DIDs, and whether a DID exists in storage, respectively. Implementations are thus expected to maintain a list or index of stored DIDs. An identity created with did_create is added to the index, while an identity deleted through did_purge is removed from the index.

If the storage implementation can be accessed concurrently, then access to the index needs to be synchronized, since it is unique per storage instance.

Implementation​

The IOTA Identity framework ships two implementations of Storage. The MemStore is an insecure in-memory implementation intended as an example implementation and for testing. The secure and recommended Storage is Stronghold. Stronghold may be interesting for implementers to look at, as it needs to deal with some challenges the in-memory version does not have.

This section will detail some common challenges and embeds the MemStore implementations in Rust and TypeScript.

Challenges​

The did_create method takes the fragment of the initial verification method, the name of a network in which the DID will eventually exist, and an optional private key. From these inputs, it either generates a key or uses the passed private key to calculate the public key and from that derive the DID. In case a key needs to be generated, the challenge is to obtain the location for the key to be stored at. Since the key location depends on the public key, but key generation likely needs a location for the key to be stored at, there is a circular dependency that needs to be resolved. This can be resolved in at least two ways.

  1. Generate the key at a random location, then derive the actual location and move the key there
  2. If moving a key is not possible, then an additional mapping from key location to some storage-internal location identifier can be maintained. Then it's possible to generate the key at some storage-internal location, calculate the key location and store the mapping.

Since this also needs to happen before the DID can be derived from the public key, similar approaches can be used to work around the not-yet available DID partition key. Storages may choose to have one statically identified partition where keys are generated initially, and then moved from there. Storages whose restrictions do not allow for this, may want to use the flattened storage layout described in example layout and use the mapping approach.

Storage Test Suite​

The StorageTestSuite can be used to test the basic functionality of storage implementations. See its documentation (Rust docs, Wasm docs) for more details.

Examples​

This section shows the Rust and TypeScript MemStore implementations, which are thoroughly commented.

// Copyright 2020-2022 IOTA Stiftung
// SPDX-License-Identifier: Apache-2.0

import { ChainState, DID, Document, Ed25519, KeyLocation, KeyPair, KeyType, Signature, Storage, StorageTestSuite, EncryptionAlgorithm, CekAlgorithm, EncryptedData } from '../../node/identity_wasm.js';

/** An insecure, in-memory `Storage` implementation that serves as an example.
This can be passed to the `AccountBuilder` to create accounts with this as the storage. */
// Refer to the `Storage` interface docs for high-level documentation of the individual methods.
export class MemStore implements Storage {
    // We use strings as keys rather than DIDs or KeyLocations because Maps use
    // referential equality for object keys, and thus a primitive type needs to be used instead.

    // The map from DIDs to chain states.
    private _chainStates: Map<string, ChainState>;
    // The map from DIDs to DID documents.
    private _documents: Map<string, Document>;
    // The map from DIDs to vaults.
    private _vaults: Map<string, Map<string, KeyPair>>;

    /** Creates a new, empty `MemStore` instance. */
    constructor() {
        this._chainStates = new Map();
        this._documents = new Map();
        this._vaults = new Map();
    }

    public async didCreate(network: string, fragment: string, privateKey?: Uint8Array): Promise<[DID, KeyLocation]> {
        // Extract a `KeyPair` from the passed private key or generate a new one.
        // For `did_create` we can assume the `KeyType` to be `Ed25519` because
        // that is the only currently available signature type.
        let keyPair;
        if (privateKey) {
            keyPair = KeyPair.tryFromPrivateKeyBytes(KeyType.Ed25519, privateKey);
        } else {
            keyPair = new KeyPair(KeyType.Ed25519);
        }

        // We create the location at which the key pair will be stored.
        // Most notably, this uses the public key as an input.
        const keyLocation: KeyLocation = new KeyLocation(KeyType.Ed25519, fragment, keyPair.public());

        // Next we use the public key to derive the initial DID.
        const did: DID = new DID(keyPair.public(), network);

        // We use the vaults as the index of DIDs stored in this storage instance.
        // If the DID already exists, we need to return an error. We don't want to overwrite an existing DID.
        if (this._vaults.has(did.toString())) {
            throw new Error("identity already exists");
        }

        const vault = this._vaults.get(did.toString());

        // Get the existing vault and insert the key pair,
        // or insert a new vault with the key pair.
        if (vault) {
            vault.set(keyLocation.canonical(), keyPair);
        } else {
            const newVault = new Map([[keyLocation.canonical(), keyPair]]);
            this._vaults.set(did.toString(), newVault);
        }

        return [did, keyLocation];
    }

    public async didPurge(did: DID): Promise<boolean> {
        // This method is supposed to be idempotent,
        // so we only need to do work if the DID still exists.
        // The return value signals whether the DID was actually removed during this operation.
        if (this._vaults.has(did.toString())) {
            this._chainStates.delete(did.toString());
            this._documents.delete(did.toString());
            this._vaults.delete(did.toString());
            return true;
        }

        return false;
    }

    public async didExists(did: DID): Promise<boolean> {
        return this._vaults.has(did.toString());
    }

    public async didList(): Promise<Array<DID>> {
        // Get all keys from the vaults and parse them into DIDs.
        return Array.from(this._vaults.keys()).map((did) => DID.parse(did));
    }

    public async keyGenerate(did: DID, keyType: KeyType, fragment: string): Promise<KeyLocation> {
        // Generate a new key pair with the given key type.
        const keyPair: KeyPair = new KeyPair(keyType);
        // Derive the key location from the fragment and public key and set the `KeyType` of the location.
        const keyLocation: KeyLocation = new KeyLocation(KeyType.Ed25519, fragment, keyPair.public());

        const vault = this._vaults.get(did.toString());

        // Get the existing vault and insert the key pair,
        // or insert a new vault with the key pair.
        if (vault) {
            vault.set(keyLocation.canonical(), keyPair);
        } else {
            const newVault = new Map([[keyLocation.canonical(), keyPair]]);
            this._vaults.set(did.toString(), newVault);
        }

        // Return the location at which the key was generated.
        return keyLocation;
    }

    public async keyInsert(did: DID, keyLocation: KeyLocation, privateKey: Uint8Array): Promise<void> {
        // Reconstruct the key pair from the given private key with the location's key type.
        const keyPair: KeyPair = KeyPair.tryFromPrivateKeyBytes(keyLocation.keyType(), privateKey);

        // Get the vault for the given DID.
        const vault = this._vaults.get(did.toString());

        // Get the existing vault and insert the key pair,
        // or insert a new vault with the key pair.
        if (vault) {
            vault.set(keyLocation.canonical(), keyPair);
        } else {
            const newVault = new Map([[keyLocation.canonical(), keyPair]]);
            this._vaults.set(did.toString(), newVault);
        }
    }

    public async keyExists(did: DID, keyLocation: KeyLocation): Promise<boolean> {
        // Get the vault for the given DID.
        const vault = this._vaults.get(did.toString());

        // Within the DID vault, check for existence of the given location.
        if (vault) {
            return vault.has(keyLocation.canonical());
        } else {
            return false
        }
    }

    public async keyPublic(did: DID, keyLocation: KeyLocation): Promise<Uint8Array> {
        // Get the vault for the given DID.
        const vault = this._vaults.get(did.toString());

        // Return the public key or an error if the vault or key does not exist.
        if (vault) {
            const keyPair: KeyPair | undefined = vault.get(keyLocation.canonical());
            if (keyPair) {
                return keyPair.public()
            } else {
                throw new Error('Key location not found')
            }
        } else {
            throw new Error('DID not found')
        }
    }

    public async keyDelete(did: DID, keyLocation: KeyLocation): Promise<boolean> {
        // Get the vault for the given DID.
        const vault = this._vaults.get(did.toString());

        // This method is supposed to be idempotent, so we delete the key
        // if it exists and return whether it was actually deleted during this operation.
        if (vault) {
            return vault.delete(keyLocation.canonical());
        } else {
            return false;
        }
    }

    public async keySign(did: DID, keyLocation: KeyLocation, data: Uint8Array): Promise<Signature> {
        if (keyLocation.keyType() !== KeyType.Ed25519) {
            throw new Error('Unsupported Method')
        }

        // Get the vault for the given DID.
        const vault = this._vaults.get(did.toString());

        if (vault) {
            const keyPair: KeyPair | undefined = vault.get(keyLocation.canonical());

            if (keyPair) {
                // Use the `Ed25519` API to sign the given data with the private key.
                const signature: Uint8Array = Ed25519.sign(data, keyPair.private());
                // Construct a new `Signature` wrapper with the returned signature bytes.
                return new Signature(signature)
            } else {
                throw new Error('Key location not found')
            }
        } else {
            throw new Error('DID not found')
        }
    }

    public async dataEncrypt(did: DID, plaintext: Uint8Array, associatedData: Uint8Array, encryptionAlgorithm: EncryptionAlgorithm, cekAlgorithm: CekAlgorithm, publicKey: Uint8Array): Promise<EncryptedData> {
        throw new Error('not yet implemented')
    }

    public async dataDecrypt(did: DID, data: EncryptedData, encryptionAlgorithm: EncryptionAlgorithm, cekAlgorithm: CekAlgorithm, privateKey: KeyLocation): Promise<Uint8Array> {
        throw new Error('not yet implemented')
    }

    public async chainStateGet(did: DID): Promise<ChainState | undefined> {
        // Lookup the chain state of the given DID.
        return this._chainStates.get(did.toString());
    }

    public async chainStateSet(did: DID, chainState: ChainState): Promise<void> {
        // Set the chain state of the given DID.
        this._chainStates.set(did.toString(), chainState);
    }

    public async documentGet(did: DID): Promise<Document | undefined> {
        // Lookup the DID document of the given DID.
        return this._documents.get(did.toString())
    }

    public async documentSet(did: DID, document: Document): Promise<void> {
        // Set the DID document of the given DID.
        this._documents.set(did.toString(), document);
    }

    public async flushChanges(): Promise<void> {
        // The MemStore doesn't need to flush changes to disk or any other persistent store,
        // which is why this function does nothing.
    }
}

export async function storageTestSuite() {
    await StorageTestSuite.didCreateGenerateKeyTest(new MemStore());
    await StorageTestSuite.didCreatePrivateKeyTest(new MemStore());
    await StorageTestSuite.keyGenerateTest(new MemStore());
    await StorageTestSuite.keyDeleteTest(new MemStore());
    await StorageTestSuite.keyInsertTest(new MemStore());
    await StorageTestSuite.didListTest(new MemStore());
    await StorageTestSuite.keySignEd25519Test(new MemStore());
    await StorageTestSuite.didPurgeTest(new MemStore());
}