Store, Table-Based Introspectable Storage

Abstract

This standard introduces a flexible on-chain storage pattern that organizes data into structured tables that consist of records with fixed key and value schemas, similar to a traditional database. This storage pattern consists of a unified contract interface for data access, along with a compact binary encoding format for both static and dynamic data types. State changes are tracked through standardized events that enable automatic, schema-aware state replication by off-chain indexers. New tables can be dynamically registered at runtime through a special table that stores schema metadata for all tables, allowing the system to evolve without breaking existing contracts or integrations.

Motivation

The absence of consistent standards for on-chain data management in smart contracts can lead to rigid implementations, tightly coupled contract logic with off-chain services, and challenges in updating or extending a contract’s data layout without breaking existing integrations.

Using the storage mechanism defined in this ERC provides the following benefits:

1. Automatic Indexing: By emitting consistent, standardized events during state changes, off-chain services can automatically track on-chain state and provide schema-aware indexer APIs.
2. Elimination of Custom Getter Functions: Any contract or off-chain service can read stored data through a consistent interface, decoupling smart contract implementation from specific data access patterns and reducing development overhead.
3. Simpler Upgradability: This pattern leverages unstructured storage, making it easier to upgrade contract logic without the risks associated with using a fixed storage layout.
4. Flexible Data Extensions: New tables can be added at runtime without without breaking existing integrations with other data consumers.
5. Reduced gas costs: Using efficient data packing reduces gas costs for both storage and event emissions.

Specification

Definitions

Store

A smart contract that implements the interface proposed by this ERC and organizes data in Tables. It emits events for each data operation so that off-chain components can replicate the state of all tables.

Table

A storage structure that holds Records sharing the same Schema.

• On-chain Table: Stores its state on-chain and emits events for off-chain- indexers.
• Off-chain Table: Does not store state on-chain but emits events for off-chain indexers.

Record

A piece of data stored in a Table, addressed by one or more keys.

ResourceId

A 32-byte value that uniquely identifies each Table within the Store.

type ResourceId is bytes32;

Encoding:

Table Type Identifiers:

• 0x7462 ("tb") for on-chain tables
• 0x6f74 ("ot") for off-chain tables

Schema

Used to represent the layout of Records within a table.

type Schema is bytes32;

Each Table defines two schemas:

• Key Schema: the types of the keys used to uniquely identify a Record within a table. It consists only of fixed-length data types.
• Value Schema: the types of the value fields of a Record within a table, which can include both fixed-length and variable-length data types.

SchemaType

Single byte that represents the type of a specific static or dynamic field.

enum SchemaType { ... }

Type Encoding:

FieldLayout

Encodes the concrete value Schema information, specifically the total byte length of the static fields, the number of dynamic fields and the length of each static field on its own.

This encoding serves as an optimization for on-chain operations. By having the exact lengths readily available, the Store doesn’t need to repeatedly compute or translate the schema definitions into actual field lengths during execution.

type FieldLayout is bytes32;

EncodedLengths

Encodes the byte length of all the dynamic fields of a specific Record. It is returned by the Store methods when reading a Record, as it is needed for decoding dynamic fields.

type EncodedLengths is bytes32;

Packed Data Encoding

Record data returned by Store methods and included in Store events uses the following encoding rules.

Field Limits

• Maximum Total Fields: A record can contain up to 28 fields in total (both static and dynamic fields combined).
  • This limit is due to the Schema type structure, which uses 28 bytes (bytes 4 to 31) to define field types, with one byte per field (SchemaType).
• Dynamic Fields Limit: A record can have up to 5 dynamic fields.
  • This is due to the fact that a single 32 bytes word (EncodedLengths) to encode the byte lengths of each dynamic field, instead of encoding each length separately as Solidity’s abi.encode would.
• Static Fields Limit: The maximum number of static fields is 28 minus the number of dynamic fields.
  • For example, if there are 5 dynamic fields, the maximum number of static fields is 23 (28 - 5).

Encoding Rules

• Static-length fields are encoded without any padding, and concatenated in the order they are defined in the schema, which is equivalent to using Solidity’s abi.encodePacked.
• For dynamic-length fields (arrays, bytes, and strings):
  • If the field is an array, its elements are tightly packed without padding.
  • All dynamic fields are concatenated together without padding and without including their lengths.
  • The lengths of all dynamic fields are encoded into a single EncodedLengths.

Example

Suppose a table has the following value schema:

(uint256 id, address owner, string description, uint8[] scores)

Encoding (Pseudocode):

bytes memory staticData = abi.encodePacked(id, owner);

// This is a custom function as Solidity does not provide a way to tightly pack array elements
bytes memory packedScores = packElementsWithoutPadding(scores);

// abi.encodePacked concatenates both description and packedScores without including their lengths
bytes memory dynamicData = abi.encodePacked(description, packedScores);

// Total length is encoded in the 56 least significant bits
EncodedLengths encodedLengths = dynamicData.length;

// Each length is encoded using 5 bytes
encodedLengths |= (description.length << (56));
encodedLengths |= (encodedData.length << (56 + 8 * 5));

// The full encoded record data is represented by the following tuple:
// (staticData, encodedLengths, dynamicData)

Store Interface

All Stores MUST implement the following interface.

interface IStore {
  /**
   * Get full encoded record (all fields, static and dynamic data) for the given tableId and key tuple.
   */
  function getRecord(
    ResourceId tableId,
    bytes32[] calldata keyTuple
  ) external view returns (bytes memory staticData, EncodedLengths encodedLengths, bytes memory dynamicData);

  /**
   * Get a single encoded field from the given tableId and key tuple.
   */
  function getField(
    ResourceId tableId,
    bytes32[] calldata keyTuple,
    uint8 fieldIndex
  ) external view returns (bytes memory data);

  /**
   * Get the byte length of a single field from the given tableId and key tuple
   */
  function getFieldLength(
    ResourceId tableId,
    bytes32[] memory keyTuple,
    uint8 fieldIndex
  ) external view returns (uint256);
}

The return values of both getRecord and getField use the encoding rules previously defined in the Packed Data Encoding section. More specifically, getRecord returns the fully encoded record tuple, and the data returned by getField is encoded using the encoding rules as if the field was being encoded on its own.

Store Operations and Events

This standard defines three core operations for manipulating records in a table: setting, updating, and deleting. For each operation, specific events must be emitted. The implementation details of these operations are left to the discretion of each Store implementation.

The fundamental requirement is that for on-chain tables the Record data retrieved through the Store interface methods at any given block MUST be consistent with the Record data that would be obtained by applying the operations implied by the Store events up to that block. This ensures data integrity and allows for accurate off-chain state reconstruction.

Store_SetRecord

Setting a Record means overwriting all of its fields. This operation can be performed whether the record has been set before or not (the standard does not enforce existence checks).

The Store_SetRecord event MUST be emitted whenever the full data of a record has been overwritten.

event Store_SetRecord(
  ResourceId indexed tableId,
  bytes32[] keyTuple,
  bytes staticData,
  EncodedLengths encodedLengths,
  bytes dynamicData
);

Parameters:

Store_SpliceStaticData

Splicing the static data of a Record consists in overwriting bytes of the packed encoded static fields. The total length of static data does not change as it is determined by the table’s value schema.

The Store_SpliceStaticData event MUST be emitted whenever the static data of the Record has been spliced.

event Store_SpliceStaticData(
  ResourceId indexed tableId,
  bytes32[] keyTuple,
  uint48 start,
  bytes data
);

Parameters:

Store_SpliceDynamicData

Splicing the dynamic data of a Record involves modifying the packed encoded representation of its dynamic fields by removing, replacing, and/or inserting new bytes in place.

The Store_SpliceDynamicData event MUST be emitted whenever the dynamic data of the Record has been spliced.

event Store_SpliceDynamicData(
  ResourceId indexed tableId,
  bytes32[] keyTuple,
  uint8 dynamicFieldIndex,
  uint48 start,
  uint40 deleteCount,
  EncodedLengths encodedLengths,
  bytes data
);

Parameters:

Store_DeleteRecord

The Store_DeleteRecord event MUST be emitted whenever the Record has been deleted from the Table.

event Store_DeleteRecord(ResourceId indexed tableId, bytes32[] keyTuple);

Parameters:

See the reference implementation section for an example on how to index store events.

The Tables table

To keep track of the information of each table and support registering new tables at runtime, the Store implementation MUST include a special on-chain Tables table, which behaves the same way as other on-chain tables except for the special constraints mentioned below.

The Tables table MUST use the following Schemas:

• Key Schema:
  • tableId (ResourceId): ResourceId of the table this record describes.
• Value Schema:
  • fieldLayout (FieldLayout): encodes the byte length of each static data type in the table.
  • keySchema (Schema): represents the data types of the (composite) key of the table.
  • valueSchema (Schema): represents the data types of the value fields of the table.
  • abiEncodedKeyNames (bytes): ABI encoded string array of key names.
  • abiEncodedFieldNames (bytes): ABI encoded string array of field names.

Records stored in the Tables table are considered immutable:

• The Store MUST emit a single Store_SetRecord event for each table being registered.
• The Store SHOULD NOT emit any other Store events for a Table registered in the Tables table.

The Tables table MUST store a record that describes itself before any other table is registered, emitting the corresponding Store_SetRecord event. The record must use the following tableId:

// First two bytes indicates that this is an on-chain table
// The next 30 bytes are the unique identifier for the Tables table
// bytes32("tb") | bytes32("store") >> (2 * 8) | bytes32("Tables") >> (2 * 8 + 14 * 8)
ResourceId tableId = ResourceId.wrap(0x746273746f72650000000000000000005461626c657300000000000000000000);

By using a predefined ResourceId and Schema for the Tables table, off-chain indexers can interpret store events for all registered tables. This enables the development of advanced off-chain services that operate on structured data rather than raw encoded data like in the previous indexer implementation example.

Rationale

Splice Events

While the Store_SetRecord event suffices for tracking the data of each record off-chain, including Splice events (Store_SpliceStaticData and Store_SpliceDynamicData) allows for more efficient partial updates. When only a portion of a record changes, emitting a full SetRecord event would be inefficient because the entire record data would need to be read from storage and emitted. Splice events enable the store to emit only the minimal necessary data for the update, reducing gas consumption. This is particularly important for records with large dynamic fields, as the cost of updating them doesn’t grow with the field’s size.

Disallowing Arrays of Dynamic Types

Arrays of dynamic types (e.g., string[], bytes[]) are intentionally not included as supported SchemaTypes. This restriction enforces a flat data schema, which simplifies the store implementation and enhances efficiency. If users need to store such data structures, they can model them using a separate table with a schema like { index: uint256, data: bytes }, where each array element is represented as an individual record.

FieldLayout Optimization

Including the FieldLayout in the Tables schema provides an on-chain optimization by precomputing and storing the exact byte lengths of static fields. This eliminates the need to repeatedly compute field lengths and offsets during runtime, which can be gas-intensive. By having this information readily available, the store can perform storage operations more efficiently, while components reading from the store can retrieve it from the Tables table to decode the corresponding records.

Special Tables table

Including a special Tables table provides significant benefits for off-chain indexers. While emitting events for table registration isn’t strictly necessary for basic indexers that operate on raw encoded data, doing so makes indexers aware of the schemas used by each table. This awareness enables the development of more advanced, schema-aware indexer APIs (e.g., SQL-like query capabilities), enhancing the utility and flexibility of off-chain data interactions.

By reusing existing Store abstractions for table registration, we also simplify the implementation and eliminate the need for additional, specific table registration events. Indexers can leverage the standard Store events to access schema information, ensuring consistency and reducing complexity.

Reference Implementation

Store Event Indexing

The following example shows how a simple in-memory indexer can use the Store events to replicate the Store state off-chain. It is important to note that this indexer operates over raw encoded data which is not that useful on its own, but can be improved as we will explain in the next section.

We use TypeScript for this example but it can easily be replicated with other languages.

type Hex = `0x${string}`;

type Record = {
  staticData: Hex;
  encodedLengths: Hex;
  dynamicData: Hex;
};

const store = new Map<string, Record>();

// Create a key string from a table ID and key tuple to use in our store Map above
function storeKey(tableId: Hex, keyTuple: Hex[]): string {
  return `${tableId}:${keyTuple.join(",")}`;
}

// Like `Array.splice`, but for strings of bytes
function bytesSplice(
  data: Hex,
  start: number,
  deleteCount = 0,
  newData: Hex = "0x"
): Hex {
  const dataNibbles = data.replace(/^0x/, "").split("");
  const newDataNibbles = newData.replace(/^0x/, "").split("");
  return `0x${dataNibbles
    .splice(start, deleteCount * 2)
    .concat(newDataNibbles)
    .join("")}`;
}

function bytesLength(data: Hex): number {
  return data.replace(/^0x/, "").length / 2;
}

function processStoreEvent(log: StoreEvent) {
  if (log.eventName === "Store_SetRecord") {
    const key = storeKey(log.args.tableId, log.args.keyTuple);

    // Overwrite all of the Record's fields
    store.set(key, {
      staticData: log.args.staticData,
      encodedLengths: log.args.encodedLengths,
      dynamicData: log.args.dynamicData,
    });
  } else if (log.eventName === "Store_SpliceStaticData") {
    const key = storeKey(log.args.tableId, log.args.keyTuple);
    const record = store.get(key) ?? {
      staticData: "0x",
      encodedLengths: "0x",
      dynamicData: "0x",
    };

    // Splice the static field data of the Record
    store.set(key, {
      staticData: bytesSplice(
        record.staticData,
        log.args.start,
        bytesLength(log.args.data),
        log.args.data
      ),
      encodedLengths: record.encodedLengths,
      dynamicData: record.dynamicData,
    });
  } else if (log.eventName === "Store_SpliceDynamicData") {
    const key = storeKey(log.args.tableId, log.args.keyTuple);
    const record = store.get(key) ?? {
      staticData: "0x",
      encodedLengths: "0x",
      dynamicData: "0x",
    };

    // Splice the dynamic field data of the Record
    store.set(key, {
      staticData: record.staticData,
      encodedLengths: log.args.encodedLengths,
      dynamicData: bytesSplice(
        record.dynamicData,
        log.args.start,
        log.args.deleteCount,
        log.args.data
      ),
    });
  } else if (log.eventName === "Store_DeleteRecord") {
    const key = storeKey(log.args.tableId, log.args.keyTuple);

    // Delete the whole Record
    store.delete(key);
  }
}

Security Considerations

Access Control

This standard only defines functions to read from the Store (getRecord, getField, and getFieldLength). The methods for setting or modifying records in the store are left to each specific implementation. Therefore, implementations must provide appropriate access control mechanisms for writing to the store, tailored to their specific use cases.

On-Chain Data Accessibility

All data stored within a store is accessible not only off-chain but also on-chain by other smart contracts through the provided read functions (getRecord, getField, and getFieldLength). This differs from the typical behavior of smart contracts, where internal storage variables are private by default and cannot be directly read by other contracts unless explicit getter functions are provided. Thus, developers must be mindful that any data stored in the store is openly accessible to other smart contracts.

Copyright

Copyright and related rights waived via CC0.