EIP3668 - CCIP Read: Secure offchain data retrieval

# Simple Summary

CCIP Read provides a mechanism to allow a contract to fetch external data.

# Abstract

Contracts wishing to support lookup of data from external sources may, instead of returning the data directly, revert using OffchainLookup(address sender, string[] urls, bytes callData, bytes4 callbackFunction, bytes extraData). Clients supporting this specification then make an RPC call to a URL from urls, supplying callData, and getting back an opaque byte string response. Finally, clients call the function specified by callbackFunction on the contract, providing response and extraData. The contract can then decode and verify the returned data using an implementation-specific method.

This mechanism allows for offchain lookups of data in a way that is transparent to clients, and allows contract authors to implement whatever validation is necessary; in many cases this can be provided without any additional trust assumptions over and above those required if data is stored onchain.

# Motivation

Minimising storage and transaction costs on Ethereum has driven contract authors to adopt a variety of techniques for moving data offchain, including hashing, recursive hashing (eg Merkle Trees/Tries) and L2 solutions. While each solution has unique constraints and parameters, they all share in common the fact that enough information is stored onchain to validate the externally stored data when required.

Thus far, applications have tended to devise bespoke solutions rather than trying to define a universal standard. This is practical - although inefficient - when a single offchain data storage solution suffices, but rapidly becomes impractical in a system where multiple end-users may wish to make use of different data storage and availability solutions based on what suits their needs.

By defining a common specification allowing smart contract to fetch data from offchain, we facilitate writing clients that are entirely agnostic to the storage solution being used, which enables new applications that can operate without knowing about the underlying storage details of the contracts they interact with.

Examples of this include:

Interacting with 'airdrop' contracts that store a list of recipients offchain in a merkle trie.
Viewing token information for tokens stored on an L2 solution as if they were native L1 tokens.
Allowing delegation of data such as ENS domains to various L2 solutions, without requiring clients to support each solution individually.
Allowing contracts to proactively request external data to complete a call, without requiring the caller to be aware of the details of that data.

# Specification

# Overview

Answering a query via CCIP read takes place in three steps:

Querying the contract.
Querying the gateway using the URL provided in (1).
Querying or sending a transaction to the contract using the data from (1) and (2).

In step 1, a standard blockchain call operation is made to the contract. The contract reverts with an error that specifies the data to complete the call can be found offchain, and provides the url to a service that can provide the answer, along with additional contextual information required for the call in step (3).

In step 2, the client calls the gateway service with the callData from the revert message in step (1). The gateway responds with an answer response, whose content is opaque to the client.

In step 3, the client calls the original contract, supplying the response from step (2) and the extraData returned by the contract in step (1). The contract decodes the provided data and uses it to validate the response and act on it - by returning information to the client or by making changes in a transaction. The contract could also revert with a new error to initiate another lookup, in which case the protocol starts again at step 1.

┌──────┐                                          ┌────────┐ ┌─────────────┐
│Client│                                          │Contract│ │Gateway @ url│
└──┬───┘                                          └───┬────┘ └──────┬──────┘
   │                                                  │             │
   │ somefunc(...)                                    │             │
   ├─────────────────────────────────────────────────►│             │
   │                                                  │             │
   │ revert OffchainData(sender, url, callData,       │             │
   │                     callbackFunction, extraData) │             │
   │◄─────────────────────────────────────────────────┤             │
   │                                                  │             │
   │ RPC (sender, callData)                           │             │
   ├──────────────────────────────────────────────────┼────────────►│
   │                                                  │             │
   │ Response (result)                                │             │
   │◄─────────────────────────────────────────────────┼─────────────┤
   │                                                  │             │
   │ callbackFunction(result, extraData)              │             │
   ├─────────────────────────────────────────────────►│             │
   │                                                  │             │
   │ answer                                           │             │
   │◄─────────────────────────────────────────────────┤             │
   │                                                  │             │

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Contract interface

A CCIP read enabled contract MUST revert with the following error whenever a function that requires offchain data is called:

error OffchainLookup(address sender, string[] urls, bytes callData, bytes4 callbackFunction, bytes extraData)

sender is the address of the contract that raised the error, and is used to determine if the error was thrown by the contract the client called, or 'bubbled up' from a nested call.

urls specifies a list of URLs to services (known as gateways) that implement the CCIP read protocol and can formulate an answer to the query. urls can be the empty list [], in which case the client MUST specify the URL. The order in which URLs are tried is up to the client, but contracts SHOULD return them in order of priority, with the most important entry first.

callData specifies the data to call the gateway with. This value is opaque to the client. Typically this will be ABI-encoded, but this is an implementation detail that contracts and gateways can standardise on as desired.

callbackFunction is the 4-byte function selector for a function on the original contract to which a callback should be sent.

extraData is additional data that is required by the callback, and MUST be retained by the client and provided unmodified to the callback function. This value is opaque to the client.

The contract MUST also implement a callback method for decoding and validating the data returned by the gateway. The name of this method is implementation-specific, but it MUST have the signature (bytes response, bytes extraData), and MUST have the same return type as the function that reverted with OffchainLookup.

If the client successfully calls the gateway, the callback function specified in the OffchainLookup error will be invoked by the client, with response set to the value returned by the gateway, and extraData set to the value returned in the contract's OffchainLookup error. The contract MAY initiate another CCIP read lookup in this callback, though authors should bear in mind that the limits on number of recursive invocations will vary from client to client.

In a call context (as opposed to a transaction), the return data from this call will be returned to the user as if it was returned by the function that was originally invoked.

# Example

Suppose a contract has the following method:

function balanceOf(address addr) public view returns(uint balance);

Data for these queries is stored offchain in some kind of hashed data structure, the details of which are not important for this example. The contract author wants the gateway to fetch the proof information for this query and call the following function with it:

function balanceOfWithProof(bytes calldata response, bytes calldata extraData) public view returns(uint balance);

One example of a valid implementation of balanceOf would thus be:

function balanceOf(address addr) public view returns(uint balance) {
    revert OffchainLookup(
        address(this),
        [url],
        abi.encodeWithSelector(Gateway.getSignedBalance.selector, addr),
        ContractName.balanceOfWithProof.selector,
        abi.encode(addr)
    );
}

1
2
3
4
5
6
7
8
9

Note that in this example the contract is returning addr in both callData and extraData, because it is required both by the gateway (in order to look up the data) and the callback function (in order to verify it). The contract cannot simply pass it to the gateway and rely on it being returned in the response, as this would give the gateway an opportunity to respond with an answer to a different query than the one that was initially issued.

# Gateway Interface

The URLs returned by a contract may be of any schema, but this specification only defines how clients should handle HTTPS URLs. Compliant gateways MUST accept POST requests with a Content-Type of application/json, matching the following schema:

{
    id: string,
    data: {
        to: string,
        data: string
    }
}

1
2
3
4
5
6
7

Where:

id - string - A request ID, which will be returned in the response.
to - string - The Ethereum address of the contract that reverted with OffchainLookup, in '0x' prefixed hex format.
data - string - The value of the callData argument from the OffchainLookup error, in '0x' prefixed hex format.

Compliant gateways MUST respond with a Content-Type of application/json, matching the following schema:

{
    jobRunID: string,
    statusCode: integer,
    data: {
        result: string
    },
    error: {
        name: string,
        message: string
    }
}

1
2
3
4
5
6
7
8
9
10
11

Where:

jobRunID - string - The request ID from the original request.
statusCode - number - A status code, matching the semantics in RFC 2616 §6.1.1 (opens new window). 200 indicates a successful response.
data - object - Required if the status code is between 200 and 299, forbidden otherwise. Contains the response data.
- result - string - The response data from the call to the gateway, in '0x' prefixed hex format.
error - object - Required if the status code is between 400 and 599, forbidden otherwise. Contains the error information.
- name - string - A short human-readable message giving the nature of the error.
- message - string - A human-readable message describing the error.

# Example

// Request
curl -X POST --data '{"id": "1","data":{to:"0x226159d592E2b063810a10Ebf6dcbADA94Ed68b8", data: "0xd5fa2b00...", }}"}'

// Successful result
{
    "jobRunID": 1,
    "statusCode": 200,
    "data": {
        "result": "0x12345678..."
    }
}

// Error result
{
    "jobRunID": 1,
    "statusCode": 400,
    "error": {
        "name": "InvalidArgument",
        "message": "Could not decode input data"
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

# Client Lookup Protocol

A client that supports CCIP read MUST make contract calls using the following process:

Set data to the call data to supply to the contract, and to to the address of the contract to call.
Call the contract at address to function normally, supplying data as the input data. If the function returns a successful result, return it to the caller and stop.
If the function returns an error other than OffchainLookup, return it to the caller in the usual fashion.
Otherwise, decode the sender, urls, callData, callbackFunction and extraData arguments from the OffchainLookup error.
If the sender field does not match the address of the contract that was called, return an error to the caller and stop.
Make an HTTP POST request to one of the URLs specified in urls. The client may choose which URLs to try in which order, but SHOULD prioritise URLs earlier in the list over those later in the list. POST requests are formatted as specified in Gateway Interface.
If the response code from step (5) is in the range 400-499, return an error to the caller and stop.
If the response code from step (5) is in the range 500-599, go back to step (5) and pick a different URL, or stop if there are no further URLs to try.
Otherwise, replace data with an ABI-encoded call to the contract function specified by the 4-byte selector callbackFunction, supplying data.result from step (6) and extraData from step (4), and return to step (1).

This protocol can result in multiple lookups being requested by the same contract. Clients MUST implement a limit on the number of lookups they permit for a single contract call, and this limit SHOULD be at least 4.

The lookup protocol for a client is described with the following pseudocode:

async function rpccall(urls, to, callData) {
    for(const url of urls) {
        const response = await fetch(
            url,
            {
                method: "POST",
                "headers": {"Content-Type":"application/json"},
                "body": JSON.stringify({
                    "id": "1",
                    "data": {
                        "address": to,
                        "data": callData
                    }
                })
            }
        );
        const result = await response.json();
        if(result.statusCode >= 400 && result.statusCode <= 499) {
            throw new Error(data.error.message);
        }
        if(result.statusCode >= 200 && result.statusCode <= 299) {
            return result;
        }
    }
}
async function durin_call(provider, to, data) {
    for(let i = 0; i < 4; i++) {
        try {
            return await provider.call(to, data);
        } catch(error) {
            if(error.code !== "CALL_EXCEPTION") {
                throw(error);
            }
            const {sender, urls, callData, callbackFunction, extraData} = error.data;
            if(sender !== to) {
                throw new Error("Cannot handle OffchainLookup raised inside nested call");
            }
            const result = rpccall(urls, to, callData);
            data = abi.encodeWithSelector(callbackFunction, result.result, extraData);
        }
    }
    throw new Error("Too many CCIP read redirects");
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

Where:

provider is a provider object that facilitates Ethereum blockchain function calls.
to is the address of the contract to call.
data is the call data for the contract.

If the function being called is a standard contract function, the process terminates after the original call, returning the same result as for a regular call. Otherwise, a gateway from urls is called with the callData returned by the OffchainLookup error, and is expected to return a valid response. The response and the extraData are then passed to the specified callback function. This process can be repeated if the callback function returns another OffchainLookup error.

# Use of CCIP read for transactions

While the specification above is for read-only contract calls (eg, eth_call), it is simple to use this method for sending transactions (eg, eth_sendTransaction or eth_sendRawTransaction) that require offchain data. While 'preflighting' a transaction using eth_estimateGas or eth_call, a client that receives an OffchainLookup revert can follow the procedure described above in Client lookup protocol, substituting a transaction for the call in the last step. This functionality is ideal for applications such as making onchain claims supported by offchain proof data.

# Glossary

Client: A process, such as JavaScript executing in a web browser, or a backend service, that wishes to query a blockchain for data. The client understands how to fetch data using CCIP read.
Contract: A smart contract existing on Ethereum or another blockchain.
Gateway: A service that answers application-specific CCIP read queries, usually over HTTPS.

# Rationale

# Use of `revert` to convey call information

For offchain data lookup to function as desired, clients must either have some way to know that a function depends on this specification for functionality - such as a specifier in the ABI for the function - or else there must be a way for the contract to signal to the client that data needs to be fetched from elsewhere.

While specifying the call type in the ABI is a possible solution, this makes retrofitting existing interfaces to support offchain data awkward, and either results in contracts with the same name and arguments as the original specification, but with different return data - which will cause decoding errors for clients that do not expect this - or duplicating every function that needs support for offchain data with a different name (eg, balanceOf -> offchainBalanceOf). Neither solutions is particularly satisfactory.

Using a revert, and conveying the required information in the revert data, allows any function to be retrofitted to support lookups via CCIP read so long as the client understands the specification, and so facilitates translation of existing specifications to use offchain data.

# Passing contract address to the gateway service

address is passed to the gateway in order to facilitate the writing of generic gateways, thus reducing the burden on contract authors to provide their own gateway implementations. Supplying address allows the gateway to perform lookups to the original contract for information needed to assist with resolution, making it possible to operate one gateway for any number of contracts implementing the same interface.

# Existence of `extraData` argument

extraData allows the original contract function to pass information to a subsequent invocation. Since contracts are not persistent, without this data a contract has no state from the previous invocation. Aside from allowing arbitrary contextual information to be propagated between the two calls, this also allows the contract to verify that the query the gateway answered is in fact the one the contract originally requested.

# Use of JSON for the gateway interface

JSON is widely used as a low-overhead way of making RPC calls over HTTP. We have used it here in order to minimise cognitive and tooling overhead for developers.

# Backwards Compatibility

Existing contracts that do not wish to use this specification are unaffected. Clients can add support for CCIP read to all contract calls without introducing any new overhead or incompatibilities.

Contracts that require CCIP read will not function in conjunction with clients that do not implement this specification. Attempts to call these contracts from non-compliant clients will result in the contract throwing an exception that is propagaged to the user.

# Security Considerations

In order to prevent a malicious gateway from causing unintended side-effects or faulty results, contracts MUST include sufficient information in the extraData argument to allow them to verify the relevance and validity of the gateway's response. For example, if the contract is requesting information based on an address supplied to the original call, it MUST include that address in the extraData so that the callback can verify the gateway is not providing the answer to a different query.

Contracts must also implement sufficient validation of the data returned by the gateway to ensure it is valid. The validation required is application-specific and cannot be specified on a global basis. Examples would include verifying a Merkle proof of inclusion for an L2 or other Merkleized state, or verifying a signature by a trusted signer over the response data.

Client libraries implementing CCIP read may want to ensure that callers have to explicitly enable its use on a contract, so users are not surprised when their client library begins making HTTPS calls to external services.

# Simple Summary

# Abstract

# Motivation

# Specification

# Overview

# Contract interface

# Example

# Gateway Interface

# Example

# Client Lookup Protocol

# Use of CCIP read for transactions

# Glossary

# Rationale

# Use of revert to convey call information

# Passing contract address to the gateway service

# Existence of extraData argument

# Use of JSON for the gateway interface

# Backwards Compatibility

# Security Considerations

# Copyright

# Use of `revert` to convey call information

# Existence of `extraData` argument