Fuel Indexer

Fuel Logo

The Fuel Indexer is a standalone binary that can be used to index various components of the blockchain. These indexable components include blocks, transactions, receipts, and state within a Fuel network, allowing for high-performance read-only access to the blockchain for advanced dApp use-cases.

Events can be indexed by the Fuel Indexer by using WASM modules, as described in the Hello World example.

Getting Started

This section provides an outline regarding how to get started using a Fuel Indexer, including dependency installation, basic usage, as well as examples.

Setup instructions for the Fuel SDK

The Fuel Indexer requires a few components to be installed prior to installing system and application dependencies.

What you will need

git clone https://github.com/FuelLabs/fuel-indexer

System Requirements

There are several system requirements including fuelup, llvm, clang and postgres.

Fuel system dependencies

Getting started with a Fuel indexer requires a single primary dependency from the Fuel ecosystem -- fuelup

  • fuelup installs the Fuel toolchain from Fuel's official release channels, enabling you to easily keep the toolchain updated. For more info, take a look at the fuelup repo.

Installation

To install fuelup

fuelup toolchain install latest

Other system dependencies

Ubuntu/Debian

apt update
apt install -y cmake pkg-config git \
    gcc build-essential clang libclang-dev llvm libpq-dev
DependencyRequired For
cmakeManages the build process in an operating system and in a compiler-independent manner
pkg-configLanguage-agnostic helper tool used when compiling applications and libraries
gitVersion control system
gccCompiler tools required to build various Fuel Indexer crates
clangCompiler tools required to build various Fuel Indexer crates on Unix-like OSes
llvmRequired for building Fuel Indexer crate dependencies
libpq-devSet of library function helping facilitate interaction with the PostgreSQL backend

MacOS

brew update
brew install cmake llvm libpq postgresql
DependencyRequired For
cmakeManages the build process in an operating system and in a compiler-independent manner
llvmCompiler infrastructure for building Fuel Indexer crate dependencies
libqPostgres C API library
postgresqlInstalls the command line console (psql) as well as a PostgreSQL server locally

Arch

pacman -Syu --needed --noconfirm cmake \
    gcc pkgconf git clang llvm11 llvm11-libs postgresql-libs
DependencyRequired For
cmakeManages the build process in an operating system and in a compiler-independent manner
gitVersion control system
gccCompiler tools required to build various Fuel Indexer crates
llvm11Compiler infrastructure for building Fuel Indexer crate dependencies
llvm11-libsCompiler infrastructure libs for building Fuel Indexer crate dependencies
pkgconfSystem for configuring build dependency information
postgresql-libsProvides the essential shared libraries for any PostgreSQL client program or interface
clangCompiler required to build various Fuel Indexer crates Unix-like OSes

Application dependencies

We'll need to install the diesel CLI

cargo install diesel_cli --no-default-features --features "postgres sqlite"

And we'll run the migrations to create our graph registry, types, and columns. Note that this part assumes that you're familiar with basic steps involved in getting a postgres user/role and database setup.

cd fuel-indexer/

DATABASE_URL=postgres://postgres@localhost/your-database \
    bash scripts/run_migrations.local.sh

A Fuel Indexer Project

The Fuel Indexer service can currently be used in two different ways:

  1. The service can either be run as a standalone binary, outside the scope of a Sway project.
  2. The service can be included in a Sway project, as a tandem service.

We'll describe these two different implementations below.

As a standalone service

When running a Fuel Indexer service as a standalone binary, you can just start the service (with or without Docker) after you've defined your manifest file. For an example of this, checkout the Hello World example section.

With a Sway project

The convetion for a Sway project layout including a Fuel Indexer is:

➜  my-project tree . -I target/ -I out/
.
├── contracts
│   └── my-contract
│       ├── Forc.lock
│       ├── Forc.toml
│       ├── out
│       │   └── debug
│       │       ├── my-contract-abi.json
│       │       ├── my-contract-storage_slots.json
│       │       └── my-contract.bin
│       ├── src
│       │   └── main.sw
│       └── tests
│           └── harness.rs
├── frontend
│   └── index.html
└── indexer
    ├── contracts
    │   └── my-contract-abi.json
    ├── my-index.manifest.yaml
    ├── my-indexer
    │   ├── Cargo.toml
    │   └── src
    │       ├── my-index.rs
    │       └── my-second-index.rs
    ├── my-second-index.manifest.yaml
    └── schema
        └── schema.graphql

12 directories, 15 files

Indexer Configuration

Below you will find a list of CLI configuration options that can be used to configure either the Fuel Indexer service, the standalone Fuel Indexer GraphQL API service, or both. For those who prefer using a configuration file, you can checkout the default service configuration file, which also shows the default values used for these configuration options.

Usage:

Using the main Fuel Indexer service binary.

cargo run --bin fuel-indexer -- [options]

Using the standalone GraphQL API server.

cargo run --bin fuel-indexer-api-server -- [options]

Options:

-c --config

  • Path to the configuration file.

-m --manifest

  • Path to manifest file from which initial indices will be loaded

Fuel node: The node running the Fuel client implementation.

--fuel-node-host

  • IP of the Fuel node

--fuel-node-port

  • Port of the Fuel node

GraphQL API: The enpoint at which GraphQL queries will be processed. This is context dependent. If ran using the fuel-indexer binary, these options apply to the GraphQL service run in that binary. If ran using the fuel-indexer-api-server binary, these options will apply to that service.

--graphql-api-host

  • IP at which to bind the GraphQL server

--graphql-api-port

  • Port at which to bind the GraphQL server

  • --run-migrations

  • Whether to run the migrations on the GraphQL API's connected database

Postgres: Standard Postgres connection options.

--postgres-host

  • Postgres host

--postgres-port

  • Postgres port

--postgres-username

  • Postgres username

--postgres-password

  • Postgres password (redacted from logging)

--postgres-database

  • Postgres database

SQLite: An alternative database implementation using standard SQLite connection options

  • --sqlite-database

  • Path to SQLite database

Examples

  • Hello World
    • A simple application representing the "Hello world" version of a Fuel indexer
    • This example includes WASM execution
  • Counter
    • A simple example of a fully-fledged Fuel Indexer project used with a Counter app.

A basic "Hello World" indexer

Write your contract

We're assuming here you have a sway contract written, and you're ready to start indexing. If not you can check out The Sway Book to get started, then come back!

Project structure

We'll start with the project structure:

hello-indexer/
├── .cargo
│   └── config
├── Cargo.toml
├── contracts
│   └── hello_indexer.json
├── manifest.yaml
├── README.md
├── schema
│   └── schema.graphql
└── src
    └── lib.rs
  • The project must compile to WASM, and the way to do that would be to have a .cargo/config file with the following contents:
[build]
target = "wasm32-unknown-unknown"
  • Cargo.toml - the basic dependencies are:
[package]
name = "hello-indexer"
version = "0.1.0"
edition = "2021"
publish = false

[lib]
crate-type = ['cdylib']

[dependencies]
fuel-indexer = "0.1"
fuel-indexer-macros = "0.1"
fuel-tx = "0.9"
fuels = "0.13"
fuels-abigen-macro = "0.13"
fuels-core = "0.13"
getrandom = { version = "0.2", features = ["js"] }
serde = { version = "1.0", default-features = false, features = ["derive"] }
  • Next up is the manifest. This will be configuration for your indexer. For example, what contract addresses do your indexers watch?
---
namespace: hello_namespace
graphql_schema: schema.graphql
module:
  wasm:
    hello_indexer.wasm
handlers:
  - event: an_event_name
    handler: function_one
  • namespace - your graph will live within this namespace in the database, and will be important to remember once you get to querying your data.
  • graphql_schema - we'll have more to say about this in the next section on data types, but this is a file specifying your indexed data types.
  • module - the code you'll be writing to transform your data
  • handlers - this maps event types to the function names in your WASM that should handle the events

Defining your data types

The WASM indexer basically takes two sets of data types, one set coming from your sway contract, and the other defined in the form of a GraphQL schema, specifying how you would like your data to be indexed. The code you'll be writing will just transform the data from your contract types and inserts them into a database.

With that in mind, let's start with a sample contract.json ABI spec:

[
    {
        "type":"contract",
        "inputs":[
            {
                "name": "my_event",
                "type": "struct LogEvent",
                "components":
                [
                    {
                        "name":"contract",
                        "type":"b256"
                    },
                    {
                        "name":"event_id",
                        "type":"u64"
                    },
                    {
                        "name":"count",
                        "type":"u64"
                    }
                ]
            }
        ],
        "name":"takes_struct",
        "outputs":[]
    }
]
  • We have one type here, a LogEvent type, with a few attributes. This json will come from your sway contract, and defines the types you want to manipulate in WASM.

As for the data structures defining your index, let's use this example:

schema {
    query: QueryRoot
}

type QueryRoot {
    event_counts: EventCount
    data2: SomeOtherData
}

type EventCount {
    id: ID!
    account: Address!
    count: UInt8!
}


type SomeOtherData {
    id: ID!
    hash: Bytes32!
}

At a minimum, the schema needs a schema definition with the query type defined. In this example, they are defined at the top of the file.

Now for the code! Fuel indexers use two proc macros to generate the rust types from these specifications.

extern crate alloc;
use fuel_indexer_macros::{graphql_schema, handler};
use fuels_abigen_macro::wasm_abigen;

graphql_schema!("hello_namespace", "schema/schema.graphql");
wasm_abigen!(
    no_name,
    "hello-indexer/contracts/hello_indexer.json"
);

#[handler]
fn function_one(event: LogEvent) {
    Logger::info("Callin' the event handler");
    let LogEvent {
        contract,
        event_id,
        count,
    } = event;

    let mut t1 = match EventCount::load(event_id) {
        Some(t) => t,
        None => EventCount {
            id: event_id,
            account: Address::from(contract),
            count: 0,
        },
    };

    t1.count += count;

    t1.save();
}
  • function_one will take the ABI type LogEvent, unpack it into a few variables. It is attempting to load an EventCount type from the database with the id event_id, if it's not found it will create one. It will increment the count attribute either way and then save it back to the database.
  • graphql_schema translates your graphql schema into rust types, so you can have access to them in your handler code.
  • wasm_abigen will similarly translate the contract ABI types into rust types.
  • handler is another macro that is required for the indexer to be accessible from the indexer service, it is mostly generates some glue code that bridges the interface between native code and the WASM runtime.

Build it

With all this in place, we can now build the indexer:

cargo build --release

Run it

  • We've previously described how to bring up the fuel service. The manifest.yaml described here is the one we promised to get to in that section. You may now run that command to bring up your indexer.
  • Now, you can send in some test transactions, and the data will be available via the API.

Querying the data

With some data in our indexer, we can now query the API endpoint, remembering that our namespace defined in the manifest file is hello_namespace, some example queries will look like this:

curl -s localhost:29987/api/graph/hello_namespace -XPOST -H 'content-type: application/json' -d '{"query": "query { event_counts { id count } }", "params": "b"}'
curl -s localhost:29987/api/graph/hello_namespace -XPOST -H 'content-type: application/json' -d '{"query": "query { event_counts { id account count } }", "params": "b"}'
curl -s localhost:29987/api/graph/hello_namespace -XPOST -H 'content-type: application/json' -d '{"query": "query { event_counts { count } }", "params": "b"}'

With those queries, the response might look something like:

{
  "count": 7,
  "id": 10
}
{
  "account": "0000000000000000000000000000000000000000000000000000000000000000",
  "count": 7,
  "id": 10
}
{
  "count": 7
}

Counter

In this example project we'll walk through how to run a Fuel Indexer service that uses WASM-based execution. We'll go through project structure, how to actually use the service, as well as a few key concepts.

Setup

First we'll walk through the basic setup and usage of the project

➜  tree . -I 'target/'
.
├── README.md
├── contracts
│   └── counter
│       ├── Forc.lock
│       ├── Forc.toml
│       ├── out
│       │   └── debug
│       │       ├── counter-abi.json
│       │       └── counter.bin
│       └── src
│           └── main.sw
├── counter-indexer
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
├── frontend
│   └── index.html
├── manifest.yaml
├── schema
│   └── counter.graphql
└── web-api-and-fuel-node
    ├── Cargo.toml
    └── src
        └── main.rs

11 directories, 13 files

With this recommended project structure you'll notice:

  • contracts is where our Sway smart contract counter is located.
  • web-api-and-fuel-node is a combination of a tiny HTTP web application, and a simple Fuel node, that we use to pass messages to/from the counter contract.
  • frontend is unused (as there is no frontend for this small example)
  • manifest.yaml is the configuration file for the index that we will be using.

Usage

In this section we'll cover the exact steps needed to spin up this example.

Database setup

We'll start by creating a database. This step assumes some familiarity with creating Postgres roles and databases. In this example we're using an indexer database owned by a postgres role without a password.

Next we'll bootstrap our database by running some migrations.

DATABASE_URL=postgres://postgres@127.0.0.1 bash scripts/run_migrations.local.sh

Starting the web server & Fuel node

As previously mentioned, web-api-and-fuel-node contains both a tiny web server used to pass messages to/from our Sway smart contract, as well as our Fuel node that we use to interact with the blockchain. We will start both of these services with the following command.

cd fuel-indexer/examples/counter/web-api-and-fuel-node && cargo run

Start the Fuel Indexer service

With our Fuel node and web server up and running, we'll next start our Fuel Indexer service.

cargo build -p fuel-indexer

./target/debug/fuel-indexer --manifest examples/counter/manifest.yaml

Send a transaction to the smart contract via the web server

curl -X POST http://127.0.0.1:8080/count | json_pp

Verify data was posted to the database

In this example we just created an entity with id = 1

➜  echo "SELECT max(id) FROM counter.count;" | psql -U postgres -d postgres
 max
-----
   1
(1 row)

So that's what we query for

curl -X POST http://127.0.0.1:29987/api/graph/counter -H 'content-type: application/json' -d '{"query": "query { count(id: 1) { id count timestamp } }", "params": "b"}' | json_pp
[
   {
      "count" : 1,
      "id" : 1,
      "timestamp" : 123
   }
]

Hooray! 🎉 we've successfully created our first Fuel Indexer service.

Components

A list of components used by the Fuel Indexer.

Manifest

A manifest serves as the configuration for your indexer and it is written in YAML format. A proper manifest has the following structure:

---
namespace: ...
graphql_schema: ...
module:
  wasm:
    ...
handlers:
  - event: ...
    handler: ...

Namespace

Your graph will live within the namespace set as part of this field. As such, it's important that you remember it when querying the API endpoint for your data. For example, if the field is set to my_special_space, then your queries would look similar to this:

curl -s localhost:29987/api/graph/my_special_space -XPOST -H ...

GraphQL Schema

The graphql_schema field contains the file path to a GraphQL schema. This schema holds the structures of the data that will eventually reside in your database. You can read more about the format of the schema file here.

Module

The module field contains a file path to custom code that will be executed as part of the indexer. There are two available options: wasm and native. If you choose to use WASM, the path must lead to a compiled WASM module. Alternatively, if you choose to use the native option, the path must lead to a module that contains native Rust code. In both cases, the functions included as part of the handlers field should be present in the module.

Handlers

The handlers field maps event types to the names of function that will handle each event. The event should map to an input type that is present in a contract's ABI specification.

GraphQL Server

The fuel-indexer-api-server crate of the Fuel Indexer contains a standalone GraphQL API server that acts as a queryable endpoint on top of the data layer.

Note that the main binary of the fuel-indexer crate also contains a queryable GraphQL API endpoint. However, the fuel-indexer-api-server crate offers a standalone GraphQL API endpoint, whereas the fuel-indexer-api-server bundles its GraphQL API endpoint with other Fuel Indexer functionality (e.g., execution, handling, data-layer contruction, etc).

To run the standalone Fuel Indexer GraphQL API server using a configuration file:

cd fuel-indexer/

RUST_LOG=debug cargo run --bin fuel-indexer-api-server -- --config config.yaml

Where config.yaml is based on the default service configuration file.

GraphQL Schema

The GraphQL schema is a required component of the Fuel Indexer. When data is indexed and placed into the database, it will be done through the use of data structures defined in the schema.

In its most basic form, a schema should have a schema definition that contains a defined query type. The rest of the implementation is up to you. Here's an example of a well-formed schema:

schema {
    query: QueryRoot
}

type QueryRoot {
    thing1: FirstThing
    thing2: SecondThing
}

type FirstThing {
    id: ID!
    value: UInt8!
}

type SecondThing {
    id: ID!
    other_value: UInt8!
    timestamp: Timestamp!
}

You should also familiarize yourself with the information under the Database section in order to ensure that data from your Sway contract is stored in the database as intended.

Database

The Fuel Indexer uses Postgres as the database on the data layer.

  • Types
    • How to use different data types from your Sway contract, all the way to your Postgres table
  • Conventions
    • Some of the conventions used in the Fuel Indexer's data layer
  • Foreign Keys
    • How foreign keys are handled in GraphQL schema, Postgres, and SQLite
  • Directives
    • How GraphQL schema directives are translated into data-layer constraints

Types

Below is a mapping of GraphQL schema types to their Postgres equivalents, referencing Postgres 14 data types.

Sway TypeGraphQL Schema TypePostgres Type
u64IDbigint primary key
b256Addressvarchar(64)
str[4]Bytes4varchar(16)
str[8]Bytes8varchar(64)
str[32]Bytes32varchar(64)
str[32]AssetIdvarchar(64)
b256ContractIdvarchar(64)
str[32]Saltvarchar(64)
u32Int4integer
u64Int8bigint
u64Timestamptimestamp
str[]Blobbytes

Example

Let's define an Event struct in a Sway contract:

struct Event {
    address: Address,
    id: u64,
    block_height: u64,
}

Our GraphQL schema should resemble:

type Event {
    id: ID!
    account: Address! @indexed
    block_height: Int8! @indexed
}

This will generate the following Postgres schema:

                                           Table "schema.event"
  Column   |     Type    | Collation | Nullable | Default | Storage  | Compression | Stats target | Description 
-----------+-------------+-----------+----------+---------+----------+-------------+--------------+-------------
 id        |    bigint   |           | not null |         | plain    |             |              | 
 count     |    bigint   |           | not null |         | plain    |             |              | 
 address   | varchar(64) |           | not null |         | plain    |             |              | 
 object    |    bytea    |           | not null |         | extended |             |              | 
Indexes:
    "count_pkey" PRIMARY KEY, btree (id)
Access method: heap

The source code for these types can be found here.

Conventions

IDs

IDs are often included in Sway smart contracts as a means of identifying an entity or object. Giving an indexable entity an ID, allows you to easily find the entity after it's been indexed, using the GraphQL API included with the Fuel Indexer.

With regard to the Fuel Indexer, the ID data type in GraphQL schema maps to a bigint primary key in Postgres. Given this, ID fields in GraphQL schema should be paired with u64 types in a Sway contract.

  • Note that a Sway contract developer can always use another data type (e.g., str[32]) as a type of identifier field. However, you should reference the Fuel Indexer types table when deciding which data types to use in certain scenarios.

One important thing to note is that when it comes to indexing data, the developer of the Sway smart contract is responsible for creating IDs. The Fuel Indexer does not use any type of auto-incrementing ID mechanism.

  • This is important because if a Fuel Indexer operator is expecting IDs to be (for example) globally unique, then the indexer operator would have to ensure that the Sway contract generating the indexable events has some mechanism to generate those unique IDs.

Foreign Keys

The Fuel Indexer service supports Foreign Key constraints and relationships using a combination of GraphQL schema and a data layer (whether Postgres or SQLite). Below you'll find a simple example of how to use Foreign Keys.

IMPORTANT: At the moment, due to some SQLite quirks, Fuel Indexer SQLite only supports foreign key relationships, not foreign key constraints. We are very much open to changing this in the future.

To demonstrate how the Indexer uses GraphQL schema to resolve foreign key relationships, given the following schema:

schema {
    query: QueryRoot
}

type QueryRoot {
    book: Book
    library: Library
}

type Book {
    id: ID!
    name: Bytes8!
}

type Library {
    id: ID!
    book: Book!
}

Two entities will be created: a Book entity, and a Library entity. As you can see, we add the Book entity as an attribute on the Library entity, thus conveying that we want a one-to-many or one-to-one relationship between Library and Book. This means that for a given Library, we may also fetch one or many Book entities.

Directives

Per GraphQL:

A directive is an identifier preceded by a @ character, optionally followed by a list of named arguments, which can appear after almost any form of syntax in the GraphQL query or schema languages.

As of now the Fuel Indexer supports a single directive: @indexed.

Using our Library and Book example again, given the following schema:

schema {
    query: QueryRoot
}

type QueryRoot {
    book: Book
    library: Library
}

type Book {
    id: ID!
    name: Bytes8! @indexed
}

type Library {
    id: ID!
    book: Book!
}

A single BTREE INDEX constraint will be created on the book table's name column.

IMPORTANT: At the moment, index constraint support is limited to BTREE on Postgres with ON DELETE, and ON UPDATE actions not being supported. Note that @indexed directives are also available using SQLite. Finally, multi-column indices are not supported at the moment.