Factories with Billions of Addresses

Previously, HyperIndex began to struggle when handling 8 million addresses. With this release, we have removed this limitation—now the number of supported addresses is virtually unlimited, constrained only by your available resources.

We achieved this by making HyperIndex automatically switch from server-side to client-side filtering, which drastically reduces the number of queries sent. Additionally, a partial Rust rewrite optimizes resource usage during event processing.

#1483, #1490, #1493

Deferred Postgres Index Creation

This change led to a 2.5x backfill speedup for some users:

Previously, Postgres indexes defined in schema.graphql were created upfront, which could cause significant write backpressure. In this version, indexes are created just before entering realtime indexing after the backfill is complete. Additionally, when a handler makes a getWhere call, HyperIndex will automatically generate an index for the required field—so you no longer need to define these indexes in schema.graphql. Now, indexes defined in schema.graphql are specifically for your production GraphQL server

#1506

Bottleneck Observability

We've expanded our Prometheus metrics suite to more precisely identify where bottlenecks occur:

• envio_processing_stalled_on_fetch_seconds — Time spent idle with an empty buffer, waiting for events to be fetched
• envio_processing_stalled_on_storage_write_seconds — Time spent idle due to write backpressure

Combined with the existing counters, these metrics provide a comprehensive view of indexer performance. For more details, refer to the Production Observability guide: https://docs.envio.dev/docs/HyperIndex/observability.

We've also added several utility metrics to make each metrics scrape self-describing—useful for monitoring tools and AI agents:

• envio_process_start_time_seconds — Indexer run start time (now surfaced at the top; otherwise unchanged)
• envio_process_metric_time_seconds — Timestamp when the metrics snapshot was generated
• envio_process_elapsed_seconds — Elapsed seconds since start

The elapsed metric allows you to easily interpret counter values from a single payload—no need for rate() or an external clock. For example, envio_processing_stalled_on_storage_write_seconds / envio_process_elapsed_seconds yields the fraction of the run spent stalled on writes.

#1486

Numeric (Int/BigInt) Entity IDs

You can now use Int or BigInt as the id type for entities in schema.graphql. Relationship fields will be correctly inferred, and handler types will be generated as number.

type Chain {
  id: Int! # Numeric entity ID
  vaults: [Vault!]! @derivedFrom(field: "chain")
}

type Vault {
  id: ID!
  chain: Chain! # Stored as chain_id (Int)
  // ...
}

#1482

Support Chain IDs Above 2^31

Added for a Tron Testnet user and beneficial for other chains as well.

If your config includes a chain with an ID greater than 2^31, HyperIndex now automatically uses larger database types to support these values.

#1507

Other Changes

• Fixed handlers for entities with lowercase name #1479
• Upgrade TypeScript target to ES2023 for envio init templates #1497
• Skip ClickHouse-only entities on reorg rollback. Note: This might lead to invalid data, but at least there's not crash. Use mixed Postgres and ClickHouse entities for now for reliable data on reorgs. #1514
• Suggest viem transport batching in external-calls skill #1520

Internal Improvements

• Consolidated TypeScript API type tests #1480
• Improved end-to-end config parsing tests #1484
• Added end-to-end user-facing indexer tests from config and source strings #1488
• Preparations to generalize the internal test indexer #1509
• Document testing strategy and development workflow #1515

Unlimited onEvent Handlers

Previously, we allowed only a single onEvent handler per event definition, making it impossible to have different handlers with different filters for the same event definition. Since 3.4.0 there's no such restriction.

import { indexer } from "envio";

// Track all ERC20 Transfers
indexer.onEvent(
  { contract: "ERC20", event: "Transfer", wildcard: true },
  async ({ event, context }) => {},
);

// And a separate handler for USDC-only is also possible
indexer.onEvent(
  { contract: "ERC20", event: "Transfer", wildcard: true, where: {
      params: [
        { from: USDC_ADDRESS },
        { to: USDC_ADDRESS },
      ],
    },
  },
  async ({ event, context }) => {},
);

// And contractRegister can have a different `where` filter for the same event
indexer.contractRegister(
  { contract: "ERC20", event: "Transfer", wildcard: false /* non-wildcard for contract register */ },
  async ({ event, context }) => {},
);

Note: It might result in a single log being processed by different handlers and calculated as multiple events in the metric. The same log execution is guaranteed to follow handler registration order inside a file.

Per-entity ClickHouse Tuning

Perfect for making your analytic queries faster. Reach out to us for ClickHouse support on Envio Cloud

type Transfer @storage(clickhouse: {
  partitionBy: "toYYYYMM(timestamp)",
  orderBy: ["timestamp"],
  ttl: "timestamp + INTERVAL 2 YEAR"
}) {
  id: ID!
  timestamp: Timestamp!
}

Test Indexer Got 12x Faster

We got many reports that our testing framework got slower after the v3 upgrade. We fixed it, and now it's as fast as before with all the new features v3 brought.

Check out the Testing Guide if your agents still don't use our testing framework for indexer development.

Note: The change made test indexer instances run in-process, which might result in shared state pollution in some edge cases.

Internal Changes

• Removed Handlebars dependency #1468
• Internal test suite is better and better #1470

Improvements for Huge Multichain & Factory Contract Indexers

Faster backfill, lower memory usage, better stability, and lower latency at the head.

#1321, #1330, #1331, #1332, #1334, #1335, #1298, #1336, #1337, #1338, #1341, #1340, #1342, #1347, #1339, #1349, #1355, #1358, #1361, #1360, #1359, #1364, #1377, #1380, #1379, #1369, #1394, #1392, #1395, #1399, #1400, #1401, #1403, #1406, #1404, #1413, #1414, #1415, #1423, #1427, #1426, #1430, #1429, #1434, #1392, #1447

Per-Chain Effect API

Effect API now supports per-chain caching and rate limiting via a new crossChain option. This controls whether an effect's input is cached and rate-limited globally or separately per chain. Default: true.

• crossChain: true (default): Uses a single shared cache for all chains. Recommended for chain-agnostic calls (e.g. token metadata, price by symbol) where the result does not depend on the chain.
• crossChain: false: Maintains a separate cache and rate-limit budget for each chain. context.chain.id becomes available in the effect handler. Use when the result is chain-specific, such as on-chain reads or per-chain RPCs; the same input will be fetched once per chain.

// Chain-specific result → crossChain: false, access context.chain.id
const getBalance = createEffect(
  {
    name: "getBalance",
    input: S.string,
    output: S.bigint,
    rateLimit: false,
    crossChain: false,
  },
  async ({ input: account, context }) =>
    rpcFor(context.chain.id).getBalance(account),
);

Cross-chain effects can only call other cross-chain effects, as they do not have a specific chain context; chain-scoped effects (crossChain: false) can call either type. Accessing context.chain in a cross-chain effect will throw an error.

Be aware that per-chain effects have different cache structure and will require a cache reset/migration when switching the crossChain option.

#1432

Nested Env Vars Interpolation In Config

Environment variable interpolation now supports nested fallback values in defaults (e.g., ${ENVIO_RPC_1:-https://my.rpc?defaultToken=${ENVIO_RPC_DEFAULT_TOKEN}}).

#1367

Custom HTTP Headers For RPC

chains:
  - id: 1
    rpc:
      - url: https://eth-mainnet.your-rpc-provider.com
        for: sync
        headers:
          authorization: "Bearer ${ENVIO_RPC_1_TOKEN}"

#1363

RPC Source Enhancements

We improved RPC source to support cases previously only possible with HyperSync:

• Event handlers using or union for params filters in where
• Indexers with multiple wildcard events

#1368

We also improved how RPC Source handles provider's response-too-large errors #1371, #1372

Fixes

• Critical fix for rollback handling for deleted entities #1431
• Critical fix for an edge case with incorrect rollback on reorg #1444
• Fix serialization of zero-parameter events in raw_events #1356
• Fix experimental SVM account_filters config parsing #1408
• Fix SVM to exclude failed transaction instructions #1428

Other Improvements

• Internal cleanup and refactoring of the codebase #1346, #1351, #1365, #1370, #1390, #1396, #1402, #1416, #1421, #1424
• Improve DX in highlighting missing field_selection config #1333
• Improve logging #1348, #1425
• Validate TestIndexer simulate srcAddress against wildcard registrations #1357
• Report dead TestIndexer simulate items instead of silently dropping them #1362
• TestIndexer automatically normalizes address format #1374
• Update handler API skills to reflect new syntax #1366
• Added a skill to guide through Envio Cloud CLI
• Automatically announce releases to Discord and Telegram #1393, #1398
• Add Robinhood as a known chain #1436

Filter by multiple fields with getWhere

await context.Account.getWhere({
  id: {
    _eq: "0x123...",
  },
  balance: {
    _gte: 1_000_000n,
    _lte: 10_000_000n,
  },
});

Additionally, we noticeably improved the performance of getWhere with a single _eq or _in filter by optimizing the number of round trips to the database.

Multi-storage improvements

Now it's possible to mark storages as default to automatically assign an entity to the storage. It allows you not to set @storage attribute for every entity in schema.graphql and only use it as a default overwrite.

storage:
  postgres:
    default: true
  clickhouse:
    default: true

Column Name Format

Configure automatic conversion for column names in your database to snake_case. It only affects the underlying database, so GraphQL and handler types will preserve the original names derived from schema.graphql.

storage:
  postgres:
    column_name_format: snake_case

More Experimental Solana Features

We added support for HyperSync-powered instruction handlers. Reach out to us if you're interested in becoming an early tester.

Critical Bug Fixes

• Fix critical bugs causing an indexer crash introduced by v3.1

• Flush in-flight batch writes before computing rollback diffs #1305
• Duplicate history rows from concurrent batch writes #1312

• Handle missing transaction data as retryable RPC errors - previously it would incorrectly disable the source #1304
• Make config path root-relative in start command #1296
• Validate duplicate addresses in config at parse time #1311

Internal Work

• Chore, refactoring and other fun things #1290

2x Cheaper And 2.5x Faster

We are yet to reach our full potential. And the new HyperIndex release comes with a nice improvement: it requires up to 2x fewer HyperSync queries for backfill and is 2.5x faster in many indexing cases.

Descriptions On Entities, Fields, And Relationships

Describe your entities, fields, and relationships in the schema.graphql file, so that they are exposed in the GraphQL API.

# Hash comments are ignored by the GraphQL parser and should NOT appear
# in introspection. Only string descriptions ("""...""" or "...") are exposed.

"An on-chain account participating in transfers"
type Account {
  "The account's Ethereum address"
  id: ID!
  "All transfers sent from this account"
  outgoing: [Transfer!]! @derivedFrom(field: "from")
}

Improve Agentic Indexer Development Experience

• Add envio tools search-docs <query> subcommand #1237
• Add envio tools fetch-docs <url> subcommand #1237
• Add envio metrics runtime subcommand #1281
• Clean up skills provided by envio init and envio skills update #1234

Rate-Limit Info In TUI And Logs

Improve HyperSync rate-limit handling by showing rate-limit information in TUI and logs.

Skip Chains From Indexing

Excludes the chain from indexing and migrations by adding the skip field to the chain in the config.yaml file.

This is an advanced feature. For testing, prefer a test framework instead.

chains:
  - id: 1
    skip: true

Fix Indexer Startup With 4.5M+ Contracts

Other Fixes And Improvements

• Remove unnecessary height polling when buffer reached the head, but events are still unprocessed #1230
• Remove cargo-cache action from publishing workflow to avoid potential exploitation #1231
• Remove ordered multichain internal implementation #1233
• Remove HyperSync chain tier tracking #1236
• Improve HyperIndex contributing experience #1251, #1253
• Simplify ERC20 template by removing unnecessary toString() calls #1252
• Persist all entity history changes to ClickHouse (note: this is internal implementation details and will change in the future) #1257
• Fix DDOS from height polling triggered by stale SSE connections #1271, #1272

HyperIndex V3 Is Here

It's production-ready, fast, and reliable!

We had an exciting journey with new features and improvements, and now that the five months of alpha releases are over, we are back to following SemVer. There are a number of breaking changes in the V3 release, but they will allow us to accelerate HyperIndex development and bring you even more features, performance, and reliability improvements.

Check out:

• What's New In V3
• Migrate To V3

What's next?

We have big plans for HyperIndex to make it the ultimate tool for interacting with blockchain data, and you can help us build it: create GitHub Issues, reach out to us with feedback, and support us with a GitHub Star and a kind word to a friend.

As for the most recent roadmap we have:

• Address your V3 feedback
• Isolated Multichain Mode
• Polish Solana Support
• Indexing 1,000,000+ events per second

Effect API - Bye-Bye Experimental Prefix

Effect API, released on May 8, served us well, and we officially removed the experimental_ prefix from createEffect.

This change comes with two new features:

• The rateLimit option (required) which allows you to set explicitely how freaquently the effect might be called. Use either false to disable, or provide an object with custom calls per duration. Duration might be second, minute or a number in milliseconds.
• Ability to overwrite cache option for specific call. Set context.cache = false to prevent caching the result which failed.

export const getMetadata = createEffect(
 {
   name: "getMetadata",
   input: S.string,
   output: S.optional(S.schema({
     description: S.string,
     value: S.bigint,
   })),
   // Protect your API from burst Effect calls
   rateLimit: {
     calls: 5,
     per: "second"
   },
   cache: true,
 },
 async ({ input, context }) => {
   try {
     const response = await fetch(`https://api.example.com/metadata/${input}`);
     const data = await response.json();
     return {
       description: data.description,
       value: data.value,
     };
   } catch(_) {
     // Don't cache failed response
     context.cache = false
     return undefined;
   }
 }
);

Development Console Insights

With this version, the Development Console now provides more performance metrics for Effect API execution, as well as loading entities from your handlers.

See a detailed time overview to identify what's slowing down your processing time. Use Batched metric to find what can be better optimized with Preload Optimization. And get useful insigts about your indexer execution in general. As a nice bonus, the Development Console now works even with TUI disabled.

Tip: Hover over question marks to get better understanding of the data.

Access Development Console at envio.dev/console

New getWhere.lt Query

Use context.<Entity>.getWhere.<FieldName>.lt to get all entities where the field value is lower than the given value.

Internal Improvements

• Librarify config and stop using global db client #803
• Update Cursor rules #806
• Librarify InMemoryTable code #807

Big Reliability Release

Rollback On Reorg Refactoring

We completely rewrote our rollback on reorg logic to make it more robust, predictable, and performant. The update introduces numerous new tests and fixes for all previously identified indexing issues:

• Fixes all known indexing and rollback on reorg issues
• Optimizes database writing logic, which reduces latency by dozens of milliseconds
• Reduces internal tables' size for managing reorg and rollback
• The Events Processed counter became reorg-resistant and can be used to verify data consistency
• Enables more amazing features

Nice Additions

• SubGraph migration cursor rules to init template #778
• Ability to get chains readiness status from context.chains #775
• Extended supported entity name length up to 63 characters

Other Fixes and Internal Changes

• Fixes https://github.com/enviodev/hyperindex/issues/786
• Fixes https://github.com/enviodev/hyperindex/issues/788
• Fixes https://github.com/enviodev/hyperindex/issues/749
• Fixes running multiple indexers with different schemas on a single database
• Adds Effect cache load Prometheus metrics
• Removes redundant processing Prometheus metrics and adds envio_progress_batches_count
• Fixes https://github.com/enviodev/hyperindex/discussions/751 - started validating cached effects when loading them from storage
• Improves batch processing log to be less messy
• Prevents stale context access after the handler finishes execution #791
• Changed the structure of the internal history tables to reference only checkpoint_id. Also replaced suffix _history with prefix envio_history_
• Replaced internal end_of_block_range_scanned_data table with envio_checkpoints
• Now history is saved per block instead of per event log
• All block events are always guaranteed to be processed in a single batch

New Contributors

• @Segfaultd made their first contribution in #775

Address Format Configuration

Choose between checksum (default) and lowercase addresses.

The newly added lowercase format can facilitate migration from SubGraphs and improve performance in some instances.

Faster Event Decoder for RPC source

RPC source now uses the HyperSync event decoder, which is multiple times faster than the Viem one.

Fixes

• Regression of 2.29, which broke its indexing at the head Prometheus metric. #769
• Race condition during Hasura configuration, which could lead to some GraphQL entities not having read permissions. #770

Internal changes

• Guarantee to include all block events in a single batch. Previously, it was possible to have events belonging to a single block processed in separate batches. #753
• Persist dynamically registered addresses only for processed events. #752
• Made the JS logic for batch creation more performant. This is relevant for indexers processing more than 100k events per second.

These changes are preparations for a bigger rollback on reorg refactoring.

Block Handlers

Run logic on every block or at an interval!

This is useful for aggregations, time-series logic, and bulk updates using raw SQL. To get started, import the onBlock function from the generated module and call it in one of your handler files.

import { onBlock } from "generated";

onBlock(
  {
    name: "MyBlockHandler",
    chain: 1,
    interval: 10,
    startBlock: 10_000_000,
  },
  async ({ block, context }) => {
    context.log.info(`Processing block ${block.number}`);
  }
);

Read a dedicated documentation page to learn more about the feature and powerful use cases it enables:

• Time intervals
• Preset handler
• Multichain mode
• Different intervals for historical and real-time sync

For inspiration, check out the Example Indexer, which uses all the latest Envio features - Block handlers with different intervals, Effect API, and Traces indexing to get all contracts deployed on Mainnet.

Other changes

• Removed internal db_write_timestamp field, which was automatically set for every entity.

New official _meta query

Returns indexing metadata per chain, which helps track the indexer's progress.

2X faster historical sync with RPC data source

Improved block range selection for logs query, halving the number of needed requests for some RPC providers.

Cheaper historical sync with HyperSync data source

Optimized response from HyperSync to make queries simpler, faster, and have smaller ingress.

Much faster indexing for big factories

Added a performance optimization for indexers with a large number of addresses (>500k). It reduced the indexing time of an indexer with 2 million addresses from 4 days to 2.

SubGraph migration Cursor rules cheatsheet

Find a Cursor rule example in our repo to help you effortlessly migrate your existing SubGraph to HyperIndex!

Other improvements

• Started tracking a real progress block number, which made recovery on restart more efficient
• The events processed counter became more reliable on restarts
• Removed cache from .gitignore for initial templates
• Improved start up time

Potential breaking changes

The release initiates a significant refactoring of internal tables, focusing on a single public entry point - _meta. To achieve this, we hide all internal tables from Hasura, as well as change the internal representation of some of them. These are: chain_metadata, event_sync_state, persisted_state, end_of_block_range_scanned_data, and dynamic_contract_registry. Please let us know if this change affected you and we'll figure out a solution

New Contributors

• @keenbeen32 made their first contribution in #725

Introducing Preload Optimization

Important! Preload optimization makes your handlers run twice. Starting from envio@2.27 all new indexers are created with preload optimization pre-configured by default.

This optimization enables HyperIndex to efficiently preload entities used by handlers through batched database queries, while ensuring events are processed synchronously in their original order. When combined with the Effect API for external calls, this feature delivers performance improvements of multiple orders of magnitude compared to other indexing solutions.

Set a single line in your config and make your handlers multiple times faster without changing a single line of code:

preload_handlers: true

For power users - This is like Loaders, but for all handlers by default.

Read more about Preload Optimization in the dedicated guide.

Contract-specific start block

A highly requested feature contributed by one of our users @rori4

Specify a later start block for a contract to optimize indexing performance:

name: nft-indexer
description: NFT Factory
networks:
  - id: 1337
    start_block: 0
    contracts:
      - name: NftFactory
        address: 0x4675a6B115329294e0518A2B7cC12B70987895C4
        handler: src/EventHandlers.ts
        events:
          - event: SimpleNftCreated(string name, string symbol, uint256 maxSupply, address contractAddress)

      - name: Nft
        # No address field - we'll discover these addresses from SimpleNftCreated events
        start_block: 
        handler: src/EventHandlers.ts
        start_block: 30000000 # Overwrite the network start block
        events:
          - event: Transfer(address from, address to, uint256 tokenId)

In the Factory Contract example above, it'll register all NFT addresses, but it'll start processing Transfers only from the block 30000000.

Implemented in #669, #670.

Contributing Improvements

To support new contributors, we have updated our CONTRIBUTING.md file with a detailed guide on navigating the HyperIndex codebase, as well as examples of what a change would look like. To make it even easier, we added .cursor rules to help develop new HyperIndex features.

Embrace Vibe-Coding

All new projects are now created with an initial .cursor rules included to help you build indexers with the agent support.

Also, feel free to create PRs with the rules suggestions to improve the experience for everyone

Fixes & Improvements

• Started including an address in a handler error log #658
• Support nullable effect cache output #665
• Fix e2e tests #666
• Error when contract start block is earlier than the network's #670

New Contributors

• @rori4 made their first contribution in #669

Built-in Cache for Effect Calls

import { experimental_createEffect, S } from "envio";

export const getMetadata = experimental_createEffect(
  {
    name: "getMetadata",
    input: S.string,
    output: {
      description: S.string,
      value: S.bigint,
    },
    cache: true, // Simply set cache to true
  },
  async ({ input, context }) => {}
})

Read more in the docs, learn how to persist the cache on reruns, and share it with Hosted Service (alpha).

Other changes

• Now it's possible to disable Hasura integration via ENVIO_HASURA=false environment variable.
• Started using Postgres 17 for local development
• Added ENVIO_INDEXER_PORT env var (Instead of deprecated METRICS_PORT)
• Added ENVIO_PG_PASSWORD env var (Instead of deprecated ENVIO_POSTGRES_PASSWORD)
• Removed uninformative text from TUI and included the GraphQL endpoint instead.
• Fixed indexers with non-public schema

New Features

• Add context.log support from the contractRegister handler in #624

Fixes

• Use correct config path for integration tests #615
• Fix processEvents return type #625
• Fix slowdown during historical sync because of tracking reorgs too early #626

• Add output directory path to config.yaml #610
• Fix envio_progress_block_number Prometheus metric to increase when there are no events to process #613
• Add mockDb.processEvents for testing batches of events #611
• Fix insert values logic (entities with array fields) #614
• Just clean up, no changes #609

• Expose context.set, context.unsafeDelete, and context.getOrCreate in loaders. Add context.isPreload to distinguish between first and second loader run #600
• Reusable and faster Read and Write PG interactions #594

Big Clean-up Release

Add context.Entity.getOrCreate and context.Entity.getOrThrow API

// Before:
// let pool = await context.Pool.get(poolId);
// if (!pool) {
//  pool = {
//    id: poolId,
//    totalValueLockedETH: 0n
//  }
//  context.Pool.set(pool);
// }
const pool = await context.Pool.getOrCreate({
  id: poolId,
  totalValueLockedETH: 0n
})

// Before:
// const pool = await context.Pool.get(poolId);
// if (!pool) {
//  throw new Error(`Pool with ID ${poolId} is expected.`)
// }
const pool = await context.Pool.getOrThrow(poolId)
// Will throw: Entity 'Pool' with ID '...' is expected to exist.
// Or you can pass a custom message as a second argument:
const pool = await context.Pool.getOrThrow(poolId, `Pool with ID ${poolId} is expected.`)

These are additional helpers for DX improvements. You can access them from both handlers and loaders.

Loaders Consistency

Loaders are a powerful feature to optimise indexer performance (Docs). The loader function is run twice: the first time in parallel for all events in the batch, and the second time immediately before the handler execution, to obtain up-to-date data. This part is unchanged, but we've done several improvements here:

• If the loader function fails on the first run, the error will be silently ignored. It may happen, depending on the entity, which will only be available on the second run. We don't want to abort indexing here, because it's a valid case.
• The HyperIndex test framework runs the loaders twice to match the actual indexer logic.

Clever Batch Creation for Unordered Multichain Mode

Previously, even with Unordered Multichain Mode, we chose events to be batched from all available chains in the order of events on-chain. This is not bad, but for significant indexers relying on loader optimisation, it's better to have all events for a single chain in the batch. This way, there is a higher chance of achieving deduplication or batching optimization.

This is what we've done in the version. Now, we try to create a processing batch using events from one chain as much as possible, and then rotate to another chain for another batch.

Flexible Project Structure

To make HyperIndex run correctly, we used to have several requirements for the user's HyperIndex project, which was confusing and limited flexibility:

• It required pnpm-workspaces.yaml file
• It required .npmrc file with shamefully hoisting dependencies
• It required to have the start script in your package.json with ts-node generated/src/Index.bs.js

Now, none of this is necessary, and you're free to remove them. You can also remove ts-node from dependencies, just replace ts-node generated/src/Index.bs.js with envio start.

Breaking Changes

Unfortunately, it required changing .bs.js suffix to .res.js in our internal code. So if you, for some reason, relied on some internal fiels with .bs.js extention, you need to change them to .res.js manually. Also, for ReScript users, it's now required to configure .res.js suffix in your rescript.json file.

Aggregate Queries Configuration

Note: Aggregate Queries might have a severe hit on Database performance. We generally recommend aggregating data using entities during indexing.

Now it's possible to enable Hasura Aggregate Queries using the ENVIO_HASURA_PUBLIC_AGGREGATE environment variable. You can set environment variables on our Hosted Service on Production plans.

Other Changes

• Fixed test framework failing to set an entity with Json field the second time #565
• Generated and exposed types for loaderArgs and handlerArgs per event #540
• Remove unused scenarios folders #563
• Librarify the entity config type #554
• Librarify persistence layer #558
• Add Prometheus metric to count Effect calls in #582

Async Contract Registrer

Starting from version 2.21, you can use async contract registration.

This is a unique feature of Envio that allows you to perform an external call to determine the address of the contract to register.

NftFactory.SimpleNftCreated.contractRegister(async ({ event, context }) => {
  const version = await getContractVersion(event.params.contractAddress);
  if (version === "v2") {
    context.addSimpleNftV2(event.params.contractAddress);
  } else {
    context.addSimpleNft(event.params.contractAddress);
  }
});

Support for accessList and EIP7702 authorizationList fields in EVM transaction

Json type support for entity fields in graphql.schema file

Example:

type User {
  id: ID!
  greetings: [String!]!
  latestGreeting: String!
  numberOfGreetings: Int!
+ metadata: Json!
}

Additional improvements

• Improve tests for the output of generated SQL query #525
• Add more metrics and logs to the indexer #526

Effect API

The new Effect API is a convenient way to perform external calls from your handlers. It's especially powerful when used with loaders:

• It automatically batches calls of the same kind
• It memoizes calls, so you don't need to worry about the loader function being called multiple times
• It deduplicates calls with the same arguments so that you won't overfetch
• And we're working on adding support for automatic retrying of failed requests as well as persisting results to use on indexer reruns

It allows you to parallelize thousands of external calls from the processing batch, instead of executing them one by one.

Read more in the dedicated Loaders guide.

Contract Registration Boost

We made indexing with contract registrations much faster. It got so fast in the default mode that we disabled the preRegisterDynamicContracts option.

Handle 9 more RPC provider errors

For the RPC data source, when we perform requests for event logs, different RPC providers might have different errors, with suggestions to use a different block range for retry. In the release, we added support for 9 more RPC providers to improve our RPC sync retry logic.

Fix CLI to init the indexer in a non-interactive way

Fix the --single-contract and --all-events flags for contract import from explorer #514

We started experimenting with Envio MCP, and creating an indexer using a CLI is definitely a fantastic feature.

Performance section in Dev Console

Demo in Loom

Other changes

• Add Prometheus metrics #507
• Better handle block range error from Alchemy #510
• Fix field selection for RPC source #509
• Fix integration tests #512

Development Console

Track the indexer's progress and access the GraphQL Playground at the Development Console, with more features coming!

Logger improvements

Added the ability to pass params on a log call:

context.log.info("Sucessfully handled Transfer()", { from: event.params.from, to: event.params.to })

These params will be displayed in the logs in your terminal as well as Hosted Service.

You can also pass an error:

} catch (error) {
  context.log.error("Failed ipfs call", error)
}

• Event filtering by contract addresses #495
• Sync chain list #500

Topic Filtering goes Multichain

Now the eventFilters option can also accept a callback, allowing for building different filters depending on chainId:

import { ERC20 } from "generated";

const ZERO_ADDRESS = "0x0000000000000000000000000000000000000000";

const WHITELISTED_ADDRESSES = {
  1: [
    "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266",
    "0x70997970C51812dc3A010C7d01b50e0d17dc79C8",
  ],
  100: ["0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC"],
};

ERC20.Transfer.handler(
  async ({ event, context }) => {
    //... your handler logic
  },
  {
    wildcard: true,
    eventFilters: ({ chainId }) => [
      { from: ZERO_ADDRESS, to: WHITELISTED_ADDRESSES[chainId] },
      { from: WHITELISTED_ADDRESSES[chainId], to: ZERO_ADDRESS },
    ],
  }
);

Stricter chainId type

The chainId type on event is now a union of chain ids the event belongs to. This is much safer than a number type used before.

Polishing

• Pass HyperSync api token to JSON api calls #493
• Update HyperSync chain list #494
• Add more links to the README #488

Enhanced Reliability with RPC Data Sources

You can now specify RPC providers for redundancy and failover, ensuring 100% uptime for your indexer. If HyperSync becomes unavailable, your indexer will automatically switch to an RPC provider.

To enable this, add an rpc field to your network configuration. It can be a URL or a list of RPC configuration objects:

networks:
  - id: 137 # Polygon
    start_block: 0
+   rpc: https://eth-mainnet.your-rpc-provider.com?API_KEY={ENVIO_MAINNET_API_KEY}
    contracts:
      - name: PolygonGreeter
        address: 0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c

The fallback RPC is triggered if the primary data source does not receive a new block within 20 seconds.

New RPC Configuration Syntax

Adding an RPC provider is now as simple as setting a single URL:

networks:
  - id: 137 # Polygon
    start_block: 0
+   rpc: https://eth-mainnet.your-rpc-provider.com?API_KEY={ENVIO_MAINNET_API_KEY}
    contracts:
      - name: PolygonGreeter
        address: 0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c

• If HyperSync is available for a chain, it remains the primary data source, with RPC as a fallback.
• For chains without HyperSync support, RPC becomes the primary data source.

Advanced Configuration

Gain more control over your RPC setup by specifying it explicitly as a primary source:

networks:
  - id: 137 # Polygon
    start_block: 0
+   rpc:
+     url: https://eth-mainnet.your-rpc-provider.com?API_KEY={ENVIO_MAINNET_API_KEY}
+     for: sync
    contracts:
      - name: PolygonGreeter
        address: 0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c

Or define multiple RPC endpoints with custom settings:

networks:
  - id: 137 # Polygon
    start_block: 0
+   rpc:
+     - url: https://eth-mainnet.your-rpc-provider.com?API_KEY={ENVIO_MAINNET_API_KEY}
+       for: fallback
+     - url: https://eth-mainnet.your-free-rpc-provider.com
+       for: fallback
+       initial_block_interval: 1000
    contracts:
      - name: PolygonGreeter
        address: 0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c

#437 and #467.

Bug Fixes

• Improved historical sync performance by fixing a critical issue related to reorg detection mode. #470 (@DZakh)
• Fixed raw events that had empty parameters. #460 (@DZakh)
• Fixed Docker Compose file for self-hosting, ensuring Hasura connects correctly to Postgres when using a custom user or database. #459 (@JasoonS)
• Added support for nested ABIs in contract imports. #472 (@diyahir, @DZakh)

Other Changes

• Defaulted unordered_multichain_mode to true for new projects. #462 (@manusoman)
• Cleaned up the project README.md. #461 (@DZakh)
• Fixed all Clippy warnings. #463 (@JonoPrest)
• Improved wording for multi-chain indexer errors. #471 (@DZakh)

New Contributors

• @diyahir made their first contribution in #472.
• @manusoman made their first contribution in #462.

• Allow to customise database schema with ENVIO_PG_PUBLIC_SCHEMA environment variable #445, #457
• Add CLA #452
• Update stale readme #454
• Fix events with empty params #456
• Enable rollback on reorg in initial templates #458

New Contributors

• @pavlovdog made their first contribution in #445

RPC source feature-parity with HyperSync

• Wildcard indexing support with RPC data source #415
• Support Event Filtering for RPC data-source applied to a single wildcard event #420
• Add Pre Registration support for RPC data-source #421
• Add support for RPC input & value transaction fields #427
• Improve RPC retries on block range error #426

New loader getWhere operator

• Add getWhere Gt operator #413

Fuel source improvements

• Support Pre Registration for Fuel #422

Other improvements & fixes

• Optimise queries with Field Selection per Event for wildcard #419
• Don't query events without handlers #425
• Fix reorg detection metadata clean up for single chain indexers #429, #430

Field Selection per Event

You can specify field selection for individual events. This feature is useful for optimizing RPC and HyperSync calls by fetching only the data relevant to specific events, avoiding over-fetching.

Per-event field selection overrides any global field selection, ensuring precise data handling for each event.

events:
  - event: "Transfer(address indexed from, address indexed to, uint256 value)"
    field_selection:
      transaction_fields:
        - "to"
        - "from"

• Added support for the from, to, gasPrice, maxPriorityFeePerGas, maxFeePerGas, contractAddress transaction fields with the RPC data-source #371

Internal

• Allow to freely refactor the internal config type #368
• Mark Tangle as Bronze HyperSync network #374

Environment Variables in the config file

The Envio config file can now use Environment Variables to offer more flexibility. If you want to switch between different configurations quickly, you don't need to edit the config file each time; you can just set Environment Variables at run time.

For the Hosted Service users, it's now possible to set custom Environment Variables

Below is a simple example:

networks:
  - id: ${ENVIO_CHAIN_ID:-137}
    start_block: ${ENVIO_START_BLOCK:-45336336}
    contracts:
      - name: Greeter
        address: ${ENVIO_GREETER_ADDRESSES}

Then you can run ENVIO_GREETER_ADDRESSES=0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c pnpm dev, or set the values via the .env or the indexer settings page on the Hosted Service.

Interpolation syntax

For the interpolation to be applied, the Environment Variable name should be placed in braces after a dollar sign (${VAR}).

For braced expressions, the following formats are supported:

Direct substitution

• ${VAR} -> value of VAR

Default value

• ${VAR:-default} -> value of VAR if set and non-empty, otherwise default
• ${VAR-default} -> value of VAR if set, otherwise default

Other Improvements

• Fix to allow naming entities in schema.graphql as Schema, Belt, EntityHistory, InternalEntity, Entity, Utils #353
• Display HyperSync tiers for networks on init #362
• Don't wait for the new block when there are partitions we still can query #364
• Remove current height from the endBlock for HyperSync/HyperFuel queries to improve latency #364
• Added deduplication for new block polling #364
• Internal: Start working on the custom RPC client #356

A complete rework of rollbacks and entity history

This slims down the amount of data saved significantly for historical entity updates and improves db performance for large scale indexers.

• [1] Entity History Rework #310
• [2] Add eval function for inserting entity history #319
• [3] Handle Setting Entity History in IO #325
• [4] Rollback diff #333
• [5] Remove old entity_history code and add history pruning #338
• [6] Implement dynamic contract registry as entity #340
• [7] Add rollback tests #342
• Add rollback benchmarks #345
• Fix hasRows check for entity history #348
• Add benchmarks to prometheus #346

Fixes

• Update fuel-ts up to 0.96.1 #334
• Mark MevCommit as HyperSync #335
• Remove broken unused package.json script #337
• Remove debug loggers #339
• Add ignore from tests to mev commit #343
• Fix npm files to include genType #344
• More flexible package files #347

New Features and Improvements

• Add more benchmarks and improve framework #299
• Support Fuel std::string::String type #307
• Sync hypersync chain list +14 new chains #308
• Use double precision for float representation #302

Fixes

• Fix sorting of composite index fields #301
• Fix skipped handler on empty contract register event case #304
• Add additional checks to prevent double contract registration #309

Fixes

• Fix querying a toBlock below fromBlock on RPC #274
• Fixes for contract registration registers and partitions #281
• Fix using onlyBelowReorgThreshold when rollback on reorg is disabled #293

Improvements

• feat: prompt for start block on rpc url networks in contract import #4
• Improve Contract Import to stop asking for API keys #279
• Improve crash error #280
• Add Mainnet selection in Fuel contract import #291

Internal

• feat: ts fuel all events deployment scripts for e2e testing #249
• remove fuel ts codegened files #276
• Update prometheus events processed on restart #284

New Feature

Dynamic Contract Pre-registration

You can now add "preRegisterDynamicContracts" flag to your event config. For all the events that have this config, there will be an end to end indexer run just for these events to run all the relevant contractRegister functions and collect all the dynamic contract addresses. The indexer then restarts with all these addresses from the start block configured for the network (rather than from the block that the contract gets registered). For a standard factory contract setup this drastically reduces indexing time since it doesn't result in many small block range queries but rather larger grouped queries.

PoolFactory.CreatePool.contractRegister(
  ({ event, context }) => {
    context.addPool(event.params.pool);
  },
  { preRegisterDynamicContracts: true },
);

What's Changed

• Dynamic Contract Pre Registration #252
• Dynamic Contract Preregistration improvements #259

New Fuel receipt types support

Previously, you could only index LOG_DATA receipts. The release supports indexing MINT, BURN, TRANSFER, TRANSFER_OUT, and CALL receipt types.

Read more on the Fuel documentation page.

Customize decimal precision of BigInt and BigDecimal fields

When working with large integers or high-precision decimal numbers in your application, you might need to customize the precision and scale of your BigInt and BigDecimal fields. This ensures that your database stores these numbers accurately according to your specific requirements. If you know your numbers will not be too big, you can also optimize the database by not over-allocating on the precision.

Example:

type Product {
  id: ID!
  price: BigDecimal @numeric(precision: 10, scale: 2)
}

In this example, the price field can store numbers with up to 10 digits in total, with 2 digits after the decimal point (e.g., 99999999.99).

Read more on the Schema documentation page.

Fixes & Improvements

• Drastically improve historical sync performance with Reorg handling enabled #220, #235
• Update npm package README.md #232

Internal work

• Benchmark Improvements #218, #242
• Remove unused to_postgres_type function from codegen #224
• Preparations for V2 Hosted Service: Add Prometheus Metric for synced at head and update pnpm install options #230
• Remove TS and Eslint dev dependencies from the Envio package #233
• Start moving code from generated to the Envio package #234, #238
• Preparations for the Fuel Mainnet launch #237
• Clean cache guide #236

Wildcard Indexing ⁂

Index the whole chain by event signature without specifying exact contract addresses.

Here's an example of indexing all ERC20 Transfer events. No need to change the config file. Simply add the wildcard: true option to the second argument in your handler:

ERC20.Transfer.handler(
  async ({ event, context }) => {},
  {
    wildcard: true,
  }
)

Final touches implemented #171, #197

Event Filtering

Filter events by indexed parameters. Simplify your handler's logic or get new indexing possibilities when joined with the Wildcard Indexing feature:

UniswapV3Factory.PoolCreated.handler(async ({ event, context }) => {},
  {
    wildcard: true,
    eventFilters: [{ token0: DAI_ADDRESS }, { token1: DAI_ADDRESS }],
  },
);

Both Wildcard Indexing and Event Filters are currently only supported with HyperSync. We are working on adding support for RPC mode in the following versions.

Implemented

Fuel Merge

The fuel ecosystem indexer is integrated into the main repository code. This was a huge refactoring which came with a big list of benefits and a big list of opportunities.

If you use envio@x.x.x-fuel version, you can switch to envio@2.3.0 and get:

• Now Fuel and Evm reuse the same code. This gives V2 API, much better test coverage, and many improvements/fixes compared to envio@x.x.x-fuel
• Support for raw_events configuration
• Support for Wildcard Indexing
• Public repository
• Active development and maintenance

Talking about opportunities, this unblocks:

• Adding support for Mint/Burn/Call/TransferIn/TransferOut events
• Indexer became more modular and library-like, simplifying the integration of new ecosystems in the future

Implemented #186, #190, #192, #193, #194, #196, #202

Other

• Add validation for end block on finite chains #191
• Upgrade Hasura to latest v2 release #169
• Use the local envio package for testing instead of defaulting to latest #200
• Fix panic on null exn passed to error handling #203

Features

New Timestamp scalar type available in schema.graphql. It's represented as an instance of Date, making your handlers and queries more flexible, fast, and explicit.

Fixes & Polishing

• Update Viem version to the latest V2 to fix type conflicts with the Viem version on user-side #159

Be careful if you're using Viem V1 in your indexer handlers. This change might cause unexpected behavior.

• Fix incorrect import of a type, which caused tsc --build to fail. Also, prevent this from happening in the future by adding a TypeScript type check to CI pipeline #165
• Fix codegen with HyperSync/RPC endpoint in the config having a trailing slash #164
• Update the supported chains list by adding new chains and removing deprecated ones #163
• Remove dependency on Ethers.Interface #155

Other

• Working on Wildcard Indexing feature implementation #135, #151, #160, #161
• Working on Fuel merge #155
• Add script to help update the chain list #162

Features

• Allow custom names for events #132 This is useful when you want to have a different name on the Envio side; or index events with the same name but different signatures:

Fixes & Polishing

• Fix progress bar for chains with 5000+ registered contract addresses #124
• Fix "Maximum call stack size exceeded" error in reorgs logic #125
• Memory and performance improvements #127
• Update schema documentation on rollback_on_reorg to say that it's defaulted to true #131
• Init Fuel indexer with the envio@2.1.5-fuel version which fixes dynamic contracts #126
• Improve error message for invalid contract and event names #133

• V2 #35
• BigInt clean up #48
• Enable Uncurried mode #49
• Support event param names as reserved rescript keywords #51
• Uncurried mode for Fuel #52
• Generate json schema for config.yaml #56
• Update rescript schema to V7 #57
• Add commands for updating schemas and help docs #58
• Move codegen from Config.res file #59
• Clean up how we codegen Config #60
• Implement partitioning of fetch state #55
• Implement Field Selection #61
• Support nested abi field #64
• Improve missing global contract error message #65
• Fix integration tests #66
• Move Converters to static codegen #67
• Treat rc as valid envio version #68
• Update fuel version #69
• Backup rpc urls #70
• Add contract import for amoy #73
• Lookup events in config #72
• Fix tests #75
• Use rescript-schema V8 #74
• Improve failure message of contract import unhandled array types #77
• Add generic ChainWorker module #78
• Opt-in raw events #80
• Remove rescript_type duplication #81
• Fix integration tests after raw events removal #84
• Use fuel 2.1.2 #85
• Implement hack for detecting invalid api key and prompting #83
• Add id validation for entity parsing #86
• Increase stallTimeout for rpc #76
• Remove codegen rescript format #89
• Improve logging in a few places so more context is shared with logs such as chainId #90
• Remove event_type #91
• Implement updated bindings for hs client #88
• Initial implementation of whereEq queries #79
• V2 Implement restart filter for partially processed block #93
• Add back shouldExitOnFailure variable #95
• Add support for dotenv #94
• Remove rescript core #96
• chore: add open licensing to repository #45
• Perf optimisations #98
• Update npm links to point to correct repo #100
• Change 'docker-compose' to 'docker compose' #101
• [1] Prompt for API key on init #97
• Make should_rollback_on_reorg default to true #104
• [2] Add TUI messages #103
• Refactor LoadLayer #99
• Evaluate rows on addEmptyIndex #105
• Use 2.1.4-fuel #107
• Add api token to template #108
• Add Fuji network to chain list #102
• Update schema descriptions #109
• Remove footer from CLI Help #110
• Preserve ordering of schema for docs #111
• Rename unstable sync config #112

Changelog

• Added No-code Quickstart to index the Fuel Network. Learn how easy it is to build a Fuel indexer with the updated tutorial https://docs.envio.dev/docs/tutorial-indexing-fuel
• Support contract import from the Avalanche network
• Made the indexer generated by the init command simpler for new Envio builders
• Internal refactoring to prepare for making the HyperIndex repository public

Internal Changelog

• Prepare cli for Fuel merge to support contract import #24
• Test workflow using Personal Access Token on new repo #27
• Upgrade hardhat to fix tests #33
• Limit pg connections and implement async task queue #29
• Add Fuel to Human config #30
• [V1] Remove EventSummary from init template #34
• V1 - Do not rely on api keys for contract verifycation but keep them as a fallback #37
• Dz/fuel contract import 5 #43

• Improve retry logging using error handler #2
• [1r] DB migrations refactor #9
• [2r] Hasura tracking refactor #16
• Add fuel greeter template #1
• Reintroduce db_write_timestamp wield to avoid breaking changes #17
• fix: use celoscan for contract import #19
• Split init by ecosystem #18
• Add a BigDecimal type that can be used in the indexer #15

New Contributors

• @DenhamPreen made their first contribution in #19
• @DZakh made their first contribution in #18