Skip to content

Introspect/address ns#47

Open
bengineer42 wants to merge 24 commits intomainfrom
introspect/address-ns
Open

Introspect/address ns#47
bengineer42 wants to merge 24 commits intomainfrom
introspect/address-ns

Conversation

@bengineer42
Copy link
Copy Markdown
Collaborator

No description provided.

MartianGreed and others added 17 commits March 23, 2026 16:07
- Updated schema handling to use `Rc<str>` instead of `Rc<PgSchema>` for better memory efficiency.
- Introduced `MultiContractEventIterator` to handle multiple event sources concurrently.
- Modified table queries to include an owner field and adjusted related queries accordingly.
- Improved error handling and logging during table initialization and event processing.
- Cleaned up unused code and optimized imports across various modules.
- Enhanced the `PgTable` struct to support new table creation and upgrades more effectively.
- Updated tests and examples to reflect changes in event handling and schema management.
- Moved SQL-related functionality from `torii-common` to a new crate `torii-sql`.
- Updated imports in various modules to reflect the new crate structure.
- Removed the `db.rs` file from the `postgres` crate and refactored migration logic into `torii-sql`.
- Introduced a new `DbConnection` trait for common database operations.
- Implemented migration handling in `torii-sql` for both PostgreSQL and SQLite.
- Updated Cargo.toml files to reflect the new dependencies and workspace structure.
- Cleaned up unused code and dependencies across the affected crates.
- Added `SqliteBackend` struct and its implementation for `IntrospectBackend`.
- Introduced `SqliteTable` and related error handling for table operations.
- Created utility functions for generating SQL queries, including `create_table_query`.
- Implemented JSON serialization for SQLite using `SqliteJsonSerializer`.
- Added runtime support for handling introspection messages in SQLite.
- Established a new module structure for SQLite backend, including processor, query, and table management.
- Integrated metrics tracking for processed introspection envelopes.
- Enhanced error handling and type conversions for database interactions.
- Changed the `Executable` trait to return a `BoxFuture` instead of using `async_trait`, improving performance and flexibility.
- Introduced `FlexStr` and `FlexQuery` types to handle SQL strings and bind arguments more effectively.
- Implemented SQLite migration functionality in the `AcquiredSchema` struct, including methods for managing migration tables and applying migrations.
- Updated SQLite connection options and added utility functions for memory database detection.
- Added custom types for SQLx integration, including encoding and decoding for `SqlFelt`.
- Refactored dependencies in `Cargo.toml` files to streamline SQLx feature usage.
- Removed unnecessary `Arc` wrapping for database connection pools in various services.
…ields; update related queries and data handling to accommodate new types. Add functionality to retrieve all extractor states for a given extractor type in EngineDb. Enhance EventExtractor to refresh dynamic contract states from the database. Enable Gzip compression for gRPC services and reflection endpoints in multiple examples and the main library.
…nhance table management and state persistence
- Introduced `DbPoolOptions` and `PoolConfig` for better database connection management.
- Updated `DbPool` to support both PostgreSQL and SQLite connections.
- Refactored `create_table`, `update_table`, and `insert_fields` methods to use the new connection structure.
- Enhanced error handling and logging in database operations.
- Updated event processing logic in the introspect example to utilize the new database connection methods.
- Removed deprecated code and improved type safety with `DbType` enum.
- Added tests to ensure compatibility with both PostgreSQL and SQLite backends.
- Added `torii-sql` as a dependency in multiple Cargo.toml files.
- Refactored database backend handling to use `DbType` and `DbUrl` enums for better clarity and type safety.
- Updated various modules to accommodate the new database handling logic, including `EntitiesHistoricalSink`, `IntrospectQueryMaker`, and others.
- Removed the `introspect_simple` example as it was deemed unnecessary.
- Improved error handling and logging in database operations.
- Adjusted SQL migration scripts for better compatibility with both Postgres and SQLite.
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 31, 2026

📊 Synthetic Benchmark Results

Comparing against baseline: 20260404T103542Z

Performance Summary

Token Type Duration (ms) Blocks Transactions Events Envelopes Blocks/sec Events/sec
erc20 2870 100 50000 50000 50000 34.83 17416.60
erc721 11710 100 50000 50000 50000 8.54 4269.81
erc1155 11296 100 50000 50000 70000 8.85 4426.22

Comparison with Baseline

Token Type Metric Baseline Current Change
erc20 Throughput (events/sec) 17416.60 17416.60 ➡️ 0.0%
erc20 Duration (ms) 2870 2870 ➡️ 0.0%
erc721 Throughput (events/sec) 4269.81 4269.81 ➡️ 0.0%
erc721 Duration (ms) 11710 11710 ➡️ 0.0%
erc1155 Throughput (events/sec) 4426.22 4426.22 ➡️ 0.0%
erc1155 Duration (ms) 11296 11296 ➡️ 0.0%

Legend:

  • ✅ Improvement (faster)
  • ⚠️ Regression (slower)
  • 📈 Throughput increase
  • 📉 Throughput decrease
  • ➡️ No significant change (<5%)

Stage Latency Breakdown

Token Type Extract (ms) Decode (ms) Sink (ms) Commit (ms)
erc20 12.239803000000004 55.982181999999995 2710.170144999999 0.023379000000000004
erc721 12.326449000000006 52.444719000000006 11564.910014000001 0.039382999999999994
erc1155 23.035798000000003 89.70296099999997 11099.089663999996 0.02917899999999999

Generated by Synthetic Benchmark CI

- Removed the `torii-entities-historical-sink` crate, consolidating its functionality into the main application.
- Deleted the `restart copy.rs` example and updated the `restart.rs` example to improve event processing logic.
- Enhanced the `run_events` function to handle `IntrospectBody` messages with `CreateTable` and set `append_only` to true.
- Increased the batch size for event processing from 1000 to 2000 for improved performance.
- Updated error handling during the initialization of the introspect SQL sink to ensure proper logging of errors.
…erialization

feat: implement conversions between StarkNet types and local Felt/Event types

feat: enhance ControllersStore with batch upsert functionality for controllers

fix: improve error handling and ensure proper transaction management in ControllersStore

test: add tests for batch upsert functionality in ControllersStore

feat: add SQLite memory URL detection for database initialization

feat: implement pathfinder example for fetching emitted events

build: create release script for SDK with npm publishing and GitHub release

refactor: add start block configuration to extractors for better control over extraction ranges
refactor(decoder): improve error handling in event data decoding
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants