Blazing.Json.Queryable

Overview

Blazing.Json.Queryable is a high-performance LINQ provider for JSON data that processes JSON directly without full deserialization. Unlike traditional approaches that load entire JSON files into memory, convert to objects, and then apply LINQ queries, this library processes JSON as a stream, providing dramatic performance improvements and memory efficiency for medium to large JSON files.

This custom JSON LINQ provider supports standard string, UTF-8, streaming, and RFC 9535 compliant JSONPath operations powered by Blazing.Json.JSONPath. Whether you're working with multi-megabyte API responses, large data exports, or log files, Blazing.Json.Queryable enables you to query JSON data efficiently with the familiar LINQ syntax you already know.

Key Features

Direct JSON Processing: Query JSON without full deserialization
Memory Efficient: Process files larger than available RAM
Early Termination: Stop reading after finding required results (Take, First, Any)
Streaming Support: Native IAsyncEnumerable<T> for real-time processing
UTF-8 Native: Zero-allocation UTF-8 processing with Span<T>
RFC 9535 JSONPath: Full RFC 9535 compliance with filters, functions, and slicing via Blazing.Json.JSONPath
Full LINQ: Comprehensive LINQ method support (60+ operations)
.NET 10 Async LINQ: Built-in async LINQ with async predicates and transformations
Multiple Input Formats: String, Stream, UTF-8 bytes, and more

Why Choose Blazing.Json.Queryable?

Performance Advantages Over Traditional JSON + LINQ

When working with medium to large JSON files, the traditional approach of deserializing to objects and then querying has significant disadvantages:

Traditional Approach:

// Traditional: Load ALL data into memory, then query
var json = File.ReadAllText("large-file.json");          // Load entire file
var all = JsonSerializer.Deserialize<List<Person>>(json); // Deserialize ALL records
var results = all.Where(p => p.Age > 25).Take(10).ToList(); // Query in-memory

Blazing.Json.Queryable Approach:

// Blazing.Json.Queryable: Stream and query simultaneously
await using var stream = File.OpenRead("large-file.json");
var results = await JsonQueryable<Person>.FromStream(stream)
    .Where(p => p.Age > 25)
    .Take(10)
    .AsAsyncEnumerable()
    .ToListAsync(); // Only deserializes matching records!

Key Advantages

Feature	Traditional	Blazing.Json.Queryable
Memory Usage	Loads entire file	Constant memory usage
Early Exit	Processes all records	Stops after Take(N)
Large Files	OutOfMemoryException risk	Handles files > RAM
Speed (Large + Take)	Slow (full deserialize)	10-20x faster
Async Support	Manual implementation	Native `IAsyncEnumerable`
UTF-8 Processing	String conversion overhead	Zero-allocation Span<T>

When to Use Each Approach

Use Blazing.Json.Queryable When:

Files are > 10 MB
You need early termination (Take, First, Any)
Working in memory-constrained environments
Processing files larger than available RAM
Building async/streaming APIs
Need real-time data processing

Use Traditional JsonSerializer + LINQ When:

Files are < 1 MB
You need all data (no early termination)
Simplicity is the priority
Working with in-memory collections

Getting Started

Installation

Install via NuGet Package Manager:

<PackageReference Include="Blazing.Json.Queryable" Version="1.1.0" />

Or via the .NET CLI:

dotnet add package Blazing.Json.Queryable

Or via the Package Manager Console:

Install-Package Blazing.Json.Queryable

Requirements

.NET 10.0 or later
System.Text.Json (included with .NET)
Blazing.Json.JSONPath - 100% RFC 9535 compliant JSONPath implementation (NuGet Package)
- Automatically included as a dependency
- Enables RFC 9535 filter expressions, functions, and slicing
Utf8JsonAsyncStreamReader - High-performance streaming JSON reader (NuGet Package)
- Automatically included as a dependency
- Enables efficient simple path navigation

Quick Start

using Blazing.Json.Queryable.Providers;

// From JSON string
var jsonString = """[{"Name":"Alice","Age":30},{"Name":"Bob","Age":25}]""";
var results = JsonQueryable<Person>.FromString(jsonString)
    .Where(p => p.Age > 25)
    .ToList();

// From file stream (memory-efficient)
await using var stream = File.OpenRead("data.json");
var streamResults = await JsonQueryable<Person>.FromStream(stream)
    .Where(p => p.Age > 25)
    .Take(10)
    .AsAsyncEnumerable()
    .ToListAsync();

// From UTF-8 bytes (zero-allocation)
byte[] utf8Json = Encoding.UTF8.GetBytes(jsonString);
var utf8Results = JsonQueryable<Person>.FromUtf8(utf8Json)
    .Where(p => p.Age > 25)
    .ToList();

public record Person(string Name, int Age);

Usage

Basic Queries

Standard LINQ operations work as expected:

var json = """
[
    {"Id":1,"Name":"Alice","Age":30,"City":"London","IsActive":true},
    {"Id":2,"Name":"Bob","Age":25,"City":"Paris","IsActive":true},
    {"Id":3,"Name":"Charlie","Age":35,"City":"London","IsActive":false}
]
""

// Filtering
var adults = JsonQueryable<Person>.FromString(json)
    .Where(p => p.Age >= 18)
    .ToList();

// Projection
var names = JsonQueryable<Person>.FromString(json)
    .Select(p => p.Name)
    .ToList();

// Ordering
var sorted = JsonQueryable<Person>.FromString(json)
    .OrderBy(p => p.Age)
    .ThenBy(p => p.Name)
    .ToList();

// Aggregation
var avgAge = JsonQueryable<Person>.FromString(json)
    .Average(p => p.Age);

var totalActive = JsonQueryable<Person>.FromString(json)
    .Count(p => p.IsActive);

// Combined operations
var results = JsonQueryable<Person>.FromString(json)
    .Where(p => p.Age >= 25 && p.City == "London")
    .OrderByDescending(p => p.Age)
    .Select(p => new { p.Name, p.Age })
    .Take(5)
    .ToList();

Streaming Queries

Process large files with constant memory usage:

// Traditional approach - loads entire file into memory
var traditionalResults = new List<Person>();
var allJson = File.ReadAllText("huge-dataset.json"); // Loads ALL 500MB
var allPeople = JsonSerializer.Deserialize<List<Person>>(allJson); // Deserializes ALL
traditionalResults = allPeople.Where(p => p.Age > 25).Take(100).ToList();

// Blazing.Json.Queryable - streams and stops early
var streamingResults = new List<Person>();
await using (var stream = File.OpenRead("huge-dataset.json"))
{
    await foreach (var person in JsonQueryable<Person>.FromStream(stream)
        .Where(p => p.Age > 25)
        .Take(100) // Stops reading after 100 matches!
        .AsAsyncEnumerable())
    {
        streamingResults.Add(person);
    }
}
// Result: 10-20x faster, uses constant memory

Advanced Streaming with .NET 10 Async LINQ

.NET 10 includes built-in async LINQ support, enabling powerful async transformations:

await using var stream = File.OpenRead("data.json");

// Async predicates - call async methods in Where clauses
await foreach (var person in JsonQueryable<Person>.FromStream(stream)
    .AsAsyncEnumerable()
    .Where(async (p, ct) => await IsValidUserAsync(p, ct)) // Async predicate!
    .OrderBy(p => p.Name)
    .Take(100))
{
    Console.WriteLine(person.Name);
}

// Async transformations with Select
await foreach (var enriched in JsonQueryable<Person>.FromStream(stream)
    .AsAsyncEnumerable()
    .Select(async (p, ct) => await EnrichPersonDataAsync(p, ct)) // Async select!
    .Where(p => p.Score > 50))
{
    await ProcessAsync(enriched);
}

// Cancellation support
using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(5));

await foreach (var item in JsonQueryable<Person>.FromStream(stream)
    .AsAsyncEnumerable()
    .WithCancellation(cts.Token))
{
    // Automatically cancelled after 5 minutes
}

UTF-8 Direct Processing

Process UTF-8 bytes without string conversion overhead:

// From UTF-8 byte array
byte[] utf8Json = Encoding.UTF8.GetBytes(jsonString);
var results = JsonQueryable<Person>.FromUtf8(utf8Json)
    .Where(p => p.Age > 25)
    .ToList();

// From ReadOnlySpan<byte> (zero-allocation)
ReadOnlySpan<byte> utf8Span = utf8Json.AsSpan();
var spanResults = JsonQueryable<Person>.FromUtf8(utf8Span)
    .Where(p => p.Age > 25)
    .ToList();

// From UTF-8 stream (most efficient for large data)
await using var utf8Stream = File.OpenRead("data.json");
await foreach (var person in JsonQueryable<Person>.FromStream(utf8Stream)
    .AsAsyncEnumerable())
{
    // Process each person with zero string allocations
}

Query Syntax (Query Expression Syntax)

Blazing.Json.Queryable supports both method syntax (fluent) and query syntax (query expression) for LINQ queries. Query syntax provides a declarative, SQL-like alternative that can be more readable for complex queries.

Tip

📚 Learn More About LINQ Query Syntax:

Method Syntax vs Query Syntax

var json = """[{"Name":"Alice","Age":30},{"Name":"Bob","Age":25},{"Name":"Charlie","Age":35}]""";

// METHOD SYNTAX (fluent) - Chain methods
var methodResults = JsonQueryable<Person>.FromString(json)
    .Where(p => p.Age > 25)
    .OrderBy(p => p.Name)
    .Select(p => new { p.Name, p.Age })
    .ToList();

// QUERY SYNTAX (declarative) - SQL-like keywords
var queryResults = (from p in JsonQueryable<Person>.FromString(json)
                    where p.Age > 25
                    orderby p.Name
                    select new { p.Name, p.Age })
                   .ToList();

// Both produce identical results!

Query Syntax with All Library Features

1. FromString - Basic Query Syntax:

var results = (from p in JsonQueryable<Person>.FromString(json)
               where p.Age > 25 && p.IsActive
               select new { p.Name, p.City })
              .ToList();

2. FromUtf8 - Zero-allocation UTF-8 Processing:

byte[] utf8Bytes = Encoding.UTF8.GetBytes(jsonString);

var results = (from p in JsonQueryable<Person>.FromUtf8(utf8Bytes)
               where p.Age >= 30
               orderby p.Name
               select p)
              .ToList();

3. FromFile - Direct File Access:

var results = (from p in JsonQueryable<Person>.FromFile("data.json")
               where p.IsActive
               orderby p.Age descending
               select new { p.Name, p.Age })
              .Take(10)
              .ToList();

4. FromStream - Async Streaming:

await using var stream = File.OpenRead("data.json");

await foreach (var person in (from p in JsonQueryable<Person>.FromStream(stream)
                               where p.Age > 25
                               orderby p.Name
                               select p)
                              .Take(10)
                              .AsAsyncEnumerable())
{
    Console.WriteLine($"{person.Name}, {person.Age}");
}

5. Multi-level Sorting:

var results = (from p in JsonQueryable<Person>.FromString(json)
               orderby p.City, p.Age descending, p.Name
               select p)
              .ToList();

6. Grouping with Aggregations:

var results = (from p in JsonQueryable<Person>.FromString(json)
               group p by p.City into cityGroup
               select new
               {
                   City = cityGroup.Key,
                   Count = cityGroup.Count(),
                   AvgAge = cityGroup.Average(p => p.Age),
                   MinAge = cityGroup.Min(p => p.Age),
                   MaxAge = cityGroup.Max(p => p.Age)
               })
              .ToList();

7. JSONPath Pre-filtering with Query Syntax:

// Combine RFC 9535 JSONPath filters with query syntax
var results = (from p in JsonQueryable<Product>
                   .FromString(json, "$[[email protected] < 100 && @.stock > 0]")
               group p by p.Category into catGroup
               orderby catGroup.Key
               select new
               {
                   Category = catGroup.Key,
                   Count = catGroup.Count(),
                   AvgPrice = catGroup.Average(p => p.Price),
                   Products = catGroup.Select(p => p.Name).ToList()
               })
              .ToList();

8. Complex Real-World Query:

// Employee department analysis with filtering, grouping, and sorting
var results = (from e in JsonQueryable<Employee>.FromString(json)
               where e.IsActive && e.Salary > 60000
               group e by e.Department into deptGroup
               where deptGroup.Count() > 1
               orderby deptGroup.Average(e => e.Salary) descending
               select new
               {
                   Department = deptGroup.Key,
                   Count = deptGroup.Count(),
                   AvgSalary = deptGroup.Average(e => e.Salary),
                   TopEarner = deptGroup.OrderByDescending(e => e.Salary).First().Name
               })
              .ToList();

When to Use Query Syntax

Both query syntax and method syntax are equally powerful and produce identical compiled code. The choice between them is purely a matter of readability and personal/team preference. Here's a side-by-side comparison to help you decide:

Use Query Syntax When:	Use Method Syntax When:
Complex queries with joins and grouping (more SQL-like readability)	Simple filtering and projection
Multiple `from` clauses (SelectMany scenarios)	Chaining many operations (more fluent)
Team prefers declarative style	Using methods not available in query syntax (Take, Skip, Distinct, etc.)
Using `let` keyword for intermediate results	Personal/team preference for fluent style

Note

Both syntaxes are fully supported and produce identical expression trees. You can even mix them:

var mixed = (from p in JsonQueryable<Person>.FromString(json)
             where p.Age > 25
             select p)
            .Take(10)  // Method syntax at the end
            .ToList();

Tip

See Full Examples: Check out samples/Blazing.Json.Queryable.Samples/Examples/QuerySyntaxSamples.cs for comprehensive demonstrations of query syntax with all library features including FromUtf8, FromFile, FromStream, and JSONPath integration.

Supported LINQ Methods

Filtering Operations

Method	Description	Example
`Where`	Filters elements based on a predicate	`.Where(p => p.Age > 18)`
`OfType<T>`	Filters elements by type	`.OfType<Employee>()`

Projection Operations

Method	Description	Example
`Select`	Projects each element to a new form	`.Select(p => p.Name)`
`SelectMany`	Flattens nested sequences	`.SelectMany(p => p.Orders)`
`Cast<T>`	Casts elements to specified type	`.Cast<Employee>()`

Ordering Operations

Method	Description	Example
`OrderBy`	Sorts elements in ascending order	`.OrderBy(p => p.Age)`
`OrderByDescending`	Sorts elements in descending order	`.OrderByDescending(p => p.Age)`
`ThenBy`	Secondary ascending sort	`.ThenBy(p => p.Name)`
`ThenByDescending`	Secondary descending sort	`.ThenByDescending(p => p.Name)`
`Order`	Sorts elements in ascending order (C# 14)	`.Order()`
`OrderDescending`	Sorts elements in descending order (C# 14)	`.OrderDescending()`
`Reverse`	Reverses the order of elements	`.Reverse()`

Quantifier Operations

Method	Description	Example
`All`	Tests if all elements satisfy a condition	`.All(p => p.Age >= 18)`
`Any`	Tests if any element satisfies a condition	`.Any(p => p.City == "London")`
`Contains`	Tests if sequence contains an element	`.Contains(person)`
`SequenceEqual`	Tests if two sequences are equal	`.SequenceEqual(other)`

Element Access Operations

Method	Description	Example
`First`	Returns first element	`.First()`
`FirstOrDefault`	Returns first element or default	`.FirstOrDefault()`
`Last`	Returns last element	`.Last()`
`LastOrDefault`	Returns last element or default	`.LastOrDefault()`
`Single`	Returns the only element	`.Single()`
`SingleOrDefault`	Returns the only element or default	`.SingleOrDefault()`
`ElementAt`	Returns element at specified index	`.ElementAt(5)`
`ElementAtOrDefault`	Returns element at index or default	`.ElementAtOrDefault(5)`

Aggregation Operations

Method	Description	Example
`Count`	Counts elements	`.Count()`
`LongCount`	Counts elements (64-bit)	`.LongCount()`
`Sum`	Sums numeric values	`.Sum(p => p.Salary)`
`Average`	Calculates average	`.Average(p => p.Age)`
`Min`	Finds minimum value	`.Min(p => p.Age)`
`Max`	Finds maximum value	`.Max(p => p.Salary)`
`MinBy`	Finds element with minimum key value	`.MinBy(p => p.Age)`
`MaxBy`	Finds element with maximum key value	`.MaxBy(p => p.Salary)`
`Aggregate`	Applies custom accumulator	`.Aggregate((a, b) => a + b)`

Set Operations

Method	Description	Example
`Distinct`	Removes duplicate elements	`.Distinct()`
`DistinctBy`	Removes duplicates by key	`.DistinctBy(p => p.Email)`
`Union`	Combines two sequences	`.Union(otherPeople)`
`UnionBy`	Combines sequences by key	`.UnionBy(other, p => p.Id)`
`Intersect`	Finds common elements	`.Intersect(otherPeople)`
`IntersectBy`	Finds common elements by key	`.IntersectBy(ids, p => p.Id)`
`Except`	Finds elements not in second sequence	`.Except(otherPeople)`
`ExceptBy`	Finds elements not in second by key	`.ExceptBy(ids, p => p.Id)`

Partitioning Operations

Method	Description	Example
`Take`	Takes first N elements	`.Take(10)`
`TakeLast`	Takes last N elements	`.TakeLast(10)`
`TakeWhile`	Takes elements while condition is true	`.TakeWhile(p => p.Age < 30)`
`Skip`	Skips first N elements	`.Skip(20)`
`SkipLast`	Skips last N elements	`.SkipLast(5)`
`SkipWhile`	Skips elements while condition is true	`.SkipWhile(p => p.Age < 18)`
`Chunk`	Divides elements into chunks of specified size	`.Chunk(10)`

Grouping Operations

Method	Description	Example
`GroupBy`	Groups elements by key	`.GroupBy(p => p.City)`
`GroupJoin`	Groups and joins sequences	`.GroupJoin(orders, ...)`
`Join`	Joins two sequences	`.Join(orders, ...)`

Conversion Operations

Method	Description	Example
`ToList`	Converts to List<T> (synchronous)	`.ToList()`
`ToArray`	Converts to array (synchronous)	`.ToArray()`
`ToDictionary`	Converts to dictionary	`.ToDictionary(p => p.Id)`
`ToHashSet`	Converts to hash set	`.ToHashSet()`
`ToLookup`	Converts to lookup	`.ToLookup(p => p.City)`
`AsEnumerable`	Returns as IEnumerable<T>	`.AsEnumerable()`
`AsQueryable`	Returns as IQueryable<T>	`.AsQueryable()`
`AsAsyncEnumerable`	Returns as IAsyncEnumerable<T> for async operations	`.AsAsyncEnumerable()`

Note

For async conversion operations, use .AsAsyncEnumerable() followed by .NET 10's built-in async LINQ methods:

await query.AsAsyncEnumerable().ToListAsync() - Async conversion to List<T>
await query.AsAsyncEnumerable().ToArrayAsync() - Async conversion to array
await query.AsAsyncEnumerable().ToDictionaryAsync(...) - Async conversion to dictionary

Sequence Operations

Method	Description	Example
`Concat`	Concatenates two sequences	`.Concat(otherPeople)`
`Append`	Appends element to end	`.Append(person)`
`Prepend`	Prepends element to start	`.Prepend(person)`
`Zip`	Combines sequences pairwise	`.Zip(ages, (p, a) => ...)`
`DefaultIfEmpty`	Returns default if empty	`.DefaultIfEmpty()`

JSONPath Support (RFC 9535)

Blazing.Json.Queryable provides powerful JSON filtering capabilities through Blazing.Json.JSONPath - a high-performance, 100% RFC 9535 compliant JSONPath implementation.

Note

📖 Full JSONPath Documentation: For complete details on JSONPath syntax, features, and RFC 9535 compliance, visit the Blazing.Json.JSONPath repository. This library is automatically included as a dependency when you install Blazing.Json.Queryable.

Overview

JSONPath provides a powerful query language for JSON documents, similar to XPath for XML. When used with Blazing.Json.Queryable, JSONPath expressions pre-filter JSON data before deserialization, dramatically improving performance and reducing memory usage.

The RFC 9535 standard defines a consistent, interoperable syntax for:

Filter Expressions: Pre-filter JSON using comparison (==, !=, <, >, etc.) and logical operators (&&, ||, !)
Built-in Functions: length(), count(), match(), search(), value()
Array Slicing: [start:end:step] with negative indices
Path Navigation: Navigate nested structures efficiently

Why Use JSONPath with Blazing.Json.Queryable?

Performance Benefits:

// Traditional LINQ: Deserialize ALL 1M products, then filter in C#
var traditional = JsonQueryable<Product>.FromString(json)
    .Where(p => p.Price < 100 && p.InStock && p.Category == "Electronics")
    .ToList();
// Deserializes: 1,000,000 objects
// Memory: ~500MB, Time: ~2000ms

// JSONPath Pre-filter: Filter in JSON BEFORE deserialization (1M → 500 items)
var optimized = JsonQueryable<Product>
    .FromString(json, "$[[email protected] < 100 && @.inStock == true && @.category == 'Electronics']")
    .ToList();
// Deserializes: 500 objects (only matches!)
// Memory: ~25MB (20x less!), Time: ~200ms (10x faster!)

Key Advantages:

10-100x faster for highly selective queries on large datasets
90%+ memory reduction by deserializing only matching items
Powered by Blazing.Json.JSONPath: 100% RFC 9535 compliant implementation
Combines best of both: JSONPath for pre-filtering + LINQ for rich operations

JSONPath Syntax Quick Reference

Feature	Syntax	Example	Description
Root	`$`	`$`	Root JSON element
Child	`.property` or `['property']`	`$.data.users`	Access nested property
Wildcard	`` or `[]`	`$.users[*]`	All array elements
Filter	`[?expression]`	`$[[email protected] > 25]`	Filter by condition
Slice	`[start:end]`	`$[2:10]`	Array slice
Step	`:step`	`$[::2]`	Every 2nd element
Current	`@`	`@.price < 100`	Current element in filter
Comparison	`==`, `!=`, `<`, `<=`, `>`, `>=`	`@.price >= 50`	Comparison operators
Logical	`&&`, `\|\|`, `!`	`@.age > 18 && @.active`	Logical operators
length()	`length(value)`	`length(@.name) > 5`	String, array, or object length
match()	`match(string, pattern)`	`match(@.email, '.*@example\\.com')`	Full regex match (I-Regexp)
search()	`search(string, pattern)`	`search(@.desc, 'wireless')`	Substring regex search
count()	`count(nodelist)`	`count($.items[*])`	Count nodes in nodelist
value()	`value(nodelist)`	`value(@.tags[0])`	Convert singular nodelist to value

Simple Path Navigation

Navigate to nested arrays using simple JSONPath expressions:

// API response: { "data": { "user": { "repositories": [...] } } }
var repos = JsonQueryable<Repository>
    .FromString(apiResponse, "$.data.user.repositories[*]")
    .Where(r => !r.IsPrivate)
    .ToList();

Filter Expressions

Apply powerful filters directly in JSONPath (powered by Blazing.Json.JSONPath):

// Comparison operators: ==, !=, <, <=, >, >=
var highEarners = JsonQueryable<Employee>
    .FromString(json, "$[[email protected] > 100000]")
    .ToList();

// Logical operators: &&, ||, !
var seniorHighEarners = JsonQueryable<Employee>
    .FromString(json, "$[[email protected] > 90000 && @.yearsEmployed >= 5]")
    .ToList();

Built-in Functions

Blazing.Json.JSONPath provides all RFC 9535 standard functions:

// length() - string, array, or object length
var longNames = JsonQueryable<Product>
    .FromString(json, "$[?length(@.name) > 10]")
    .ToList();

// match() - full regex match (I-Regexp RFC 9485)
var electronics = JsonQueryable<Product>
    .FromString(json, "$[?match(@.code, '^ELEC-.*')]")
    .ToList();

// search() - substring regex search
var wireless = JsonQueryable<Product>
    .FromString(json, "$[?search(@.name, 'Wireless')]")
    .ToList();

Array Slicing

Advanced array slicing with step support:

// [start:end] - elements from start to end (exclusive)
var middle = JsonQueryable<Item>
    .FromString(json, "$[2:5]")  // Elements 2, 3, 4
    .ToList();

// [start:end:step] - every Nth element
var everyOther = JsonQueryable<Item>
    .FromString(json, "$[0:9:2]")  // Elements 0, 2, 4, 6, 8
    .ToList();

// Negative indices - count from end
var last3 = JsonQueryable<Item>
    .FromString(json, "$[-3:]")  // Last 3 elements
    .ToList();

// Reverse with negative step
var reversed = JsonQueryable<Item>
    .FromString(json, "$[::-1]")  // All elements in reverse
    .ToList();

Combining JSONPath with LINQ

The real power: JSONPath pre-filtering + LINQ rich operations

Real-World Example: Department Workforce Analysis

var employeesJson = """
[
    {"id": 1, "name": "Alice Johnson", "department": "Engineering", "salary": 95000, "yearsEmployed": 5},
    {"id": 2, "name": "Bob Smith", "department": "Engineering", "salary": 105000, "yearsEmployed": 8},
    {"id": 3, "name": "Charlie Brown", "department": "Sales", "salary": 75000, "yearsEmployed": 3},
    {"id": 4, "name": "Diana Prince", "department": "Engineering", "salary": 120000, "yearsEmployed": 10},
    {"id": 5, "name": "Eve Davis", "department": "Marketing", "salary": 70000, "yearsEmployed": 2},
    {"id": 6, "name": "Frank Miller", "department": "Engineering", "salary": 98000, "yearsEmployed": 6}
]
""";

Console.WriteLine("\nDepartment workforce analysis [MIXED JSONPATH + LINQ]:");
var deptAnalysis = JsonQueryable<Employee>
    .FromString(employeesJson, "$[[email protected] > 60000]")  // JSONPath: Filter employees with salary > $60K
    .GroupBy(e => e.Department)                          // LINQ: Group by department
    .Select(g => new
    {
        Department = g.Key,
        EmployeeCount = g.Count(),
        AvgSalary = g.Average(e => e.Salary),
        TotalYearsExperience = g.Sum(e => e.YearsEmployed),
        TopEarner = g.OrderByDescending(e => e.Salary).First().Name
    })
    .OrderByDescending(x => x.AvgSalary)    // LINQ: Primary sort by avg salary (descending)
    .ThenBy(x => x.Department)              // LINQ: Secondary sort by department name (ascending)
    .ToList();

foreach (var dept in deptAnalysis)
{
    Console.WriteLine($"      - {dept.Department}: {dept.EmployeeCount} employees, " +
                     $"${dept.AvgSalary:N0} avg salary, " +
                     $"{dept.TotalYearsExperience} total years, " +
                     $"top earner: {dept.TopEarner}");
}

Output:

Department workforce analysis [MIXED JSONPATH + LINQ]:
      - Engineering: 4 employees, $104,500 avg salary, 29 total years, top earner: Diana Prince
      - Sales: 1 employees, $75,000 avg salary, 3 total years, top earner: Charlie Brown
      - Marketing: 1 employees, $70,000 avg salary, 2 total years, top earner: Eve Davis

Performance Analysis:

JSONPath Pre-filter ($[[email protected] > 60000]): Filters in JSON before deserialization
LINQ Operations: GroupBy, aggregations, sorting on filtered set
Result: Minimal memory usage, type-safe operations, optimal performance

Performance Best Practices (Memory & Speed)

Warning

JSONPath Memory Considerations: The RFC 9535 specification requires the entire JSON document to be loaded into memory as a JsonDocument for advanced JSONPath features. Advanced filters, functions, and array slicing (e.g., $[[email protected] < 100], $[0:10], length(@.name)) will load the entire document before parsing, regardless of streaming.

Exception: Simple wildcard-only paths (e.g., $.data[*], $.departments[*].employees[*]) are handled differently by Blazing.Json.Queryable and maintain true streaming without loading the full document, even with multiple levels of nesting.

Working with Large JSON Documents

Blazing.Json.JSONPath requires the entire JSON document to be loaded into memory as a JsonDocument when using advanced RFC 9535 features:

Filter expressions: $[[email protected] > 25], $[[email protected] < 100 && @.inStock]
Array slicing: $[0:10], $[2:5:2], $[-3:]
Functions: length(), count(), match(), search(), value()

However, simple wildcard-only paths use streaming and maintain constant memory usage:

Single-level wildcards: $.data[*], $.users[*]
Multi-level wildcards: $.departments[*].employees[*], $.organization[*].divisions[*].departments[*].employees[*]

Memory Usage Guidelines:

Small documents (<1MB): No concerns - use any JSONPath feature
Medium documents (1-100MB): Generally fine on modern systems - monitor memory usage
Large documents (100MB-1GB): Use simple wildcard paths for streaming, avoid advanced filters. Use Linq with JSONPath simple wildcard-only paths instead
Very large documents (>1GB): Use simple wildcard paths only - avoid advanced filters. Use Linq with JSONPath simple wildcard-only paths instead

Best Practices for Large Documents

DO: Use Simple Wildcard Paths for True Streaming (All File Sizes)

// GOOD: True streaming with simple wildcard paths (constant memory)
await using var stream = File.OpenRead("huge-file.json"); // 2GB file

// Single-level wildcard
var results1 = await JsonQueryable<Product>
    .FromStream(stream, "$.data[*]")  // Simple wildcard - maintains streaming!
    .Where(p => p.Price < 100 && p.InStock)  // LINQ filtering (streamed)
    .Take(10)
    .AsAsyncEnumerable()
    .ToListAsync();

// Multi-level wildcards - ALSO streams!
var results2 = await JsonQueryable<Employee>
    .FromStream(stream, "$.departments[*].employees[*]")  // Multi-level - still streams!
    .Where(e => e.Salary > 60000)  // LINQ filtering (streamed)
    .Take(10)
    .AsAsyncEnumerable()
    .ToListAsync();

// Deep nesting - STILL streams!
var results3 = await JsonQueryable<Employee>
    .FromStream(stream, "$.organization[*].divisions[*].departments[*].employees[*]")
    .Where(e => e.IsActive)  // LINQ filtering (streamed)
    .Take(10)
    .AsAsyncEnumerable()
    .ToListAsync();

// Memory: ~25MB (constant), processes 2GB file safely with ANY level of nesting

DO: Use JSONPath Advanced Filters for Selective Filtering (Small/Medium Files)

// GOOD: Pre-filter with JSONPath advanced filters (100MB file, 1M → 500 items)
var results = JsonQueryable<Product>
    .FromString(mediumJson, "$[[email protected] < 100 && @.inStock == true]")
    .OrderBy(p => p.Name)
    .Take(10)
    .ToList();
// Memory: Loads 100MB + filters → ~25MB result (acceptable for medium files)

❌ DON'T: Use Advanced JSONPath Filters on Very Large Files (>1GB)

// BAD: Advanced filter loads ENTIRE 2GB file into memory!
var results = JsonQueryable<Product>
    .FromString(hugeJson, "$[[email protected] < 100]")  // Loads ALL 2GB!
    .ToList();
// Memory: 2GB+ (OutOfMemoryException risk!)

❌ DON'T: Use LINQ Where for Initial Filtering on Large Datasets (1M+) Without JSONPath

// BAD: Deserializes ALL 1M first, then filters
var results = JsonQueryable<Product>
    .FromString(largeJson)  // No JSONPath pre-filter!
    .Where(p => p.Price < 100 && p.InStock)  // Deserializes 1,000,000 objects
    .OrderBy(p => p.Name)
    .Take(10)
    .ToList();
// Memory: ~500MB (all objects deserialized before filtering)

Streaming Strategy Summary

Document Size	Recommended Approach	JSONPath Pattern	Memory Impact
< 1MB	Any JSONPath feature	`$[[email protected] < 100]` or `$.data[*]`	Minimal
1-100MB	Advanced filters (monitor) or simple wildcards	`$[[email protected] < 100]` or `$.data[*]`	Document size + overhead OR constant
100MB-1GB	Simple wildcard paths (preferred)	`$.data[]` or `$.dept[].emp[*]`	Constant (~25MB)
> 1GB	Simple wildcard paths (required)	`$.data[]` or `$.dept[].emp[*]`	Constant (~25MB)

Key Features:

100% RFC 9535 compliant
High-performance implementation
Full filter expression support
All standard built-in functions
Array slicing with step
I-Regexp (RFC 9485) regex support

Key Takeaway:

Simple wildcard paths (e.g., $.data[*], $.departments[*].employees[*]) maintain true streaming regardless of nesting depth - perfect for large/very large files
Advanced JSONPath filters (e.g., $[[email protected] < 100], $[0:10], length()) load entire document - use only for small/medium files
For large files, use simple wildcards + LINQ filtering to get constant memory usage

Sample Code:

samples/Blazing.Json.Queryable.Samples/Examples/AdvancedJsonPathSamples.cs
Real-world scenarios demonstrating JSONPath + LINQ combinations

How It Works

Blazing.Json.Queryable uses a custom LINQ provider that translates LINQ expressions into efficient JSON processing operations:

Expression Tree Analysis: LINQ queries are analyzed to build an execution plan
Streaming JSON Parser: Uses System.Text.Json.Utf8JsonReader for efficient parsing
JSONPath Navigation: Powered by Blazing.Json.JSONPath for RFC 9535 compliant filtering when JSONPath expressions are used
Lazy Evaluation: Only processes JSON elements needed for the query
Early Termination: Stops reading JSON as soon as query requirements are met
Zero-Allocation UTF-8: Direct UTF-8 processing using Span<byte> and ReadOnlySpan<byte>
Async Enumeration: Native IAsyncEnumerable<T> support for non-blocking I/O

Execution Flow Example

// Query: Find first 10 active users over 25 in London
var results = await JsonQueryable<Person>.FromStream(stream)
    .Where(p => p.Age > 25)
    .Where(p => p.City == "London")
    .Where(p => p.IsActive)
    .OrderBy(p => p.Name)
    .Take(10)
    .AsAsyncEnumerable()
    .ToListAsync();

// Internal execution:
// 1. Build execution plan from LINQ expression tree
// 2. Open JSON stream and start parsing
// 3. For each JSON object:
//    a. Check Age > 25 (filter early)
//    b. Check City == "London" (filter)
//    c. Check IsActive (filter)
//    d. If all pass, deserialize to Person object
//    e. Add to ordered buffer
// 4. Stop reading after 10th match (early termination!)
// 5. Return results without processing entire file

Performance Advantages

Real-World Performance Comparison

Scenario: 100,000 record JSON file (52MB), find first 10 matching records

Approach	Time	Memory	Notes
Traditional	1,200ms	450MB	Loads entire file, deserializes all
Blazing.Json.Queryable	120ms	25MB	Streams, stops after 10 matches
Improvement	10x faster	18x less	90% time reduction

Key Performance Benefits

Early Termination:
- Traditional: Processes ALL 100K records
- Blazing.Json.Queryable: Stops after finding 10 matches
- Result: 10-20x faster for Take/First operations
Memory Efficiency:
- Traditional: Loads 52MB file + 450MB objects
- Blazing.Json.Queryable: Uses ~25MB constant memory
- Result: Can process files larger than available RAM
Zero-Allocation UTF-8:
- Traditional: UTF-8 → String → Object
- Blazing.Json.Queryable: UTF-8 → Object (direct)
- Result: Fewer allocations, less GC pressure
Async Streaming:
- Traditional: Blocking I/O during file read
- Blazing.Json.Queryable: Non-blocking async enumeration
- Result: Better scalability in web APIs

When Performance Gains Are Largest

Large files (> 10MB): Bigger file = larger advantage
Early termination (Take/First): Smaller N = larger speedup
Complex filters: More filtering = more benefit from stream processing
Memory constraints: Limited RAM = Blazing.Json.Queryable enables processing that's impossible traditionally

Samples

The library includes comprehensive sample and benchmark projects demonstrating different usage patterns and performance characteristics:

Sample Projects

All samples are located in the samples/Blazing.Json.Queryable.Samples directory (basic to advanced order):

BasicQueries.cs - Fundamental LINQ operations (Where, Select, OrderBy, Take, Skip, First, Count, Any)
Utf8Queries.cs - Zero-allocation UTF-8 processing examples
StreamQueries.cs - Async streaming with IAsyncEnumerable<T>
AsyncQueries.cs - .NET 10 async LINQ with async predicates and transformations
CustomConverters.cs - Custom JSON converters and serialization options
ElementAccessSamples.cs - Element access operations (ElementAt, Last, Single with C# Index support)
ConversionOperationsSamples.cs - Conversion operations (ToDictionary, ToHashSet, ToLookup)
AdvancedScenarios.cs - Real-world patterns, error handling, and best practices
LargeDatasetSamples.cs - In-memory processing of large datasets (100K-1M records)
LargeDatasetFileStreamingSamples.cs - True I/O streaming for memory-efficient large file processing
ComplexGroupingSamples.cs - GroupBy, Join, and GroupJoin operations with aggregations
JsonPathSamples.cs - Simple JSONPath navigation for nested structures
AdvancedJsonPathSamples.cs - RFC 9535 filters, functions, slicing, and real-world scenarios
AdvancedLinqOperationsSamples.cs - Advanced operations (Chunk, Zip, DistinctBy, ExceptBy, IntersectBy, UnionBy)
QuerySyntaxSamples.cs - Query expression syntax (SQL-like declarative LINQ queries)
PerformanceComparison.cs - Benchmarks comparing traditional vs streaming approaches

Benchmark Suites

All benchmarks are located in the benchmarks/Blazing.Json.Queryable.Benchmarks directory:

SyncInMemoryBenchmarks - In-memory performance (100, 1K, 10K records)
SyncFileIOBenchmarks - File I/O performance (1K, 10K records)
AsyncStreamBenchmarks - Async streaming (10K, 100K records)
MemoryAllocationBenchmarks - Zero-allocation validation
LargeFileStreamingBenchmarks - Constant memory proof (10MB, 100MB files)
ComprehensiveComparisonBenchmarks - Side-by-side comparison across all scenarios

Running the Samples

# Clone the repository
git clone https://github.com/gragra33/Blazing.Json.Queryable.git
cd Blazing.Json.Queryable

# Run the samples project (interactive menu)
dotnet run --project samples/Blazing.Json.Queryable.Samples

The samples include:

Interactive console menu
Performance benchmarks with timing
Memory usage comparisons
Real-world scenarios
Best practices demonstrations

Running the Benchmarks

# Run benchmarks (interactive mode)
cd benchmarks/Blazing.Json.Queryable.Benchmarks
dotnet run -c Release

# Or run specific benchmark suite
dotnet run -c Release -- --filter *SyncInMemory*
dotnet run -c Release -- --filter *AsyncStream*
dotnet run -c Release -- --filter *Comprehensive*

# List all available benchmarks
dotnet run -c Release -- --list flat

The benchmarks provide:

Detailed performance metrics (Mean, StdDev, Min, Max)
Memory allocation tracking
GC collection statistics
Side-by-side comparisons (Traditional vs Blazing.Json.Queryable)
HTML/Markdown reports in BenchmarkDotNet.Artifacts

Give a ⭐

If you like or are using this project to learn or start your solution, please give it a star. Thanks!

Also, if you find this library useful and you're feeling really generous, please consider buying me a coffee ☕.

Support

Documentation: Full API documentation included in NuGet package
Samples: Comprehensive samples in the repository
Issues: Report bugs or request features on GitHub Issues
Discussions: Ask questions on GitHub Discussions

History

V1.1.1 - 12 January, 2026

Streaming Restoration for Simple Wildcard Paths
- Restored true streaming for simple wildcard-only JSONPath expressions
- Multi-level wildcard paths (e.g., $.departments[*].employees[*]) now stream with constant memory usage
Documentation Updates:
- Updated Performance Best Practices with accurate streaming behavior
- Clear guidance on when simple wildcards stream vs when advanced features materialize
- Memory usage guidelines for different file sizes
Backward Compatibility:
- All existing code continues to work without changes
- Advanced RFC 9535 features (filters, functions, slicing) continue to use optimized materialization
- No breaking changes to public API
New Samples:
- ElementAccessSamples.cs - Element access operations
- ConversionOperationsSamples.cs - Conversion operations

V1.1.0 - 12 January, 2026

RFC 9535 JSONPath Support
- Full compliance with RFC 9535 - the official IETF standard
- Filter expressions in JSON before deserialization
- Regex pattern matching with I-Regexp (RFC 9485)
- Substring search capabilities
- Combine JSONPath filters with LINQ operations
- Nested path navigation with filters
Performance Improvements:
- Pre-filter in JSON reduces deserialization overhead
- 10-100x faster for filtered queries on large datasets
- Seamless integration with existing LINQ pipeline
- Automatic optimization routing
New Samples:
- AdvancedJsonPathSamples - RFC 9535 features demonstration
- Real-world scenarios with complex filters
- Performance comparison examples

V1.0.0 - 11 January, 2026

Initial Release - Full production release

Core Features:
- Custom LINQ provider for JSON with 60+ LINQ methods
  - Query Expression and Method Syntax support
- Direct JSON processing without full deserialization
- Early termination support for Take, First, Any operations
- Memory-efficient streaming for files larger than RAM
Input Format Support:
- Standard JSON strings
- UTF-8 byte arrays and spans (zero-allocation)
- File streams with async enumeration
- Advanced streaming with IAsyncEnumerable<T>
LINQ Operations:
- Filtering: Where, OfType
- Projection: Select, SelectMany, Cast
- Ordering: OrderBy, OrderByDescending, ThenBy, ThenByDescending, Order, OrderDescending, Reverse
- Quantifiers: All, Any, Contains, SequenceEqual
- Element Access: First, FirstOrDefault, Last, LastOrDefault, Single, SingleOrDefault, ElementAt, ElementAtOrDefault
- Aggregation: Count, LongCount, Sum, Average, Min, Max, MinBy, MaxBy, Aggregate
- Set Operations: Distinct, DistinctBy, Union, UnionBy, Intersect, IntersectBy, Except, ExceptBy
- Partitioning: Take, TakeLast, TakeWhile, Skip, SkipLast, SkipWhile, Chunk
- Grouping: GroupBy, GroupJoin, Join
- Conversion: ToList, ToArray, ToDictionary, ToHashSet, ToLookup, AsEnumerable, AsQueryable, AsAsyncEnumerable
- Sequence: Concat, Append, Prepend, Zip, DefaultIfEmpty
.NET 10 Features:
- Built-in async LINQ support with async predicates
- Async transformations with Select
- Native cancellation support
- Enhanced performance with latest runtime optimizations
Performance Optimizations:
- Zero-allocation UTF-8 processing with Span
- Early termination (10-20x faster for Take/First on large files)
- Constant memory usage regardless of file size
- Efficient execution plan optimization
Advanced Capabilities:
- Simple JSONPath support for nested JSON navigation
- Complex query translation and optimization
- Async streaming with proper cancellation
Documentation:
- Comprehensive README with examples
- Full XML documentation in NuGet package
- Sample project with 10+ demonstrations of all features
- Performance comparison benchmarks
Quality Assurance:
- Comprehensive test coverage
- Real-world performance benchmarks
- Production-ready error handling
- Best practices implementation

License: MIT License - see LICENSE file for details

Name		Name	Last commit message	Last commit date
Latest commit History 15 Commits
benchmarks/Blazing.Json.Queryable.Benchmarks		benchmarks/Blazing.Json.Queryable.Benchmarks
samples/Blazing.Json.Queryable.Samples		samples/Blazing.Json.Queryable.Samples
src/Blazing.Json.Queryable		src/Blazing.Json.Queryable
tests/Blazing.Json.Queryable.Tests		tests/Blazing.Json.Queryable.Tests
.gitattributes		.gitattributes
.gitignore		.gitignore
Blazing.Json.Queryable.slnx		Blazing.Json.Queryable.slnx
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Directory.Build.props		Directory.Build.props
Directory.Packages.props		Directory.Packages.props
LICENSE		LICENSE
README.md		README.md
cleanbuild.ps1		cleanbuild.ps1

License

gragra33/Blazing.Json.Queryable

Folders and files

Latest commit

History

Repository files navigation

Blazing.Json.Queryable

Table of Contents

Overview

Key Features

Why Choose Blazing.Json.Queryable?

Performance Advantages Over Traditional JSON + LINQ

Key Advantages

When to Use Each Approach

Getting Started

Installation

Requirements

Quick Start

Usage

Basic Queries

Streaming Queries

Advanced Streaming with .NET 10 Async LINQ

UTF-8 Direct Processing

Query Syntax (Query Expression Syntax)

Method Syntax vs Query Syntax

Query Syntax with All Library Features

When to Use Query Syntax

Supported LINQ Methods

Filtering Operations

Projection Operations

Ordering Operations

Quantifier Operations

Element Access Operations

Aggregation Operations

Set Operations

Partitioning Operations

Grouping Operations

Conversion Operations

Sequence Operations

JSONPath Support (RFC 9535)

Overview

Why Use JSONPath with Blazing.Json.Queryable?

JSONPath Syntax Quick Reference

Simple Path Navigation

Filter Expressions

Built-in Functions

Array Slicing

Combining JSONPath with LINQ

Real-World Example: Department Workforce Analysis

Performance Best Practices (Memory & Speed)

Working with Large JSON Documents

Best Practices for Large Documents

Streaming Strategy Summary

How It Works

Execution Flow Example

Performance Advantages

Real-World Performance Comparison

Key Performance Benefits

When Performance Gains Are Largest

Samples

Sample Projects

Benchmark Suites

Running the Samples

Running the Benchmarks

Give a ⭐

Support

History

V1.1.1 - 12 January, 2026

V1.1.0 - 12 January, 2026

V1.0.0 - 11 January, 2026

About

Topics

Resources

License

Code of conduct

Contributing

Uh oh!

Stars

Watchers

Forks

Packages