Redis Search Revolution: A LINQ-Inspired API Proposal

Redisearch can be difficult for some developers to fully utilize, due to its complex syntax and the nuances between FT.SEARCH, FT.AGGREGATE, and FT.HYBRID. Moreover, its weakly typed model necessitates manual data mapping, adding another layer of complexity. This article explores a proposal to overhaul RediSearch with a LINQ-like API, drawing inspiration from .NET's Language Integrated Query (LINQ) to provide a more intuitive and strongly-typed querying experience.

The Challenge with RediSearch

Currently, RediSearch requires developers to learn a specific syntax and understand the subtle differences between its core commands. This can be a significant barrier to entry, especially for those already familiar with more modern query languages and ORMs. The lack of a strong type model further complicates matters, requiring developers to manually manage data mapping between their application models and RediSearch's string-based query system. To fully utilize RediSearch, developers often grapple with the intricacies of its query language, the distinctions between commands like FT.SEARCH, FT.AGGREGATE, and FT.HYBRID, and the challenges of manual data mapping due to its weakly typed model. This complexity can hinder productivity and increase the learning curve for new users.

Introducing a LINQ-Inspired API for RediSearch

To address these challenges, a proposal has emerged to revamp RediSearch with an API inspired by .NET's LINQ. LINQ, introduced in .NET 3.5, provides a powerful and intuitive way to query data from various sources using a consistent syntax. By adopting a similar approach, RediSearch can offer a more developer-friendly experience, leveraging the familiarity and expressiveness of LINQ.

The core idea is to create a bespoke API that mimics the LINQ experience without directly implementing IQueryable<T>. This new API would allow developers to define their data models using C# classes, annotated with attributes to map properties to RediSearch fields. This approach would bring strong typing and compile-time checking to RediSearch queries, reducing the risk of runtime errors and improving code maintainability. For example, consider a Customer class:

[Index("ft_myindex")]
public class Customer
{
    [Key("id")] public int Id { get; set; }
    [Text("name")] public string? Name { get; set; }
    [Text("url")] public string? Url { get; set; }
    [Text("country")] public string? Country { get; set; }
    [Text("title")] public string? Title { get; set; }
    [Timestamp("timestamp")] public DateTime Date { get; set; }
    [Tags("cat")] public string[] Categories { get; set; } = [];
}

With this model in place, developers could use the new API to construct queries in a natural and expressive way, similar to LINQ. This approach not only simplifies query construction but also aligns with modern development practices, making RediSearch more accessible to a broader range of developers.

Querying with the New API: Examples

Let's delve into some examples to illustrate how the proposed LINQ-inspired API would work in practice. These examples showcase the API's ability to translate complex RediSearch commands into simple, readable C# code, making RediSearch more accessible and developer-friendly.

Aggregation Queries

One of the key areas where the new API shines is in aggregation queries. Currently, performing aggregations in RediSearch requires writing complex commands with specific syntax. With the LINQ-inspired API, these queries become much more straightforward. For instance, to count the number of customers, you could write:

var query = db.Query<Customer>();
// FT.AGGREGATE ft_myindex GROUPBY 0 REDUCE count 0
var count = query.Aggregate().Count();

This simple line of code translates into a complex FT.AGGREGATE command, but the developer doesn't need to worry about the underlying syntax. Similarly, to find the most recent customer date, you could use:

// FT.AGGREGATE ft_myindex GROUPBY 0 REDUCE max 1 @timestamp
var maxDate = query.Aggregate().Max(x => x.Date);

These examples demonstrate how the API simplifies common aggregation tasks, making it easier for developers to extract valuable insights from their data without getting bogged down in complex command syntax. This streamlined approach enhances productivity and reduces the potential for errors, allowing developers to focus on the logic of their queries rather than the mechanics of the RediSearch command language.

Search Queries

The API also simplifies search queries, allowing developers to express complex search criteria using familiar LINQ syntax. This is a significant improvement over the current string-based query language, which can be cumbersome and error-prone. By leveraging LINQ, the new API enables developers to write more readable and maintainable search queries, reducing the complexity and improving the overall development experience.

For example, to search for customers with the title "dogs" and retrieve their IDs and names, you could write:

// FT.SEARCH ft_myindex @title:$x0 RETURN 2 id name PARAMS 2 x0 dogs DIALECT 2
var rows =
    from x in query
    where x.Title == "dogs"
    select new { x.Id, x.Name, Score = Index.Score() };

This query is not only easier to read but also strongly typed, ensuring that the properties being accessed exist on the Customer class. To search for customers in the "foo" category, you could use:

// FT.SEARCH ft_myindex @cat:{$x0} RETURN 1 id PARAMS 2 x0 foo DIALECT 2
var rows2 =
    from x in query
    where x.Categories.Contains("foo")
    select x.Id;

These examples highlight the API's ability to translate complex search requirements into concise and expressive C# code. This not only simplifies the development process but also makes the code more maintainable and less prone to errors. The use of LINQ syntax allows developers to leverage their existing knowledge and apply it to RediSearch, reducing the learning curve and increasing productivity.

Pagination and Sorting

The API also provides intuitive ways to handle pagination and sorting, which are common requirements in many applications. These features are often implemented with verbose and complex syntax in traditional RediSearch queries. The LINQ-inspired API simplifies these operations, making them more accessible and easier to use. By providing a more streamlined approach to pagination and sorting, the API enhances the overall developer experience and reduces the effort required to implement these features.

To implement pagination, you can use the Skip and Take methods:

// FT.SEARCH ft_myindex @cat:{$x0} RETURN 1 id LIMIT 10 20 PARAMS 2 x0 foo DIALECT 2
var page = rows2.Skip(10).Take(20);

This code snippet demonstrates how easy it is to retrieve a specific page of results, without having to manually construct the LIMIT clause in the RediSearch command. To sort results, you can use the orderby keyword:

// FT.SEARCH ft_myindex @title:$x0 RETURN 2 id name SORTBY id DESC PARAMS 2 x0 dogs DIALECT 2
var ordered = from row in rows
    orderby row.Id descending
    select row;

This example showcases how the API simplifies sorting, allowing developers to specify the sorting criteria in a natural and readable way. These features make it easier to build efficient and user-friendly applications that require pagination and sorting of search results. The intuitive syntax reduces the complexity of these operations, making them more accessible to developers of all skill levels.

Advanced Features: Scoring, Grouping, and Hybrid Queries

Beyond the basics, the proposed API also aims to support more advanced RediSearch features, such as custom scoring, grouping, and hybrid queries. These features are essential for building sophisticated search applications, and the API seeks to make them more accessible and easier to use. By providing a unified and intuitive interface for these advanced capabilities, the API empowers developers to leverage the full potential of RediSearch without getting bogged down in complex syntax and configurations.

To use a custom scorer, you can use the WithScorer method:

// FT.SEARCH ft_myindex @title:$x0 RETURN 2 id name SCORER TFIDF PARAMS 2 x0 dogs DIALECT 2
var withScorers = rows.WithScorer(Scorers.TfIdf);

This allows you to specify a scoring function to rank the search results based on your specific needs. Grouping can be achieved using the group by keyword:

var urls = from row in query
    where row.Url == "about.html"
    group row by (row.Country, row.Date.Date)
    into grp
    orderby grp.Key.Date descending
    select (grp.Key, visits: grp.Count());

var grouped = from row in rows
    group row by row.Name
    into g
    select new { g.Key, Count = g.Count() };

These examples demonstrate how the API simplifies complex grouping operations, making it easier to aggregate and analyze data. While the exact syntax for hybrid queries is still under consideration, the proposal suggests a CombineVector method to integrate vector similarity searches:

var hybrid =  query.Where(x => x.Name == "foo").CombineVector(x => x.VectorField, vectorValue) ... blah

This would allow developers to combine traditional text-based searches with vector-based similarity searches, enabling more powerful and nuanced search capabilities. These advanced features, when combined with the API's intuitive syntax, empower developers to build cutting-edge search applications with RediSearch.

Advantages of the LINQ-Inspired API

The proposed LINQ-inspired API offers several key advantages over the current RediSearch API. These advantages contribute to a more streamlined, efficient, and developer-friendly experience, making RediSearch more accessible to a wider range of users.

Strong Typing

One of the most significant benefits is the introduction of strong typing. By defining data models as C# classes, developers can leverage compile-time checking and reduce the risk of runtime errors. This ensures that queries are valid and that data is handled consistently, leading to more robust and maintainable code.

Familiar Syntax

Developers familiar with LINQ will find the new API easy to learn and use. This reduces the learning curve and allows them to quickly start building RediSearch-powered applications. The consistency with LINQ syntax also makes the code more readable and understandable, improving collaboration among developers.

Data Binding

The API handles data binding automatically, eliminating the need for manual mapping between RediSearch results and application models. This simplifies the development process and reduces the amount of boilerplate code required. The automatic data binding ensures that data is correctly populated into the desired objects, saving developers time and effort.

Parameterized Queries

The API supports parameterized queries, which can be pre-processed and executed multiple times with different parameters. This improves performance and security by preventing SQL injection attacks. Parameterized queries allow developers to write more efficient and secure code, optimizing the performance of their RediSearch applications.

Implementation Considerations

Implementing this LINQ-inspired API is a non-trivial undertaking, but it is also not infeasible. The key is to leverage LINQ metaprogramming techniques to translate the LINQ expressions into RediSearch commands. This involves analyzing the expression tree generated by the LINQ query and constructing the corresponding RediSearch query string.

One approach is to create a custom expression visitor that traverses the expression tree and generates the appropriate RediSearch syntax. This visitor would need to handle various LINQ operators, such as Where, Select, OrderBy, GroupBy, and Aggregate, and translate them into their RediSearch equivalents. The visitor would also need to handle data binding and parameterization, ensuring that the generated queries are efficient and secure.

Another consideration is the design of the API itself. It should be intuitive and easy to use, while also providing access to the full range of RediSearch features. This may involve creating a set of extension methods that extend the IQueryable<T> interface, or creating a custom query builder class that allows developers to construct queries in a fluent style.

Conclusion

The proposal to overhaul RediSearch with a LINQ-inspired API represents a significant step towards making RediSearch more accessible and developer-friendly. By leveraging the familiarity and expressiveness of LINQ, this new API promises to simplify query construction, improve code maintainability, and unlock the full potential of RediSearch for a wider audience. While the implementation effort is considerable, the benefits of a strongly-typed, LINQ-based API for RediSearch are substantial. This approach not only reduces the learning curve for new users but also empowers experienced developers to build more sophisticated and efficient search applications. The future of RediSearch with a LINQ-inspired API looks promising, offering a blend of power, flexibility, and ease of use.

For more information about Redis and its capabilities, visit the official Redis website.