Valkey-Search (BSD-3-Clause), provided as a Valkey module, is a high-performance Search engine optimized for AI-driven / Search / Analytics / Recommendation System related workloads. It delivers single-digit millisecond latency and high QPS, capable of handling billions of vectors with over 99% recall as part of vector searches. It also provides support for hybrid / pure non vector workloads including Numeric, Tag, and Full-text searches.
Valkey-Search allows users to create indexes and perform searches, incorporating complex filters. Users can index data using either Valkey Hash or Valkey-JSON data types. The vector queries support Approximate Nearest Neighbor (ANN) search with HNSW and exact matching using K-Nearest Neighbors (KNN).
Valkey-Search’s ability to search billions of vectors with millisecond latencies makes it ideal for real-time applications such as:
FT.CREATE
FT.DROPINDEX
FT.INFO
FT._LIST
FT.SEARCH
FT.AGGREGATEFor a detailed description of the supported commands, examples and configuration options, see the Command Reference.
A Valkey index is similar to a table in a relational database. The rows of the table are Valkey keys and the columns of the table are populated from data extracted from those keys. Query expressions are constructed like a filter that uses the data in the columns to identify keys. The actual implementation is based on content-driven indexes, not linear filtering. Each index is given a unique name by the application and each Valkey database has a separate namespace of indexes.
Indexes exist separate from the Valkey database. Updates of the database trigger updates of one or more indexes which are done by background threads. Query operations are also performed by background threads optionally switching to the main thread in order to access the Valkey database. The consistency model between these two domains is further described below.
Indexes are created through the FT.CREATE command
which defines the rows and columns of the table-like abstraction. The
command creates an empty index which can then be populated with data.
Applications don’t directly put data into an index, rather mutation
operations on keys within the declared keyspace of an index
automatically update the index with the value of that key. This update
operation happens at the key level, i.e., even if only part of a key is
mutated, the entire key is updated in the index. In other words, on an
index of hash keys, a command like HSET which modifies a
field of a key causes the entire key to be updated.
Index updates are side effects of data mutation, based on prefix matching. A Valkey keyspace notification is used to capture a copy of the data associated with a key mutation. If this key belongs to multiple indexes, then the captured data goes into the mutation queue for each index independently. The client which executed the mutation command is blocked until all indexes are updated.
Processing of the mutation queue does not respect the order of mutations. If multiple clients are updating different keys the updates may happen in any order not necessarily related to the order in which the mutation commands were executed. If a previous update for the same key is already in the mutation queue then these can be combined. An exception to the reordering rules applies to keys which are updated as part of a multi/exec block or a Lua script. These keys are always ingested as a group which cannot be split or reordered.
The automatic ingestion only applies to keys which are modified
after the creation of an index. But what about keys that exist
before the creation of an index? These keys cannot be instantly
inserted into the index. By default, the creation of an index initiates
an internal background process which scans the keyspace to locate and
insert preexisting keys into the index. This process, known as
backfilling, can be monitored via the FT.INFO command. The
backfill process runs once after the creation of the index. Once it has
completed it will not be initiated again. The backfill process does not
directly change the behavior of an index. Applications may continue to
modify indexed keys and they will be updated in the same way as if no
backfill was executing. Query operations can freely be executed, but
will only contain the results from keys which are currently in the
index, i.e., all keys modified after creation of the index and
some of the keys that existed before creation of the index.
The backfill process can be quite lengthy on a large system, even if
no keys are found. Applications that know there are no preexisting keys
or that no preexisting keys need to be inserted into the index, can skip
the backfill process by specifying SKIPINITIALSCAN on the
FT.CREATE command.
The snapshot process (save or full sync) only partially preserves the
state of the backfilling process. The process is able to save the fact
that a backfill is in progress, but does not save the backfill cursor
(because a SCAN cursor isn’t valid across reloads). Thus on
reload a backfilling index must restart the backfill at the beginning.
However, because the indexes for vector fields are saved and restored,
the indexed content of vector fields (the really slow part) is
preserved.
Query commands operate by blocking the client and sending the query to the background threads to locate a set of keys. Then the client is unblocked and control resumes on the main thread which can access the database to generate the final result. If a key is modified by another client while a query is in progress then that key may or may not be included in the query result. In no case will a key be returned which does not satisfy the query expression.
A generated RDB file (either due to an explicit save or full-sync operation) contains index definitions (index metadata), any vector field indexes, and a list of the keys currently in the index. On a load operation, each index is recreated from the definitions, any vector field indexes are reloaded, and any non-vector fields are rebuilt from the loaded Valkey database using the list of keys. If the index had a backfill in progress at the time of the save, then on completion of a load a backfill will be initiated for it.
Indexes are node-local. Each node, regardless of whether it’s a primary or a replica maintains its own index independently. Indexes on replicas are updated by key mutations transmitted on the replication channel and thus are subject to replication lag just like the Valkey database itself. No additional replication channel traffic is generated for updates. The index backfill process is also node-local, meaning just because one node (a primary) in a replication group has completed its backfill the other nodes may not have.
Search fully supports cluster mode and uses gRPC and protobuf for intra-cluster communication, requiring system configuration to make this additional port available. The gRPC port address is set based on the main Valkey port address. The gRPC port number is usually the Valkey port number plus 20294 (for the default port: 6379 + 20294 = 26673), unless the Valkey port is 6378 in which case the offset is 20295 (6378 + 20295 = 26673 also).
In cluster mode, Valkey distributes keys according to the hash algorithm of the keyname. This placement of data is not affected by the presence of the search module or any search indexes. Since search commands operate at the index level – not the key level – search is responsible for dealing with the distribution of data, performing intra-cluster RPC to execute commands as needed. Thus the application interface to search operates the same in cluster and non-cluster mode.
Search uses a simple architecture where index definitions are replicated on every node but the corresponding index only contains the data which is co-resident on that node. Index update operations remain wholly local to a node and will scale horizontally (save/restore operations also wholly node local). Vertical scaling is also effective because of the multi threaded architecture of search.
Query operations are performed by one node of each shard on its local index and the results are transparently merged together to form a full command response. Query operations are subject to increasing overhead as the cluster shard count increases, so they may scale sub-linearly with increasing shard count.
Search recognizes that certain data patterns can be optimized. In particular, it recognizes that if the data for an index is confined to a single slot, then only a single node needs to perform a search operation. Operations on single-slot indexes do not experience increased overhead as cluster size increases and thus will scale horizontally, vertically, and through replicas.
Client-side routing of query operations can have a profound effect on performance. For indexes with data on all shards, performance is improved if the query operations are distributed across the cluster. Any well-known load balancing algorithm should be fine, i.e., round-robin, random, etc. For indexes that are confined to a single slot, routing a query to a node that contains that slot is optimal. To simplify the logic of a client, the system restricts the assignment of index names as follows.
Index names without a hash tag are considered to be cluster-wide
indexes (even if the prefix list is confined to a single slot). Index
names with a hash tag are considered to be single-slot indexes. The
prefix list of a single-slot index must consist solely of prefixes which
contain the same hash tag or an error will be generated by the
FT.CREATE command.
The Search architecture relies on having identical index definitions distributed across the cluster. This is implemented with an eventually consistent cross-shard protocol. The protocol relies on a Merkle-tree checksum of all indexes defined on a node being broadcast over the cluster bus periodically. Nodes which discover a mismatch in the checksum contact each other and negotiate a resolution using version numbers and last-writer-wins timestamps, one index at a time. If a node loses the negotiation for an index, it will delete its version of the index and recreate it using the winning definition.
On top of the eventual consistency machinery the individual commands also perform additional consistency checks on the involved index, typically retrying operations until consistency is achieved or a timeout occurs, terminating the command with a consistency error message.
The metadata mutation commands (FT.CREATE and
FT.DROPINDEX) use the consistency machinery described
above. The commands operate by mutating the local copy of the metadata
and then triggering the convergence protocol. If convergence cannot be
achieved within a bounded period of time the command is terminated with
an error. No attempt is made to undo any failed metadata mutation. The
most likely cause of failure is a shard-down or network partition
situation.
The FT.INFO command has options that allow aggregation
of index statistics and status across the cluster.
The query operations: FT.SEARCH and
FT.AGGREGATE can only be executed by nodes that share the
same index definition and slot ownership map. Cross-shard query commands
contain a checksum of the coordinator’s index definition and slot
ownership. If a receiving node’s index checksum or slot ownership
checksum mismatches then the query is rejected and the coordinator will
retry the operation. If a timeout occurs then by default an error is
returned. The SOMESHARDS option of the
FT.SEARCH command can be used to override this behavior to
allow a result to be generated if only a subset of the cross-shard query
operations succeed. The INCONSISTENT option of
FT.SEARCH can be used to allow results from nodes with
different views of the cluster.