Author(s): Niranjan Akella

Originally published on Towards AI.

Slice Filters & Per-Tenant IDF: Search That Actually Makes Sense for Multi-Tenant Apps | Feat Qdrant

So you’ve built a SaaS product. You’ve got dozens (or hundreds) of customers, each with their own dataset, and you’re using Qdrant as your vector store. Smart move. But then you hit the multi-tenancy wall with one collection, many tenants, and search results that are… off.

YUPP! Not broken, just subtly wrong in ways that are hard to explain to customers.

Qdrant 1.19 ships two features that directly address this. Slice filters and pertenant IDF. They sound unrelated but they solve two sides of the same problem, making search and data operations work correctly when your collection is shared across many tenants.

Let’s get into it.

1. The Multi-Tenant Mess

Okay so the standard advice for multi-tenancy in Qdrant is: put everyone in one collection, add a tenant_id field to each point's payload, filter on it at query time. Easy, scalable, works.

client.scroll(
 collection_name="documents",
 scroll_filter=models.Filter(
 must=[
 models.FieldCondition(
 key="tenant_id",
 match=models.MatchValue(value="acme_corp"),
 )
 ]
 ),
)

This works great for retrieval isolation. Tenant A never sees Tenant B’s data. But here’s the thing, the moment you add keyword search (BM25/sparse vectors) to the mix, something breaks quietly in the background.

BM25 scoring depends on IDF, Inverse Document Frequency. The idea is simple: rare terms should score higher than common ones. A document matching “cryoablation” is probably more relevant than one matching “the”. IDF quantifies this rarity. The formula looks like:

IDF(t) = log( (N - df(t) + 0.5) / (df(t) + 0.5) + 1 )

Where N is total documents and df(t) is how many documents contain term t. When df(t) is small relative to N, IDF is high. Makes sense.

Now here’s where it goes wrong. Say you have two tenants sharing a collection:

• Acme Corp (medical software): 10,000 clinical notes. The word “diagnosis” appears in 9,800 of them.
• Legal Firm LLC (contract analysis): 10,000 contract documents. The word “diagnosis” appears in 3 of them.

Global IDF calculation:

df("diagnosis") = 9,803 total documents
N = 20,000
IDF("diagnosis") ≈ log(1.027) + 1 ≈ 1.03

That’s basically zero weight. “Diagnosis” looks like a stopword globally. So when Acme Corp’s users search for “diagnosis criteria”, a totally meaningful query in their domain.

The word “diagnosis” gets near-zero weight and results degrade. Meanwhile Legal Firm’s 3 documents mentioning “diagnosis” lose their appropriate high-rarity scoring because Acme’s data inflated the document frequency count.

Both tenants get wrong results and neither can figure out why. hmm..?

Elasticsearch has this problem too, by the way. Their solution is either one index per tenant (expensive) or dfs_query_then_fetch (adds a round-trip). Qdrant 1.19 does something cleaner.

2. Per-Tenant IDF: BM25 Scoring That Actually Makes Sense

Qdrant 1.19 adds an “idf” parameter to “search_[arams”. You pass it a filter, and that filter defines which documents are included in the IDF statistics computation for that query. It's computed at query time, tenant-specific, and totally independent from your retrieval filter.

First, setup. Your sparse vector field needs the idf modifier:

client.create_collection(
 collection_name="documents",
 vectors_config={},
 sparse_vectors_config={
 "text-bm25": models.SparseVectorParams(
 modifier=models.Modifier.IDF,
 )
 },
)

And your tenant field needs a payload index (Qdrant Cloud strict mode will reject unindexed filter fields):

client.create_payload_index(
 collection_name="documents",
 field_name="tenant_id",
 field_schema=models.KeywordIndexParams(
 type=models.KeywordIndexType.KEYWORD,
 is_tenant=True,
 ),
)

Now the actual query with per-tenant IDF:

results = client.query_points(
 collection_name="documents",
 query=models.Document(text="diagnosis criteria", model="qdrant/bm25"),
 using="text-bm25",
 query_filter=models.Filter(
 must=[
 models.FieldCondition(
 key="tenant_id",
 match=models.MatchValue(value="acme_corp"),
 ),
 models.FieldCondition(
 key="year",
 match=models.MatchValue(value=2024),
 ),
 ]
 ),
 search_params=models.SearchParams(
 idf=models.IdfCorpusParams(
 corpus=models.Filter(
 must=[
 models.FieldCondition(
 key="tenant_id",
 match=models.MatchValue(value="acme_corp"),
 ),
 ]
 )
 )
 ),
 limit=10,
)

Notice the intentional difference between the two filters. The query_filter narrows retrieval to acme's 2024 documents only. The idf corpus filter covers ALL of acme's documents – not just 2024. That's deliberate. You want IDF to reflect acme's full vocabulary so the rarity statistics are meaningful, even though you're only retrieving a subset.

The filter layering looks like this:

retrieval filter ⊆ IDF corpus filter ⊆ full collection
 (acme + 2024) (acme only) (all tenants)

Concrete before->after comparison:

Say acme has 10,000 docs and “diagnosis” appears in 9,800 of them, while “criteria” appears in only 200. With global IDF across both tenants, both terms get flattened because the legal corpus pulls “criteria” frequency up and the combined N changes everything.

Query termGlobal IDF (20k docs)

Per-Tenant IDF (acme only, 10k docs)

”diagnosis”~1.03 (near-zero)~1.03 (same — it’s common in acme too)”

criteria”~2.1 (inflated df from legal docs)~4.6 (correctly rare in acme)”

cryoablation”~9.2 (rare globally)~9.2 (same — legal barely touches this)

The real win is for terms that are common in one tenant’s domain but rare in another’s. Those are exactly the domain-specific terms that matter most for relevance. Without per-tenant IDF, those terms get scored incorrectly every time.

Yup, the fix is a single parameter. No extra round-trips, no index-per-tenant overhead, no client-side hacks.

3. Slice Filters: Deterministic Splits for Parallel Work

Okay, different problem now. You need to process your entire collection — re-embed everything with a new model, export data to a new system, compute statistics, run evaluations. The naive approach: scroll with a cursor, page through sequentially.

That works, but it’s slow for large collections, and if your job fails halfway through, you have to figure out where you left off. Want to parallelize? Now you need a coordinator to hand out cursor ranges, deal with workers finishing at different speeds, handle failures without double-processing.

Honestly, that’s a lot of scaffolding for what should be a simple parallelism problem.

Slice filters are Qdrant’s answer. A slice divides the entire collection into total disjoint buckets using deterministic hashing on point IDs, and lets you select exactly one bucket with index. The key properties:

• Deterministic: same {index, total} always returns the same points
• Disjoint: no overlap between slices, union of all slices = full collection
• Stable: no randomness, no seeds — the mapping is fixed

The filter in Python:

results, next_offset = client.scroll(
 collection_name="product_embeddings",
 scroll_filter=models.Filter(
 must=[
 models.SliceCondition(
 slice=models.Slice(index=3, total=8)
 )
 ]
 ),
 limit=500,
 with_payload=True,
 with_vectors=True,
)

Now here’s the parallel worker pattern. Four workers, each handling exactly 1/4 of the collection, zero coordination needed:

import concurrent.futures
from qdrant_client import QdrantClient, models

COLLECTION = "product_embeddings"
TOTAL_WORKERS = 4
BATCH_SIZE = 500

def process_slice(slice_index: int, total_slices: int) -> list:
 # Each worker creates its own client - thread-safe this way
 client = QdrantClient(url="http://localhost:6333")
 
 offset = None
 processed = []
 
 while True:
 results, next_offset = client.scroll(
 collection_name=COLLECTION,
 scroll_filter=models.Filter(
 must=[
 models.SliceCondition(
 slice=models.Slice(index=slice_index, total=total_slices)
 )
 ]
 ),
 limit=BATCH_SIZE,
 offset=offset,
 with_payload=True,
 with_vectors=True,
 )
 
 if not results:
 break
 
 for point in results:
 # your actual work here: re-embed, export, transform, etc.
 processed.append({"id": point.id, "payload": point.payload})
 
 if next_offset is None:
 break
 offset = next_offset
 
 print(f"Worker {slice_index} done: {len(processed)} points")
 return processed


with concurrent.futures.ThreadPoolExecutor(max_workers=TOTAL_WORKERS) as executor:
 futures = {
 executor.submit(process_slice, i, TOTAL_WORKERS): i
 for i in range(TOTAL_WORKERS)
 }
 all_results = []
 for future in concurrent.futures.as_completed(futures):
 all_results.extend(future.result())

print(f"Total processed: {len(all_results)}")

Each worker is completely independent. Worker 0 failing doesn’t affect Workers 1, 2, or 3. If Worker 2 crashes, you just re-run it with index=2, total=4 and it picks up the same points again from scratch. No shared cursor, no coordination overhead. Pretty wild right?

Note: within a single slice, you still use cursor pagination (next_page_offset). Slice filter doesn't replace pagination inside the slice – it just bounds the working set for each worker. So the pattern is: slice filter to split work, cursor to page through each worker's portion.

Wait, what about reproducible sampling? Same deal. Want a consistent 10% evaluation sample? Use Slice(index=0, total=10). It always returns the same 10%. Want a train/test split? Slices 0-7 for training, slices 8 and 9 for test. Run your pipeline six months later on the same data? Same split. That's actually pretty handy compared to ORDER BY RANDOM() LIMIT n approaches that give you a different sample every time.

One important caveat: if points are added or deleted between runs, slice membership can shift because hash assignments change as the collection evolves. The slice is stable for a point’s lifetime, not across collection mutations. For migration jobs where the source is read-only, this isn’t a problem. For ongoing pipelines on live data, design accordingly.

4. Putting It Together

Here’s a realistic scenario where both features matter at the same time.

You’re running a multi-tenant document intelligence platform. Customers upload their internal knowledge bases — legal teams, medical teams, product teams — and users search across them. You’re also running nightly jobs to re-embed everything with a new model and compute per-tenant quality metrics.

The search side uses per-tenant IDF so domain-specific terminology scores correctly for each tenant. A legal tenant searching “force majeure clause” gets IDF computed over their legal corpus, not diluted by the medical tenant’s completely different vocabulary. The medical tenant searching “differential diagnosis protocol” gets their medical terms scored against their corpus.

The batch processing side uses slice filters to parallelize the nightly re-embedding job across N workers. Each worker independently processes its slice, can be restarted independently on failure. No coordinator needed, no shared state, horizontal scaling is as simple as increasing the total count and spinning up more workers.

You can even combine both in a single query. Slice within a tenant for stratified sampling:

# Get a reproducible 25% sample of just acme's documents
client.scroll(
 collection_name="documents",
 scroll_filter=models.Filter(
 must=[
 models.SliceCondition(slice=models.Slice(index=0, total=4)),
 models.FieldCondition(
 key="tenant_id",
 match=models.MatchValue(value="acme_corp"),
 ),
 ]
 ),
 limit=500,
)

This gives you 25% of acme’s documents deterministically. Useful for canary testing a new embedding model on a subset of a tenant’s data before full rollout.

5. Running It Yourself

Here’s a complete demo you can run locally. Start Qdrant first:

docker run -p 6333:6333 -p 6334:6334 \
 -v $(pwd)/qdrant_storage:/qdrant/storage:z \
 qdrant/qdrant

Install the client:

pip install qdrant-client

Then this script creates a two-tenant collection, inserts sample documents, and demonstrates both per-tenant IDF search syntax and slice-based parallel processing:

import concurrent.futures
from qdrant_client import QdrantClient, model

client = QdrantClient(url="http://localhost:6333"
''

# Setup collection with sparse vectors + IDF modifier
client.recreate_collection(
 collection_name="demo",
 vectors_config={},
 sparse_vectors_config={
 "text-bm25": models.SparseVectorParams(
 modifier=models.Modifier.IDF,
 )
 },


client.create_payload_index(
 collection_name="demo",
 field_name="tenant_id",
 field_schema=models.KeywordIndexParams(
 type=models.KeywordIndexType.KEYWORD,
 is_tenant=True,
 ),


# Insert sample documents for two tenants
medical_docs = [
 {"id": i, "tenant": "medical", "text": f"Clinical case {i}: diagnosis and treatment protocol for patient"}
 for i in range(1, 201)
]
medical_docs += [
 {"id": 201, "tenant": "medical", "text": "Hospital billing and insurance reimbursement procedures"},
 {"id": 202, "tenant": "medical", "text": "Staff scheduling and shift management system"},


legal_docs = [
 {"id": i + 300, "tenant": "legal", "text": f"Contract analysis {i}: force majeure clause and indemnification terms"}
 for i in range(1, 201)
]
legal_docs += [
 {"id": 501, "tenant": "legal", "text": "Personal injury case: patient diagnosis injuries from accident"},
]all_docs = medical_docs + legal_docs
# NOTE: In production, sparse vectors would be populated by BM25 tokenization
# (fastembed, Qdrant Cloud inference, or your own tokenizer).
# For this local demo we insert with empty vectors to show the API structure.
points = [
 models.PointStruct(
 id=doc["id"],
 payload={"tenant_id": doc["tenant"], "text": doc["text"]},
 vector={},
 )
 for doc in all_docs
]
client.upsert(collection_name="demo", points=points)
print(f"Inserted {len(points)} documents")

# Demonstrate slice filter: parallel scroll across 4 workers
def count_slice(slice_index, total):
 c = QdrantClient(url="http://localhost:6333")
 count = 0
 offset = None
 while True:
 results, next_offset = c.scroll(
 collection_name="demo",
 scroll_filter=models.Filter(
 must=[
 models.SliceCondition(
 slice=models.Slice(index=slice_index, total=total)
 )
 ]
 ),
 limit=100,
 offset=offset,
 with_payload=False,
 with_vectors=False,
 )
 count += len(results)
 if next_offset is None:
 break
 offset = next_offset
 return count
total_slices = 4
with concurrent.futures.ThreadPoolExecutor(max_workers=total_slices) as executor:
 futures = {
 executor.submit(count_slice, i, total_slices): i
 for i in range(total_slices)
 }
 slice_counts = {}
 for future in concurrent.futures.as_completed(futures):
 idx = futures[future]
 slice_counts[idx] = future.result()
print("\nSlice distribution across 4 workers:")
for idx in sorted(slice_counts):
 print(f" Slice {idx}: {slice_counts[idx]} points")
print(f" Total: {sum(slice_counts.values())} (should equal {len(points)})")

# Per-tenant IDF query - this requires BM25 sparse vectors to be populated.
# On Qdrant Cloud with cloud inference enabled, the vector is generated automatically
# from the Document text at query time.
print("""
Per-tenant IDF query (requires BM25 sparse vectors populated via cloud inference):
results = client.query_points(
 collection_name="demo",
 query=models.Document(text="diagnosis criteria", model="qdrant/bm25"),
 using="text-bm25",
 query_filter=models.Filter(
 must=[
 models.FieldCondition(
 key="tenant_id",
 match=models.MatchValue(value="medical"),
 )
 ]
 ),
 search_params=models.SearchParams(
 idf=models.IdfCorpusParams(
 corpus=models.Filter(
 must=[
 models.FieldCondition(
 key="tenant_id",
 match=models.MatchValue(value="medical"),
 )
 ]
 )
 )
 ),
 limit=5,
)
""")

Run this and you’ll see the slice distribution across 4 workers. It won’t be perfectly equal (hash distribution variance) but it’ll be approximately total_docs / 4 per slice.

For the full BM25 search demo with actual scored results, sign up for Qdrant Cloud which handles BM25 sparse vector generation at ingestion time via cloud inference — no separate embedding step needed.

The Qdrant docs have the full reference on both features including the REST API format and edge cases around combining slice with nested filters (short version: put the slice condition in an outer must, not inside a nested block).

So yeah, two features that solve real problems. Per-tenant IDF fixes a subtle but meaningful bug in BM25 scoring for shared collections, the kind of thing that’s hard to diagnose but clearly degrades search quality in multi-domain setups. Slice filters give you a clean primitive for parallel batch work that scales horizontally without coordination overhead.

Both are available now in Qdrant 1.19. If you want to stay on top of what’s coming in future releases, the Qdrant newsletter is worth subscribing to — the release notes are actually written by engineers and tend to be pretty honest about tradeoffs.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI