Tessera
Folgen

Ben Trent

@benwtrent.bsky.social

Doer of things | Builder of things | software engineer @elastic

39 Posts139 Follower65 Folgt
Doug Turnbull@softwaredoug.bsky.social · last yr.

Next in the series of building a search engine from scratch - we focus on hybrid retrieval with @benwtrent.bsky.socialof Elastic. How do you add filtering to a vector search index? I'll code. He'll yell at me. maven.com/p/430592/hyb...

Hybrid search live coded from scratch

RAG systems all use vector databases. HNSW (Hierarchical Navigable Small Worlds) is the most common algorithm. If you want to build RAG, you should appreciate how this algorithm works (Missed previous...

maven.com

Ben Trent@benwtrent.bsky.social · last yr.

Indexing and merging times are getting better for #Apache #Lucene vector search. Lucene has a read-only segment architecture. One of the drawbacks of this approach is throwing away previously completed work when merging HNSW graphs. Well, this got better :)

Bild
Ben Trent@benwtrent.bsky.social · last yr.

It's wonderful to see practical & important programming work. Debugging concurrent programs is incredibly difficult, here is a bug found in Apache Lucene by the CMU Pasta Lab using their new Fray testing framework www.elastic.co/search-labs/...

Concurrency bugs in Lucene: How to fix optimistic concurrency failures - Elasticsearch Labs

Thanks to Fray, a deterministic concurrency testing framework from CMU’s PASTA Lab, we tracked down a tricky Lucene bug and squashed it

elastic.co

Ben Trent@benwtrent.bsky.social · 2y ago

My team wrote a new backing algorithm for our BBQ indices, called Optimized Scalar Quantization. Here is a high level overview of its implementation in Elasticsearch (and soon Apache Lucene). www.elastic.co/search-labs/... for the math nerds, skip to Tom's blog: www.elastic.co/search-labs/...

Optimized Scalar Quantization: Even Better Binary Quantization - Elasticsearch Labs

Here we explain optimized scalar quantization in Elasticsearch and how we used it to improve Better Binary Quantization (BBQ).

elastic.co

Ben Trent@benwtrent.bsky.social · 2y ago

Something a little different from my typical blogs. This line of code in Apache Lucene took me 3 days to write. For fixing bugs, it's about the journey, not necessarily the destination. www.elastic.co/search-labs/... (the cover art was provided by one of my kids :))

Lucene bug adventures: Fixing a corrupted index exception - Elasticsearch Labs

Sometimes, a single line of code takes days to write. Here, we get a glimpse of an engineer's pain and debugging over multiple days to fix a potential Apache Lucene index corruption.

elastic.co

Ben Trent@benwtrent.bsky.social · 2y ago

Our Better Binary Quantization (BBQ) index in Elasticsearch has a new backing algorithm. Better(er) recall & query speed for vector search. Its a natural evolution of our scalar quantization. Shipping soon. It's pretty neat www.elastic.co/search-labs/...

Understanding optimized scalar quantization - Elasticsearch Labs

In this post we explain a new form of scalar quantization we've developed at Elastic that achieves state-of-the-art accuracy for binary quantization

elastic.co

Ben Trent@benwtrent.bsky.social · 2y ago

I cannot adequately express how proud I am of the #Elasticsearch team for delivering this. It is a humungous engineering achievement and the results of (metaphorical) blood, sweat, and (maybe real ;) ) tears. go.es.io/3CVo82X

Do less with serverless: Elastic Cloud Serverless — Now GA

Elastic Cloud Serverless is the easiest way to start and scale your capabilities in search, observability and security. Built on a reimagined Elasticsearch architecture, it ensures low-latency queryin...

go.es.io

Ben Trent@benwtrent.bsky.social · 2y ago

Better Binary Quantization (BBQ) is now in 8.16. When it comes to index time, query time, and recall, we found it way better than the old faithful Product Quantization (PQ) in Elasticsearch vector search. Here is how we compared them when developing BBQ: www.elastic.co/search-labs/...

Better Binary Quantization vs. Product Quantization - Search Labs

Why we chose to spend time working on better binary quantization instead of production quantization in Lucene and Elasticsearch.

elastic.co

Ben Trent@benwtrent.bsky.social · 2y ago

Did you know Elasticsearch added the "retrievers" framework back in 8.12? As a further extension of _search, it gives extra flexibility that our traditional "query" API does not. Like RRF, global reranking, and multi-phased retrieval. It's now GA: elastic.co/search-labs/... [🧵 begin]

Elasticsearch retrievers are generally available with Elasticsearch 8.16.0! - Search Labs

Elasticsearch retrievers have gone through a significant revamp and are now generally available for all to use. Learn all about their architecture and use-cases.

elastic.co