Leading this project was the hardest thing I’ve done in my career. It dragged me out of my comfort zone and into work I didn’t expect to do. Software engineering is a craft you can only learn by doing, and nothing teaches you faster than being thrown in at the deep end.
Looking back, some of my most useful contributions weren’t technical. My focus shifted from writing code to evaluating tradeoffs, making difficult decisions, setting the roadmap, communicating plans, and keeping the team aligned as the work evolved. With AI, code has become cheaper to produce, but context, judgement and leadership are more valuable than ever.
In addition to indexing text for search, we added a document data model that supports flexible fields and types, a schema to describe how documents should be indexed, and powerful new APIs. All of these features were threaded through a mature database without disrupting a single customer in production.
The result is a search engine that can find exact matches using full-text search, similar items using vector search, or combine both approaches in a single query to find the most relevant results. Whether you need billions of documents, millions of namespaces, thousands of requests per second, or millisecond response times, Pinecone can handle your most demanding search workloads.
Thanks to everyone who got this over the line:
Rajat Tripathi,
Austin DeNoble,
Ivan Kelly,
Clint Wang,
Yaakov Sokolik, Jennifer Hamon,
Nico Alba,
Jenna Pederson,
Arjun Patel,
Manish Talreja,
Keith Corbalis,
Chris Bolton,
Jörg Schad , Ph.D.
Jenna Pederson reposted this
🚀 Full-text search is generally available in Pinecone Database.
Semantic search has never been good at literal strings. Query a string literal like a part number and all the similar part numbers score about as high as the one you actually wanted, and your agent answers from whatever lands on top, whether the data is correct or not.
Full-text search solves this. BM25 keyword ranking now runs in the same index as your vectors. So, exact matches are returned and ranked at the top of the list.
Link in the comments.