Search index

DataTier 2 · build and shipPhase 2 · Make it a store

A pre-organized copy of your data built so it can be searched very fast, instead of scanning every item each time.

As a catalogue grows, finding items by scanning every row for a matching word gets slow and gives poor results, with no ranking and no typo tolerance. Users expect fast, relevant search anyway. A search index exists as a separate structure built for text lookup, so a query like kacchi biryani returns good matches in milliseconds instead of a full table scan.

Finding "biryani" among 500 products by checking each one is slow, and hopeless at millions. A search index solves this the way a book's index does: the words are organized in advance so a lookup jumps straight to matches.

The index is built and updated separately from the main database. That is why a brand-new product can take a moment to appear in search: the index has to catch up.

A junior PM, stuck

A restaurant partner added Beef Tehari back to their menu ten minutes ago, and now they are calling because search cannot find it. The item is clearly saved, I can see it on the menu screen, but typing tehari into search returns nothing. The dev says index lag, not a bug, and I do not know if that is a real answer or a brush-off.

It is a real answer, and you can prove it to yourself in one look. Search does not read the products table directly; it reads a separate copy called the index, and that copy updates on its own cycle, a beat behind. Below is the products table, the search index next to it, and the exact query the partner ran. Five minutes with these three and you will see why the row exists but the search comes back empty.

The products table, the search index, and the query that returns nothing
products table, the source of truth
id | name | price | in_stock
p_1042 | Kacchi Biryani (Family Pack) | 649 | true
p_1043 | Beef Tehari | 280 | true
p_1044 | Morog Polao | 320 | true
p_1047 | Daily Veg Tiffin | 150 | true
the search index, a separate copy built for lookups
token | product_ids
biryani | p_1042
kacchi | p_1042
morog | p_1044
polao | p_1044
veg | p_1047
(5 rows)
the partner's search, right now
GET /api/search?q=tehari
{ "count": 0, "products": [] }

Click a step to see the lines it points at.

Filing an index-lag delay as a search bug. The row is saved and the index will catch up on its next cycle; a ticket that says search is broken sends an engineer hunting for a fault that does not exist.
Promising instant search results. Indexes refresh on a schedule, so a just-added item is not findable the same second; tell partners the real lag, about two minutes, instead of implying it is immediate.
Checking search instead of the source table. If you look only at the empty search result you cannot tell a lagging index from a failed save; open the products table first to see whether the row is even there.
Assuming a new searchable field is free. Making items findable by a brand-new attribute means changing what goes into the index, which is real build work, not a setting someone flips.

Tell the partner: "The item is saved and live on the menu now. Search reads a separate index that refreshes every couple of minutes, so tehari will start returning it shortly; this is lag, not a lost item." You checked the source table before trusting the search, which is how you tell index lag from a real bug.

If search feels slow or misses results, the index is the conversation. Making new fields searchable (say, by brand) means changing what goes into the index, which is real work.
"Why doesn't my new product show up in search?" is usually indexing lag, not a bug.

"New items take about two minutes to appear, the index refreshes on a cycle."

Appears in Phase 2, Make it a store.