Embedding model selection: discovery breadth over search precision

We tested an embedding model that scores better on search benchmarks and kept the one that doesn’t — because Pond’s value is cross-domain discovery, and a looser, more evenly distributed space surfaces more of those connections than a tightly clustered one.

Project: Pond Window: Dec 2025 – Mar 2026 Recorded: Apr 2026

Status: decided & held

At a glance

We needed an embedding model for production. A newer option — Google’s gemini text-embedding-004 — looked attractive: strong on search benchmarks and natively multimodal.
We compared it against the incumbent, OpenAI’s text-embedding-3-small, by running identical real queries from the library on two environments and looking at which sources the results came from. Evidence
The newer model clustered passages from the same source tightly. On one query a single source supplied 7 of 20 results; on another, one source held 4 of 12. The incumbent spread results across more sources. Evidence
We tried to tune the breadth back by loosening similarity thresholds. It helped one case (a dominant source dropped from 7 to 2) but a second case stayed concentrated. Evidence
Decision: keep the incumbent’s broader space. Recover exact-passage lookup with hybrid keyword search, not by switching models or maintaining a second vector index.
Accepted downside: weaker search-style precision, and no native multimodal embeddings — covered by describing images as text at import time.
The newer model also cost roughly 65% more per exploratory run and pushed about 2.5× more text into the agent’s context. Evidence
Revisit trigger: if we ever move to the tighter model, a hard per-source cap during result assembly becomes a prerequisite, not an option.

The problem

Pond is a synthesis layer over a personal library. Its job is to connect ideas across domains — a thread about attention might pull from neuroscience, architecture, and literary criticism.

The embedding space is the substrate underneath almost everything that makes that work: thread-building, proactive suggestions, conceptual framing, and the companion agent that explores the library. So the choice of embedding model isn’t a search-box detail; it shapes what kinds of connections the product can even see.

That reframes the usual selection criterion. The model that tops public search benchmarks is optimised for precise recall of a known passage — “find me that paragraph.” Pond mostly needs the opposite: surprising, cross-domain neighbours the reader didn’t go looking for. A model can be excellent at the first job and worse at the second.

We had already moved off a small self-hosted open-source embedding model to a hosted API earlier in the alpha, for startup time and memory reasons rather than quality. S-1 That set the incumbent baseline. The open question by early 2026 was whether to switch to a newer, more capable-looking hosted model before the space grew.

Decision map

Causal map. The model that looked “better” (H1) lost to the constraint surfaced by the probes: its concentration on a single source couldn’t be tuned away cheaply, and it cost more. Text alternative in the diagram description.

Timeline

Dec 2025

Move off the self-hosted model to a hosted API Evidence S-1

Operational, not quality-driven: faster startup and lower memory. This set OpenAI text-embedding-3-small as the incumbent. We run it at reduced dimensionality (768 of its native 1536) to cut storage and compute — already a deliberate “smaller is fine” choice.
12 Mar 2026

A/B test: identical queries, incumbent vs. newer model Evidence S-2

We ran the same real library queries against the incumbent and against Gemini text-embedding-004, then looked at the spread of source material. The newer model formed “gravity wells”: one rich source dominated a result set (7/20; 4/12 on a second query). The incumbent distributed similarity more evenly across sources.
12 Mar 2026

Probe: can thresholds restore the breadth? Evidence S-2

Loosening the newer model’s similarity thresholds pulled in more of the diverse tail — a dominant source dropped from 7 results to 2. But a second query stayed concentrated. The clustering was a property of the space, not only of where we set the cut-off.
12 Mar 2026

Cost and context observation Evidence S-2

The newer model’s exploratory run cost about 65% more, and its looser, larger result sets pushed roughly 2.5× more text into the agent’s context window — bumping against the model’s input limit far sooner.
12 Mar 2026

Decision lands and gets encoded Evidence S-3

We committed to the incumbent space the same day: consolidated every similarity threshold into a single, model-calibrated module and removed a dead search path. The cleanup is the decision made durable — the thresholds are now explicitly tuned for this model.
Mar – Apr 2026

Held Evidence S-4

The production configuration is unchanged a month on: same model, 768 dimensions. The decision stuck.

Hypotheses and experiments

Each hypothesis with the probe that tested it and the conclusion drawn. Results are qualitative — see Method and limitations.
Hypothesis	Probe	Expected signal	Observed	Verdict
H1 · The tighter model’s clustering hurts cross-domain discovery	A/B identical queries, compare source spread	Results concentrate on few sources	“Gravity wells” — 7/20 and 4/12 from one source	Supported
H2 · Threshold tuning can restore the breadth	Loosen similarity thresholds on the tighter model	Diversity returns across all test queries	Helped one case (7→2); a second stayed concentrated	Weakened
H3 · We need native multimodal embeddings	Reasoning about Pond’s value, not a measurement	Visual similarity is core to connecting ideas	Describing images as text at import covers the need	Rejected as a requirement
H4 · The newer model is cost-comparable	Compare run cost and context size	Similar cost and token footprint	~65% pricier; ~2.5× more context tokens	Rejected

Inference

The same looseness that helps discovery probably also inflates the agent’s context, since broader results mean more material retrieved per step. Inference The cost observation is consistent with this but doesn’t isolate it as the only cause.

Options and trade-offs

The chosen route against the serious alternatives. “Precision” here means search-style recall of a specific known passage.
Option	Discovery breadth	Search precision	Multimodal	Cost / context	Maintenance
Incumbent, broader space chosen	Broad, even spread	Weaker — recovered via keyword search	None; images described as text	Lowest	One vector index
Switch to the tighter model	Narrow; gravity wells	Strong	Native	~65% more; ~2.5× context	One index, but needs a per-source cap
Tighter model + per-source cap	Partially restored	Strong	Native	Higher	Added enforcement logic to maintain
Two vector indexes, one per model	Best of both	Strong	Native	Highest	Two pipelines to keep in sync

The dual-index option is the one that technically gets everything. We rejected it on maintenance and cost: two embedding pipelines, two columns to keep consistent, and double the embedding spend, to buy a precision we could approximate far more cheaply with keyword search.

The decision

Keep the incumbent, broader embedding space in production at 768 dimensions. Where a user genuinely wants to find a specific known passage, a hybrid search that blends semantic similarity with keyword matching handles it well — keyword matching is exactly what exact-passage lookup needs, and it’s cheap.

The acceptance, stated plainly: we took on a model that is worse at search-style clustering, because that weakness is recoverable and the strength we kept — breadth — is the thing the product is actually built on.

Deliberately not solved

The exploring agent has no hard source-diversity enforcement, only soft guidance. With the broader model that gap is mostly harmless because the space spreads results on its own. It becomes load-bearing only if we later switch to the tighter model — which is why a per-source cap is the named prerequisite for that switch. Inference

Evidence and implementation trail

Sources are generalised for public release — internal file names, identifiers, and exact tuning values are omitted. Each item below is referenced inline above by its label.

Source ledger

S-1 Version history · Dec 2025. The migration from a small self-hosted open-source embedding model to the hosted OpenAI API, recorded as an alpha change motivated by startup time and memory, with a one-off re-embed of existing content. Evidence
S-2 A/B session notes · 12 Mar 2026. Contemporaneous notes from running identical real queries on the incumbent and the newer model: the gravity-well counts (7/20, 4/12), the threshold-loosening result (7→2, second case unchanged), and the cost/context comparison (~65% more, ~2.5× tokens). Evidence
S-3 Version history · 12 Mar 2026. The commit that centralised all similarity thresholds into one model-calibrated module and deleted a dead search path — the change that encoded the commitment to this space. Evidence
S-4 Current configuration · Apr 2026. Production still set to OpenAI text-embedding-3-small at 768 dimensions; the embedding module notes the move off local models for startup and memory. Evidence

Outcome, open questions, revisit triggers

Outcome

The decision has held without change in the month since, which is the main signal we have that it was right for the product as it stands. Evidence

Open questions

The comparison was qualitative: a small number of real queries, scored by how results distributed across sources, not a formal benchmark. Unresolved We don’t have a measured discovery-quality metric — the gravity-well effect was visible and stark, but it was counted, not statistically characterised. A reproducible eval harness would let us re-run this the next time a model lands.

Revisit triggers

Adopting a tighter-clustering model for any reason — first build the per-source cap.
A shift where precise lookup matters more than discovery (e.g. a heavy search use-case).
A genuinely visual use-case that describing-images-as-text can’t cover.

Method and limitations

This devlog was reconstructed after the fact by Claude from version history, a contemporaneous decision note, and the current configuration. It was not written live alongside the work.

Internal file names, symbol names, ticket identifiers, and exact threshold values are intentionally generalised for public release. Model names are public products and are kept.
The A/B evidence comes from one dated session on a handful of queries; it is not a reproducible benchmark, and the numbers should be read as illustrative of a clear effect, not as precise measurements.
The final configuration was verified against current settings. The per-query retrieval behaviour was not independently re-benchmarked for this writeup.