Finding Bathroom Faucets with Embeddings

Abstraction: Using CLIP embeddings to build image-similarity product navigation tool

Key points:

• Embeddings place data in a high-dimensional contextual map (vector); CLIP from OpenAI embeds both images and text enabling cross-modal similarity search
• Built a decision-tree faucet finder from ~20,000 product images: user selects a faucet, app shows similar options via repeated nearest-neighbor queries
• Stack: llm CLI (Simon Willison) for embedding generation, SQLite storage, datasette + datasette-faiss for the query interface, deployed to Fly.io
• Text search works via CLIP: "Gawdy", "Bond Villain", "Nintendo 64" all return contextually matching faucets
• Key limitation: inconsistent image backgrounds (isolated vs. in-situ) skew similarity results; domain metadata filters would substantially improve quality

Connections: Clip · Simon Willison · Embeddings · Vector Similarity Search · Semantic Search

Source: https://www.dbreunig.com/2023/09/26/faucet-finder.html