How Spotify Made Data Discovery a Great Product (PM Perspective)

Photo by Heidi Fin on Unsplash

In data-driven companies, we often assume the biggest challenge is lacking data.

But at Spotify, the real problem was the opposite: they had too much data — yet no one could find it.

To solve this, Spotify built Lexikon — an internal data discovery and catalog platform designed to help data scientists quickly find, understand, and connect with the people behind the data.

Even after launch, data discovery remained one of the most complained-about pain points. Instead of treating it as a simple search problem, the Spotify team approached Data Discovery as a real product challenge.

This article breaks down Spotify’s journey from a product manager’s perspective (especially useful for Data PMs and AI PMs) — covering problem diagnosis, user intent frameworks, key design decisions, and lessons that remain highly relevant in the age of AI Agents and RAG.

When Data Explodes, Search Alone Isn’t Enough

In 2016, as Spotify migrated to Google Cloud Platform (GCP), the number of datasets in BigQuery exploded. At the same time, the company dramatically increased hiring of data scientists and analysts.

This led to classic problems:

• Datasets lacked clear ownership and documentation
• Insights were trapped in silos and rarely shared across teams
• Data scientists worked with 25–30 different datasets per month, spending too much time just trying to find the right ones

In 2017, Spotify launched the first version of Lexikon. It gained solid adoption (~75% of data scientists used it regularly, ~550 monthly active users). However, follow-up surveys revealed that data discovery was still a major bottleneck.

Instead of declaring victory, the team dug deeper into user behavior.

Key Insight #1: Not All Searches Are the Same — Understand User Intent

Spotify applied its world-class music discovery expertise to data discovery, distinguishing between two modes:

Low-Intent Discovery (Exploratory)

Users don’t know exactly what they’re looking for — common for new hires or those starting new projects.

Solution: Complete redesign of Lexikon’s homepage with personalized recommendations:

• Popular datasets across the company
• Team-relevant datasets
• Recently used datasets
• “You might find this useful” suggestions

Interestingly, complex NLP or recommendation models weren’t necessary. Simple heuristics based on usage statistics performed extremely well. The homepage recommendations now drive ~20% of table traffic.

💡 PM Insight: Before jumping to advanced AI, validate whether simple usage signals are already sufficient.

High-Intent Discovery (Goal-Oriented)

Users have a clear target — e.g., a specific table, schema field, project, or colleague’s dataset.

Solutions:

1. Improved Search Ranking — Significantly increased weight on popularity (top datasets account for most usage).
2. New Entity Pages — Expanded beyond datasets to Schema Fields, BigQuery Projects, People, and Teams.

Example: Searching “track_uri” leads to a dedicated schema field page showing all popular tables containing it. These entity pages are used by ~44% of monthly active users.

💡 PM Insight: Great search products don’t force everything into one box — they understand intent and create multiple discovery paths.

Key Insight #2: Knowledge Lives in People, Not Just Documents

Spotify initially followed a classic codification strategy (turning knowledge into documents). They soon realized that when people couldn’t find data, they still asked colleagues.

Solutions:

• Expert Discovery: Searching keywords like “Podcast” or “Retention” now surfaces not only datasets but also relevant experts (dataset owners, dashboard owners, research authors). Experts are ranked by production signals (creating/owning assets) rather than just consumption.
• Slackbot Integration: When someone shares a Lexikon link in Slack, the bot automatically expands a rich preview with owner, usage stats, key fields, etc. This increased Lexikon link sharing by ~25%.

💡 PM Insight: Don’t fight human connection — design tools that enhance it and build a true Knowledge Network.

Key Insight #3: Finding Data ≠ Knowing How to Use It (The Last Mile)

Many users still got stuck after finding a dataset.
Spotify added critical “last mile” features:

• Field Usage Statistics: Shows query count and unique users per column — helping quickly identify important fields.
• Real Query Examples: Replaced static manual examples with actual recent queries (~25% of users view them).
• Commonly Joined Tables: Shows tables frequently joined with the current one (~15% of users use this).

💡PM Insight: The last mile often determines whether a product is truly adopted.

5 Product Lessons from Spotify

• Understand User Intent before building features
• Usage signals often beat complex AI (at least initially)
• Search is about building trust, not just relevance
• Knowledge management is about networks, not just documents
• Never underestimate the Last Mile

Results

• Data scientist adoption: 75% to 95%
• Monthly active users (MAU): ~550 → ~870
• Average sessions per user: 3 → 9
• Data discovery: No longer a top pain point

Looking Ahead: AI Agent and RAG Era Knowledge Management

In the age of AI Agents and RAG (Retrieval-Augmented Generation), enterprise knowledge systems must evolve beyond passive retrieval.

Traditional RAG is mostly passive: Query → Retrieve → Generate. Agentic RAG introduces real agency — enabling agents to break down tasks, perform multi-turn retrieval, validate information, and connect with both data and people when needed.

Spotify’s Lexikon offers a powerful blueprint for the future. Successful knowledge platforms will need to excel at three core capabilities:

• Find: Precisely locate the right data and entities
• Understand: Grasp business context, logic, and relationships
• Connect: Link to the right experts and prior experience when needed

When these three elements come together, knowledge systems transform from simple data catalogs into true decision infrastructure — helping both humans and AI make faster, more reliable decisions.

If this article helped you, I’d really appreciate your claps 👏 and follow — it encourages me to keep writing more PM deep dives.

What’s your experience with data discovery at work? Drop a comment below — I’d love to hear your thoughts! 😊

This article reinterprets Spotify’s 2020 Engineering Blog post through a product management lens: How We Improved Data Discovery for Data Scientists at Spotify.