One day in March 2025, complaints started coming in from our largest customer:
“The insights page isn’t loading.”
We’d always known this might happen someday. What surprised us was how soon—the feature had only been live for a few months.
After several rounds of mitigation, we stabilized the system. But the incident forced us to confront an uncomfortable truth:
We hadn’t hit a bug. We’d hit a design limit.
The Insights page was an analytics dashboard in our web application.
It consumed an activity event stream from an external system, and presented the data in three main ways:
At first glance, this looked like a perfect fit for a time-series database, and InfluxDB popped up as we were searching for one.
We made the choice of InfluxDB for our datastore based on these considerations:
The second consideration played a major role in ruling out alternatives that were not available in AWS. It also incentivized us to use as much of Influx as possible to fit all the use cases.
From our experience developing and debugging, we can now pin down what was wrong with our approach.
The Blueprint of a Time-Series Point
In InfluxDB, every individual piece of data is called a point. Conceptually, a point represents a single row of data that is uniquely identified by its primary key, which consists of the timestamp and the specific tag set associated with it. You can visualize a point as a single row in a ledger that must contain four specific components:
• Measurement: This acts as the container or table name. For example, in a fitness tracking system, there can be one measurement called "heart_metrics" for cardiovascular data and another called "activity_metrics" for movement-related data. Each measurement groups similar points together.
• Tags: These are the metadata labels, stored as key-value pairs of strings. They identify the "who, what and where" of a record. For example: activity_type, location, etc. Crucially, InfluxDB indexes tags, making them incredibly fast to query and filter.
• Fields: These represent the actual values being measured, such as distance_km, calories, etc. Fields are not indexed.
• Timestamp: This is the anchor of the record, a nanosecond-scale Unix timestamp that orders all data by time.
Series cardinality is the total count of unique series keys in a database, which are formed by the unique combination of a measurement and its tag set. When high-granularity data—such as unique identifiers, full log messages, or nanosecond timestamps—are stored as tags, they create a massive number of unique series keys. This state is often referred to as cardinality explosion.
We had two very different query patterns when reading data for our analytics views:
The first was InfluxDB’s sweet spot, but the second was not.
We ran both against the same datastore anyway.
That decision came back to bite us.
We were dealing with significant complexity in our read queries, compounded by the fact that we were using Flux (InfluxDB v2's functional pipeline language). Unlike SQL, Flux is a procedural pipeline—range() |> filter() |> join()—which meant the query planner had less room to optimize our logic, and every workaround was harder to debug. (This is now moot for new projects, as InfluxDB v3 supports SQL.)
While the language made things difficult, it wasn't the root cause. The fundamental mistake was treating two different workloads as one: blending entity-centric querying with time-series analytics.
We were careful in choosing keys and using the least amount we needed to avoid a cardinality explosion. Still, cardinality reached 2.8 million keys within three months.
At 2.8 million unique series, we were approaching InfluxDB's recommended limit of 10 million keys—leaving little room for growth. High cardinality impacts the system in two critical ways:
(1) Memory pressure from maintaining indexes for millions of series
(2) Query performance degradation because the database must scan through more series to find relevant data, even with filters applied. This meant our queries were both memory-intensive and slow.
Additionally, we faced a complexity explosion in our queries. The "mental burden" of maintaining these scripts was matched only by the "operational fragility" of the database instance itself.
Here is an example of one of our queries:
The functionality was:
Provide a comprehensive performance overview for all items associated with a specific channel.
For each item, return:
Options:
The query looked like:
For clients with the most amount of data, the query would time out - taking more than a minute to complete, even with a low number of concurrent users.
Our biggest mistake wasn’t choosing a “bad database.” It was treating two different workloads as one.
This includes stable identifiers, attributes, and mappings: ownership, grouping, hierarchies, and “belongs to” relationships. Even if it rarely changes, the access pattern is relational:
That’s join-heavy work. Relational systems (or similar) are built for it: flexible joins, constraints, and fast entity-centric queries.
This is high-volume, append-heavy data: a timestamped stream of events, readings, status transitions etc., per entity.
The dominant access pattern is time-bounded aggregation, typically involving:
This is where time-series databases excel: ingestion and fast time-range aggregation.
We didn’t fail because we stored time-based observations in a time series database. We failed when we tried to answer relationship-heavy, join-like questions directly from a high-cardinality time-series model—things like:
Those queries behave like warehouse queries. Forcing a time-series database to serve them pushed it into an OLAP/relational role it wasn’t designed to play.
We removed InfluxDB from the critical path for entity-centric views, while keeping it for the time-series graphs.
For much of the entity-centric views, we started consuming REST APIs instead, that the external system provided on top of the event stream. We then backed the APIs with a caching layer. The caching introduces a minor consistency issue—data in the cache may be slightly stale—but we mitigate this by setting a short Time-To-Live (TTL) on cached entries.
This simplified the system and brought down the response time to avoid timeouts, as a short-term solution.
Effectively, we brought down response time from ~60 seconds to ~20 seconds for a cache miss, and ~5 seconds for a cache hit. This was good enough for our users.
The REST APIs were not very fast, and the growing cardinality would need to be handled for a scalable solution.
For a longer-term solution, we considered a few options like
InfluxDB Tasks - This is a feature in InfluxDB that allows you to run queries on a schedule, and write the results back to the database. This is useful for compacting data. However, compacting would not result in the desired performance improvement because the cardinality would remain the same and only the number of points per key would be reduced. The resulting speed-up would be good - estimated around 2.5x, but the result, ~30 seconds, would still be far beyond the ~5-second threshold that users find acceptable for interactive applications.
A DB migration approach would have required re-architecture and added dual-write complexity in the case of hybrid approaches. Considering the cost involved, we ultimately decided not to go down this road, and instead try to accommodate this in our centralized analytics engine with Snowflake that is under development.
When evaluating a time-series database, don’t just look at:
Consider the Total Cost of Ownership:
If your data is primarily relational—or if your queries can’t be cleanly constrained by time—InfluxDB is probably the wrong tool.
Just because you can fit your data into a time-series shape doesn’t mean you should; always validate your query patterns before committing to a database engine.
Also consider hybrid systems, where you have a time-series database for time-series data, and a relational database for relational data. Realize that you may not be able to fit all your query patterns into a single database, and that's okay.
Some tools like TimescaleDB and QuestDB are designed to bridge the gap between the two, by allowing you to store time-series data in a relational database, and query it using SQL. They are worth looking into.
| Scenario | Why InfluxDB Struggles | Better Alternative |
|---|---|---|
| High-cardinality data with many dimensions | Creates millions of unique series, causing memory pressure and query performance degradation | OLAP databases or data warehouses |
| Current state queries | "Latest value per entity" queries become expensive with high cardinality | Relational databases with proper indexing, or Redis for caching |
| Aggregations across non-time dimensions | Grouping by multiple attributes (not time-based) creates complex Flux pipelines | OLAP databases or data warehouses |
Validate your query patterns and scale requirements, and consider all trade-offs—
performance, cost, and complexity—before committing to a solution.
The bigger lesson here goes beyond InfluxDB: architecture decisions need to reflect how a system will actually be queried, used, and scaled. At KeyValue, we approach product engineering with the same principle this experience reinforced: understand the real workload first, then design the architecture around it.
InfluxDB is a purpose-built time-series database designed to store and query timestamped data such as application metrics, IoT sensor data, monitoring data, and other event streams. It is particularly suited to high-volume writes and time-range aggregations.
InfluxDB is typically used by writing timestamped data with measurements or tables, tags, fields, and timestamps, then querying that data over specific time ranges. Depending on the InfluxDB version, queries can use SQL, InfluxQL, or Flux; InfluxDB 3 supports SQL and InfluxQL.
InfluxDB is a purpose-built time-series database rather than a traditional relational SQL database. Query support depends on the version: InfluxDB 2 commonly uses Flux or InfluxQL, while InfluxDB 3 supports SQL and InfluxQL.
InfluxDB is primarily used to store and query time-series data, while Grafana is a visualization and observability platform that creates dashboards from data stored in systems such as InfluxDB. Grafana can connect directly to InfluxDB as a data source.
High cardinality occurs when a dataset contains a very large number of unique series, often because tags contain highly variable values such as unique IDs. At scale, this can increase memory usage and negatively affect query performance. In this case, cardinality reached 2.8 million series keys within three months.
InfluxDB may not be the best fit when your workload relies heavily on entity relationships, joins, current-state queries, or aggregations across many non-time dimensions. In the case discussed in this blog, combining these workloads with time-series analytics resulted in increasingly complex and slow queries.
