How Pydantic Logfire Builds Single-Table Observability on OpenTelemetry

Pydantic, best known across the Python ecosystem for its ubiquitous data validation library, launched its cloud observability platform, Logfire, in October 2024.

As the company expanded from data validation into infrastructure, founder Samuel Colvin argued that the developer experience in the observability market was badly outdated:

observability feels like it’s stuck where the rest of infra was 15 years ago

The business case was just as candid. In an interview with TechCrunch, Colvin admitted that building an observability platform was a direct bid to monetize the brand equity and developer trust built over years of open-source stewardship:

all the people who need Pydantic, the validation library, also need observability

Pydantic’s mission is to bring the developer ergonomics of its validation library to observability. OpenTelemetry—the vendor-neutral industry standard—gives every vendor’s backend the same uniform telemetry format.


The Single-Table Model: Treating Logs as Zero-Duration Spans

In traditional observability setups, developers constantly toggle between siloed systems: querying distributed traces in one tool, then hunting through application logs in another. Even when both capture the exact same outage, correlating them across disparate interfaces requires manual stitching.

In distributed tracing, a trace represents the end-to-end journey of a request as it traverses services, while a span is an individual timed unit of work within that journey (such as a database query or an HTTP call), nested into a tree hierarchy.

Logfire unifies logs and traces at the data model layer. Under the hood, it eliminates separate tables for logs and traces, consolidating everything into a single table named records (pre-aggregated numeric metrics are stored separately in metrics). The official SQL reference spells out this core design:

Each row in records is a span or log (essentially a span with no duration)

In Logfire’s data model, a log is simply a span with zero duration. Logs and traces naturally share the same trace IDs, span IDs, and parent-child tree hierarchy.

When debugging, a simple query like WHERE is_exception AND level >= 'error' immediately surfaces failures (is_exception flags records containing a Python exception object). Pulling the full call tree is as straightforward as filtering by the shared trace_id—no cross-database joins required to reconstruct the complete story.

import logfire

logfire.configure()

with logfire.span('process order {order_id}', order_id=42):
    logfire.info('validating payload')
    # Business logic execution

In the minimal example above, the logfire.info call becomes a zero-duration child span, automatically attached beneath the active unit of work. Every event inherits the surrounding structured attributes, allowing developers to query rich context using standard SQL.


Open Standards and Data Portability: No Proprietary Agent Lock-In

A perennial risk with monitoring tools is vendor lock-in driven by proprietary agents. Migrating down the road often means rewriting instrumentation code across every microservice.

Logfire bets entirely on OpenTelemetry (OTel), adopting the standard OTLP protocol for ingestion. The official documentation makes it clear that the standard OTel Collector works out of the box without bespoke adjustments:

As Logfire is a fully compliant OpenTelemetry SDK and backend it does not require any special configuration

Getting data out is equally unconstrained. The Python SDK supports dual-routing telemetry to other OTel-compatible backends (Protobuf over HTTP), and Pydantic provides both an S3 backup guide and a Query API for programmatic data retrieval. Switching platforms requires only updating the Collector’s exporter pipeline—leaving application logic untouched.

  1. Application (Python / Rust / TypeScript)
  2. OTLP (Protobuf over gRPC / HTTP)OpenTelemetry Collector
  3. OTLP ExportLogfire Backend (Ingestion & Live View)
  4. JSON ShreddingParquet Files on Object Storage + DataFusion SQL

The official Python package is fundamentally “an opinionated wrapper around OpenTelemetry.” It distills OTel’s boilerplate—exporters, formatters, and context propagation—into an out-of-the-box one-liner while preserving the freedom to migrate using standard OpenTelemetry down the road.


Storage Architecture: The Parquet and DataFusion Analytics Pipeline

Logfire stores telemetry in Apache Parquet files on object storage (such as AWS S3), using Apache DataFusion as its SQL query engine (official technical posts cite ClickHouse as an architectural peer and conceptual reference point, rather than an adopted dependency).

Given how wildly telemetry attributes vary, storing them as raw, monolithic JSON strings inside Parquet would crush query performance with heavy I/O and CPU deserialization overhead.

To solve this, Logfire employs a dynamic decomposition technique called JSON shredding: for each Parquet file, the ingestion engine identifies the top 128 keys by total volume or frequency and dynamically promotes them into native, strongly typed physical columns. Only lower-frequency attributes remain tucked inside a catch-all JSON column.

This layout unlocks two core OLAP optimizations in DataFusion:

With full-payload JSON decoding eliminated, Pydantic’s engineering deep-dive, “How shredding JSON is giving Logfire 1000x query speedups,” reports speedups of up to 1,000x on targeted queries.

The query interface uses DataFusion SQL, which closely mirrors PostgreSQL syntax. Developers can filter and aggregate telemetry using familiar SQL, without having to learn a proprietary domain-specific language (DSL).


Pricing Model and Positioning

Evaluating Logfire requires clarity on its licensing: Logfire itself is a proprietary, closed-source commercial service, not open-source software.

Pydantic’s Helm chart repository and enterprise documentation explicitly define its proprietary and commercial licensing:

Self-hosted Logfire is an Enterprise offering that requires a contract and payment. The chart source in this repository uses the MIT License, but the Logfire container images are private and need commercial access.

Only the language SDKs and Helm charts are open source. Deploying Logfire on Kubernetes therefore requires paying license fees while also operating the underlying databases, object storage, and authentication infrastructure yourself.

Pricing follows a cumulative usage-based model billed per “record,” where every span, log, and metric counts as a single record. The official pricing plans include a free tier of 10 million records per month, while paid tiers (such as the Team plan starting at $49/month) bundle record allowances and charge $2 per additional million records.

IMPORTANT

The free tier has a hard cap: The 10-million-record monthly allowance on the free plan is a strict hard cap. Once exhausted, Logfire stops ingesting incoming data rather than billing overages. Teams evaluating Logfire in production must set up usage alerts or prepare a billing upgrade path in advance.

Compared to traditional monitoring tools (such as Datadog) that charge fixed monthly fees per monitored host or node (per-host), usage-based pricing produces vastly different cost dynamics across workloads:

DimensionUsage-Based (Logfire)Fixed Per-Host (e.g., Datadog)
Pricing MetricTotal records ingestedMonitored hosts or nodes
Container Horizontal AutoscalingScales with record volume (traffic), independent of node countScales linearly with node count, regardless of traffic
High Traffic on Few NodesCosts climb with record volumePredictable costs tied to node count
Cost Control LeversControlled directly via sampling ratesRequires node consolidation or plan downgrades

In terms of market positioning, Logfire’s product line does not cover infrastructure security or low-level network monitoring, focusing instead on application-level logging and tracing.

For Python-centric teams looking to avoid vendor lock-in via standard OpenTelemetry and query telemetry with standard SQL, Logfire’s focused, single-table application observability offers a lightweight, pragmatic sweet spot.