ES Technical meeting
Menu
BLOG — your data, your business
8 min EN

Open-source data infrastructure: your data, your business

The proprietary software a company uses every day is itself built, on the inside, on open source. The hyperscalers build their products on open source. It isn't the risky option — it's the most solid, most audited foundation for a sovereign data infrastructure.

Opendome
Published

Why opendome builds on open standards, and why that decision is what guarantees, demonstrably, control and sovereignty over the data.

For anyone running an organization that operates with Salesforce, SAP, or Microsoft, the term "open source" may still evoke a vague notion: programmers' software, projects without guarantees, technology with no solvent company behind it to hold accountable. It's a widespread perception, and it's mistaken. It's worth correcting with precision, because open source is not the risky option next to proprietary software: it is, today, the most solid, most audited, and most reliable technical foundation on which a data infrastructure can be built. This document sets out why, with evidence.

The starting point is a fact that is rarely made explicit: the proprietary software an organization uses every day is built, internally, on open source.

Proprietary software runs on open source

In 2023, Salesforce standardized its global hybrid cloud infrastructure on Red Hat Enterprise Linux —an open-source operating system— to have "a more flexible and consistent foundation" at scale. SAP runs its solutions on that same Linux and on Kubernetes, the open-source container orchestrator that originated at Google. Microsoft Azure operates a majority share of Linux machines and contributes to thousands of open projects. The hyperscalers don't tolerate open source: they build their products on it.

These are not exceptions, but the operating principle of the internet. The web, as it works today, rests on open software that keeps it running securely, interoperably, and with no single owner:

  • Linux is the operating system of virtually every server in the world, including those of the hyperscalers themselves.
  • The Apache web server and the rest of the Apache Software Foundation projects —from Kafka to Spark— have for decades served a dominant fraction of the world's traffic and data processing.
  • PostgreSQL, the open relational database, is used by 55.6% of the world's developers according to the 2025 Stack Overflow survey —its largest annual growth on record— and by more than 48,000 companies, among them Netflix, Spotify, Uber, Reddit, Instagram, and Discord.
  • Kubernetes runs in production at 82% of IT organizations, with 5.6 million developers.

The scale of the investment the largest technology companies devote to open source confirms its strategic, not marginal, character. In May 2026, IBM and Red Hat committed 5 billion dollars and more than 20,000 engineers to securing enterprise-grade open-source software; IBM alone integrates more than 62,000 open-source packages. This isn't philanthropy: it's the recognition that open source is the infrastructure on which everything else is built.

The conclusion is unambiguous. The choice is not between "reliable proprietary" and "risky open." The open already sustains the proprietary. The relevant difference lies not in the reliability of the foundation, but in who controls the upper layer — and, consequently, who retains control over the data.

Open source as a decision about control, not cost

opendome builds on open source deliberately. It's not a cost decision, but a strategic one at the center of the proposition, and it follows from one principal reason: it is the only demonstrable way to guarantee that the customer's data is, in fact, the customer's.

When the data infrastructure rests on open standards, there are no proprietary license agreements that condition access to one's own information, no data-retention clauses, and no technical dependence on a provider with the ability to one day monetize access to what the organization generated. The data resides in an open format, readable by any engine in the ecosystem. If the organization decides to drop its provider, the data remains in a format any other actor can read. The cost of leaving tends to zero — not as a marketing argument, but as a technical property of open formats.

This connects with a concrete, documented legal risk. The US CLOUD Act empowers US authorities to compel a provider headquartered in that country to hand over data, regardless of where it is physically stored. In June 2025, Microsoft's French subsidiary acknowledged this under oath before the French Senate: it could not guarantee the sovereignty of data against US authorities, not even for data hosted in France. Building on open source, on the customer's own infrastructure and within its jurisdiction, is not a values stance: it's risk management. And only open source makes it verifiable.

The pieces, and who runs them in production

None of the pieces opendome builds on is experimental. They are mature projects, in production at the world's most demanding organizations, with broad communities and, in many cases, with solvent companies providing professional support on them. The continuity of these projects does not depend on opendome. They are described one by one below.

The three core pieces

Apache Iceberg — the open table format. It's the piece on which the data is centralized and stored. Iceberg was created at Netflix in 2017 and donated to the Apache Software Foundation in 2018. By 2025 it has become, without dispute, the de facto standard for open table formats: Apple, Netflix, LinkedIn, Airbnb, Adobe, Bloomberg, and Expedia run it in production, among many others. Apple has deployed it across hundreds of data teams and contributes actively to the project. And the three major cloud providers —AWS, Microsoft, and Google— along with Snowflake and Databricks, all support it. The reason the world has coalesced around Iceberg is exactly ours: data ownership and the absence of lock-in. By using a format tied to no platform, the organization keeps control and the freedom to switch engines whenever it wants. Airbnb reported a 50% compute saving and a 40% reduction in ingestion time after migrating to Iceberg.

Lance / LanceDB — the format for multimodal and AI data. It's the piece that links unstructured data —documents, emails, images, transcripts— with the rest, and the one that enables the semantic search agents need. Lance is an open columnar format built from the ground up for AI, created by one of the original authors of Pandas (the most widely used data library in the world in Python). Its adoption is a who's who of generative AI: Midjourney, Character.ai, Runway, World Labs (Fei-Fei Li's), Luma Labs, and the AI unit of ByteDance run Lance in production. Netflix uses it for multimodal search. WeRide reported a 90x improvement in the productivity of its ML developers. LanceDB raised a 30-million-dollar Series A in 2025: there's a solvent company behind it and, since the format is open, no dependence on that company is created.

The semantic layer and the metrics layer — the piece we extend. It's the layer that turns data into business knowledge. Where the two previous pieces centralize and store, this one describes: it establishes that a numeric field is "revenue in euros," that a row is a "customer," and that two unconnected tables relate as "the customer who signed this contract." The result is not a catalog of tables, but a semantic graph — a formal representation of which entities exist in the business, what attributes they have, and how they connect to each other. It is, strictly speaking, an ontology: the model of the reality of the business on which both people and agents pose their questions.

The technical core of this layer is a metrics layer, and it's worth dwelling on it because it's where much of the system's formal rigor resides. We start from the industry's open standard —MetricFlow, the engine that underpins the dbt Semantic Layer, distributed under the Apache 2.0 license— and we have extended it for the agentic case. How it works illustrates why a metrics layer is qualitatively superior to defining metrics by hand or inside each BI tool:

  • Formal, versioned definition. Entities, dimensions (the axes you group by: time, geography, category), and measures (the numeric aggregations) are declared once, in code, reviewable and auditable in version control. A metric like "revenue" stops having as many definitions as there are dashboards: it has one, and it's the same for every consumer.

  • Composition over a graph. Entities act as nodes and relationships as edges of a graph. The engine knows the join paths between tables and, faced with a question —"revenue by customer and by month"—, generates the optimal SQL on the fly, automatically resolving the necessary joins even when the query traverses dozens or hundreds of tables and several hops between them. It does so, moreover, structurally avoiding the classic errors of dimensional modeling —fan-out and chasm joins— that a human analyst introduces easily and that silently corrupt the results.

  • First-class metric types. The metrics layer supports as formal constructs not only simple aggregations, but ratios (revenue per customer), expressions (transactions minus cancellations), cumulative metrics (weekly active users), and time comparisons (month-over-month growth). A query that written by hand would exceed 150 lines of SQL is reduced, optimized, to a third — and to a declarative definition anyone can read and approve.

  • A single truth for BI, apps, and agents. Because the definition lives in the semantic layer and not in the consuming tool, the same metric —identical, governed, and tested— feeds a dashboard, an embedded application, or an AI agent. For the agent this is decisive: it receives the business meaning already resolved and consistent, instead of having to infer it. The entire sector today recognizes this layer as a condition for AI to give reliable answers, to the point that MetricFlow is developed within the Open Semantic Interchange (OSI) initiative, the industry's effort to standardize semantic definition and interoperate across platforms — which guarantees that the definitions modeled today are portable tomorrow, without rewriting.

On solvency: dbt is the industry-standard data transformation tool, used by JetBlue, HubSpot, Vodafone, and GitLab. On the semantic-layer side, Cube (Apache 2.0) has more than 400 companies on its platform —among them Brex, Coinbase, Intuit, and Toyota— and Brex chose it over the proprietary alternatives to build its AI financial analyst. These are pieces with formal rigor, a broad community, and solvent companies behind them.

The rest of the architecture, piece by piece

Although the three above are the heart of it, every layer is built on equally solvent projects:

  • The data in the customer's cloud: Kubernetes, Helm, OpenTofu, and CloudNativePG over S3-compatible storage. Kubernetes is operated by BBVA, ING, and Deutsche Bahn; CloudNativePG has the backing of the Red Hat community.

  • Centralization of corporate sources: Meltano + Singer (with more than 600 connectors) and Trino, the distributed query engine. Meltano originated at GitLab; Trino is used by Netflix, Airbnb, LinkedIn, and Salesforce itself. They allow data from Salesforce, SAP, or Dynamics to be brought into a single open lakehouse, without paying a fee per connector.

  • Structured and unstructured data unified: in addition to Iceberg and Lance, pgvector (used by Supabase and Microsoft) and Apache Tika (the content extractor used by Alfresco) so that documents and tables coexist and relate.

  • Data navigable from an ID: dbt models the data from raw to curated, so that from a customer_id you reach its orders, cases, contracts, or policies in milliseconds, whatever the original source.

  • Atomic security per access: Open Policy Agent (OPA) —used by Google, Netflix, and Goldman Sachs— together with Trino's row- and column-level control, so that each access is authorized by the combination of agent, purpose, entity, and field. It's not a general role: it's a specific key.

  • Immutable observability: Prometheus, Grafana, Loki, Tempo, and OpenTelemetry, the observability standard maintained by Microsoft, Google, and AWS. They produce the auditable access log required by DORA and the EU AI Act, with no additional development.

  • Structural EU compliance: isolation via NetworkPolicy, EU residency, atomic auditing, and supply-chain signing with Sigstore (from Google and GitHub). Compliance by design, not as a coat of paint added at the end.

An ecosystem of professional services, not an unsupported project

The objection that most frequently holds back the adoption of open infrastructure at the executive level is one of support: if a problem arises, who do you call?. The answer is that there is a mature ecosystem of thousands of companies providing professional services —support, implementation, managed operation, contractual guarantees— on exactly these pieces.

That ecosystem is led by top-tier companies. Red Hat —acquired by IBM— built a multi-billion-dollar business entirely on support for open software, and is the provider on which Salesforce and SAP rest their own infrastructure. Around each core project there are specialized, solvent companies: those that back PostgreSQL, those that maintain the Kubernetes distributions, those that sustain Iceberg, Trino, or Cube. The recent joint 5-billion-dollar commitment from IBM and Red Hat to secure enterprise open source is the measure of the institutional backing behind it.

It's also worth being precise about the nature of that service, because this is where the difference from the proprietary model is most relevant. Providers of services on open technology operate in a competitive, open market, and that disciplines both their technical quality and their price. They compete for the customer on the basis of their competence —because anyone can read, audit, and operate the same code— and not on the basis of an imposed dependence. The result tends to be a service more technical and less extractive than that of the partner ecosystems orbiting the proprietary platforms, whose business depends, structurally, on deepening the customer's dependence on the vendor. Where the proprietary partner has an incentive to perpetuate the tie, the open-services provider can only keep the customer by deserving it.

This reconfigures the relationship with the provider at an essential point. With a proprietary stack, the dependence is structural: faced with a price increase, a change of terms, or the decision to monetize access to one's own data, the only way out is a costly migration. With an open stack, the relationship is voluntary. An organization chooses its provider —opendome, or any other— for the quality with which it integrates, operates, and services these pieces, not out of an inability to leave. The data remains in open formats that any other actor in the ecosystem can read.

That, ultimately, is the property that defines open source well applied to data infrastructure: it doesn't reduce dependence on technology —no organization operates without it— but it eliminates dependence on a single provider to access what is one's own. The infrastructure belongs to the customer; the formats belong to the common domain; and the service is provided by whoever does it best, in a market where competing demands excellence and not captivity.

Sources

KEEP READING

Continue with

Blog · Opendome

Guardrails and data security: what each one is for

"The agent has guardrails, so the data is protected." That sentence conflates two things operating on different planes. Guardrails watch what the agent says; they don't govern what it accesses. Only structural, reproducible access control protects the data — because it isn't statistical: it's deterministic.

Read
Blog · Opendome

You'll pay to access your own data

With agents that hold no seat and open no screens, per-user licenses stop making sense. Salesforce, SAP, and Microsoft are shifting to consumption pricing: charging for every operation an agent runs against the data. The problem is that you generated that data, and now operating on it costs.

Read
Blog · Opendome

AI memory is in the data

Business memory already exists: it lives in the data. The only memory that should belong to the agent is procedural.

Read
Want to go deeper?

Book a 30-minute technical meeting for your team.

We will show you how the Cell Model fits your stack, using your current data lake with no migration required.

Book a technical meeting