In my last post, I explained the CTX/ICC construct—the beating heart of ContextOS that lets your code scale across multiple physical computers as naturally as using threads and processes on a single machine. But I left you with a question: how does ContextOS actually make this work reliably across a distributed system?
Before I dive into how this all works, it is critical for you to be aware that the end user of ContextOS doesn't need to know any of this! All of these systems are in place to facilitate a smooth experience for you, the developer. You need to understand the ContextOS VFS as much as you need to understand how the Linux kernel module loading system works. You don’t need to understand.
With that said, I am presenting this design to help people understand that ContextOS is not just another hacked together UI on top of Kubernetes. It is an entirely new system, designed to make your life easier.
Let's start with the ContextOS Virtual File System, or VFS. Usually in a distributed system we look for a single source of truth. The VFS is more than that, it is a source and a declaration of truth. The hardware nodes that make ContextOS both report state to the VFS, but also come into consistent state with the declared state in the VFS.
Therefore, the VFS is not just another configuration database. It's an active, living source of truth that fundamentally changes how distributed systems manage state. And to understand why it matters, we need to talk about what's broken in the way everyone else does this.
Let's start with how Kubernetes manages cluster state, because it's what most people know.
Kubernetes uses etcd, a distributed consistent key-value store that represents the state of the cluster at any given point of time—what nodes exist, what pods should be running, which nodes they're running on, and more. The API Server is the only Kubernetes component that connects to etcd; all other components must go through the API Server to work with the cluster state.
Here's the flow: When you create a pod using kubectl, the API Server validates the request and persists it to etcd, then invokes the Scheduler which decides where to run the pod and returns that to the API Server, which persists the new state back in etcd.
This sounds reasonable until you hit scale. Etcd's storage space is limited with a recommended maximum of 8 GiB, and a large dynamic cluster can easily generate enough data to reach that limit—if your etcd data store exceeds its configured storage limit, it can't accept any new data and your Kubernetes cluster will become inoperable.
Etcd uses the Raft consensus algorithm which requires a leader node to act as single source of truth—whenever a read or write request is made to any node, the request is forwarded to the leader. This centralization becomes a bottleneck. You're paying the coordination tax on every single operation, even when that coordination isn't necessary.
The real problem? Kubernetes treats all state the same way. Configuration data, active runtime state, desired state, temporary locks—everything flows through etcd via the API Server. This creates a system that's both over-engineered for simple operations and under-optimized for the operations that actually matter.
The ContextOS VFS takes a fundamentally different approach. Instead of treating all state uniformly, we split it into two categories: Active State and Desired State.
Active State (/hw, /sum, /run, /itg, /lk) tells you what IS happening:
Desired State (/mid, /ctx, /ztb, /sys) tells you what SHOULD happen:
Here's the key insight: Active State and Desired State have completely different performance and consistency requirements.
Desired State changes rarely and benefits from strong consistency—when you define what you want, you want every node to agree on it. Active State changes constantly and needs to be fast—hardware metrics, load balancing decisions, and temporary locks don't need global consensus, they need to be available NOW.
This is where I need to be direct about something the industry has gotten wrong: the obsession with strong consistency everywhere.
According to the CAP theorem, any distributed data store can provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Since network partitions are inevitable in distributed systems—networks fail, latency spikes occur, nodes become unreachable—partition tolerance is non-negotiable.
So you're choosing between Consistency (C) and Availability (A). MongoDB is a CP data store—it resolves network partitions by maintaining consistency while compromising on availability. Kubernetes' etcd makes the same choice.
But here's what they don't tell you: The speed of light imposes fundamental limits on distributed systems—a round-trip between New York and Tokyo takes approximately 200ms at light speed, and real networks are much slower due to routing, processing delays, and congestion.
Strong consistency requires waiting for acknowledgments from all replicas before confirming a write, which means global operations become as slow as the slowest network path and user-facing operations suffer from cross-continental latency.
For infrastructure that needs to make decisions at scale—scheduling workloads, routing traffic, allocating resources—this overhead is death. You don't need every node to agree on the exact CPU utilization of a server 3,000 miles away before you can schedule a container locally. You need good-enough information, fast.
ContextOS uses eventual consistency for Active State. Eventually consistent means that a change may not become immediately visible on all servers at the same time, but given enough time and no new updates, all replicas will converge to the same state.
In practice, convergence happens in seconds. That's fast enough for infrastructure decisions, and it means the system stays available and responsive even when network partitions occur.
The VFS is organized like a filesystem—different locations define different system components. But under the hood, individual files are stored as binary JSON blobs which represent the desired or established state of parts of the system.
This might sound like an implementation detail, but it's actually critical. Binary JSON gives us:
When a node needs to know about the state of the system, it reads from the VFS. When hardware metrics update or a CTX spins up, that information is written to the VFS. The VFS synchronizes this state across all nodes using eventual consistency, ensuring everyone has a unified (if slightly time-lagged) view of the system.
Here's a question you might be asking: if the VFS stores active hardware state in /hw and running summaries in /sum, how does that information get there?
The answer is the Props and Status systems—pluggable interfaces that continuously scan hardware and baseline system usage and load them onto the VFS.
Props (short for "properties") collects all hardware data from servers: CPU cores, memory, GPUs, storage, network interfaces, specialty compute devices. This information is critical for the scheduling stack—ContextOS needs to know what resources are available to make intelligent placement decisions.
Status monitors active system usage: CPU load, memory consumption, network throughput, disk I/O. This feeds the Scaler system, which determines when resources need to be added or removed based on actual utilization.
Both Props and Status are pluggable—they can be easily extended to detect and expose additional hardware or usage patterns. This extensibility is key to supporting new hardware types (like AI accelerators) without rewriting core ContextOS code.
These systems continuously update /hw and /sum on the VFS, providing the real-time awareness that enables smart scheduling and scaling decisions. And because this uses eventual consistency, these updates don't create coordination bottlenecks—they just flow through the system naturally.
The VFS isn't just a database. It's the nervous system of ContextOS—the mechanism by which every node maintains situational awareness of the entire infrastructure.
When you write a CTX definition to /ctx, the VFS kicks off the binpacker, scaler, and Distributed State Machine. When hardware metrics update in /hw, the scheduling stack uses that information to make better placement decisions. When the integrity system detects a failure in /itg, recovery routines automatically initiate.
All of this happens without centralized coordination, without consensus bottlenecks, and without the operational complexity of managing etcd clusters and API servers.
The VFS proves that you can have a distributed system that's both reliable and fast—you just need to stop treating all state the same way and embrace eventual consistency where it makes sense.
The VFS gives us a unified view of distributed state, but state alone doesn't get work done. In my next post, I'll dive into the Distributed State Machine and Software Drivers—the systems that take desired state from the VFS and turn it into running infrastructure.
The DSM is one of the powerhouses of ContextOS. It gives us a consistent, idempotent, loosely coupled, autohealing way to execute complex clustered routines. Combined with our Software Driver system, it allows ContextOS to deploy and configure distributed services with reliability that makes traditional configuration management look primitive.
If the VFS is the nervous system, the DSM is the hands that actually do the work. And it does it in a way that's fundamentally different from anything else out there.
Tom Hatch is the CEO, CTO, and Co-founder of ContextOS. Previously, he created SaltStack, used by 20% of Fortune 500 companies.