This is another blog post that I've ended up banging my head against, writing and then discarding multiple, relatively long revisions. Unlike the last several, however, this is not about describing a technical component of Project MAD; it's about the opposite. The truth is, when you distill a distributed applications model down to its essence, it sounds so simple as to be irrelevant. And indeed, the model - absent proper support - might do little enough. It really needs to exist inside of a context that makes it more valuable.
And the funny thing is, a properly supported model for distributed applications is insanely valuable. There are thousands of examples, spread across hundreds or thousands of different domains, where we are having difficulty writing applications that combine the resources of two or more computers together. In the world today, we are expecting and hoping that companies and open-source projects will each create solutions to each of these examples on a case-by-case basis.
Those are what I call a special-case solutions to the problem of distributed applications. What I am searching for is the general-case solution, one that fits, if not all of these examples, most of them. A single technology would make it easier to create solutions for these problems, but alone, a distributed application model still expects - nay, demands - that programmers solve the hard problems involved in making applications work across long distances. Those hard problems are the kind of thing that need to be solved centrally, on an operating system level, so that each application programmer can go about doing the thing they are there to do.
But it's worth laying out the argument, as best I can, piece by piece.
Why Distributed Applications?
I said above that a properly supported model for distributed application would be insanely valuable. It's worth running through some examples of things that are wrong with modern computing that come down to, essentially, the lack of general distributed applications.
There are a lot of examples of simply inadequate hardware. My dear mother has a relatively recent laptop that isn't upgradable, and is now all but useless for a non-technical user. (I'm sure I could install Linux and it would work fine, but she doesn't want it) That's an example of something that's too cheap; but modern GPUs are getting too expensive because they are trying to get a single chip to do a massively parallel task all at once. If someday distributed applications (generically, not counting special cases) can split their GPU load among multiple processors as easily as they can put it on a single one, a lot of people will be a lot happier. Making applications distributed likewise helps with inadequate memory as much as inadequate CPU threading; you still need the same amount of memory, but today you may already have the excess, perhaps locked up in old laptops or desktops where there's no good way to use it.
It goes beyond computers that are poorly designed. Smart devices, including TVs, watches, glasses, car dashboards, VR headsets, and so on, are not generally meant to run an entire generic PC for their user, but they have enough hardware to serve as a decent front end - but the process for making good use of them is spotty at best and terrible at worst. It isn't even, to my knowledge, acknowledged that these are all examples of the same class of problem: finding a good way to make use of compute, display, and sensors distributed across many devices. That's part of what I'm trying to solve, frankly.
There's also ownership of data. Home cloud systems exist (I use one), but larger companies put a lot more development work into the proprietary versions, so those end up more advanced and user-friendly. A large part of making these cloud-compute systems, however, involves solving the fundamental problems of distributed applications, if only in a specific case - finding a free compute node, authenticating users, deploying and updating applications, error handling, and so on. If there is a general-case solution, and one that users can deploy at home or in small businesses with relative ease, then the value proposition of cloud computing isn't as strong. This not only relieves end users, who don't have to trust the companies, but it takes companies out of the business of both safeguarding and somehow monetizing other people's data, which is an unfortunate contradiction to be in charge of.
The last example I'll point out for now is administration. It's not trivial for non-technical users to manage home servers, not least because managing the server is simply a separate task from managing your PC, so it's something that home users need to explicitly remember to do. Where your PC might (with varying levels of obnoxiousness) force you to update apps, drivers, and services, currently any machine that isn't in the foreground must either do things automatically or… not do things automatically, and leave the user to their own devices. Distributed administration means that the state of the entire system is always in the foreground; if some part needs updating, it is the same whether that part is local or remote.
In the same vein, because the distributed application model that I'm proposing is about an application expanding to use remote resources, the applications are not split into separate client and server components that must be updated and managed separately. A unified model means that local and remote pieces are understood as parts of the same whole, and that includes installation, deployment, and updates. Between distributed OS updates and application updates, the entire environment that is being used - even though it is on multiple pieces of hardware - is all in scope at once, and all managed at once.
That, however, is all about distributed applications. You could make a distributed application that manages existing operating systems. But MAD is more than just the app model, which raises the next question.
Why a Distributed Operating System?
When I originally created MAD, I took it as granted that we would need a new operating system to handle the management and operation of a distributed system. Frankly, I merely assumed it; it's fair to wonder whether or not a distributed-first operating system is ultimately required. Indeed, if you browse my blog, you will find a lot of concepts that seem to be unrelated distractions from the core concept. And in some cases, that's probably exactly right; you'll never catch me saying that MAD is perfect. Given the name, that'd be kind of silly.
The distributed operating system concept is not the distributed application model. They are separate, and must be argued for separately. And quite frankly, I can envision a future in which distributed applications are simply run on top of existing operating systems with little or no change to those operating system. Frankly, managing distributed applications comes down to a software service; it doesn't require a clean slate. I imagine that a lot of people, assuming they managed to figure out what I'm saying among all the stumbling explanations here on my blog, would make that point themselves.
There are two fundamental counter-arguments I'll make.
First, the distributed operating system is a collection of standards, and there should be a collection of standards that applies to distributed applications even if they are being run on top of existing operating systems. To some extent, when I'm talking about the distributed operating system, I'm talking about the environment that exists inside of the distributed ecosystem itself, rather than the physical disk images on top of which the system itself operates. It's important that applications and the systems onto which they are deployed, can agree on some terms, like what resources are available, what deployment constraints an application fragment needs fulfilled, how standard functions and hooks work, and so on. It's astonishingly easy to imagine getting that wrong and creating two or more completely incompatible systems, or ones that will require serious shims and compatibility patches to be made to work later on.
The other argument to be made is that operating systems, even open-source and user-focused operating systems, have become bloated - not because of any technical failing on anyone's part, but because they were developed over time, and different components represent different ideologies regarding how a computer should be programmed. One needs only familiarize themselves briefly with any one of the many arguments about Linux init systems to see that we are operating a complicated system on top of mixed metaphors. I'm well aware that any OS I create or am involved in, will not be the final operating system, and assuming it survives, it will tend back towards the current state of chaos over time; thus, that itself can't be my argument.
But the various technologies I have placed generally under the OS category of Project MAD are all distillations. For example the System API Directory and Application Folders frameworks, together unify files on disk with the abstract concept of data, as provided by drivers, services, and applications. While I expect to have arguments about this framework with experts someday, the point (as I have made it in my last couple blog posts) is having a consistent metaphor that works for all data in a distributed system, from the smallest bits of application state, to LED light indicators exposed by drivers, to database sockets, to embedded applications and libraries. One possible interpretation of the SAD/AF framework (not necessarily the one I'll push for) implies that any variable stored persistently in any part of any application is addressable (if not necessarily accessible, or even listed) via the directory, uniquely, so that it would be the same from anywhere in the system. And at the same time, every compiled-in function would be not only addressable but callable from anywhere, complete with full type safety and crash handling.
It's not about actually doing that. That would be a safety and security nightmare. But if you have a data accessibility technology built into your distributed operating system that makes that level of access possible, it can handle anything less than that. If you can go that far, then you can manage every application process on every node of a distributed system, you can bit-bang every GPIO bus on every piece of hardware, and you can read every last byte from every disk. And critically, you can do it all from the filesystem, where everything accessible may also be listed publicly for you to peruse, so that you may better understand the system you're working on.
A single consistent metaphor that allows you to do everything that a distributed system can do - that's valuable, and it's a quantity that is not at all guaranteed in modern operating systems. Building the right constraints on top to prevent abuse… I'm not going to say it's a secondary concern. But I have some confidence that security experts won't have to strain themselves to find ways to block funny looking remote procedure calls, with suspicious senders and destinations. If nothing else, IT departments are already wrestling with similar problems.
However, the SAD/AF framework only works for applications within the distributed OS's purview. It is my concept that even a dedicated MAD OS image will have a number of services and drivers that are not accessible remotely under any circumstances, for security reasons - but if you simply have an ADA-compliant server on a random operating system image, almost none of the applications, services, or drivers of the host system will be exposed to the distributed system. Rather than an intentional screening of things that do not deserve to be accessed remotely, you simply have the common and average fact of incompatibility. And that's… fine. It'll probably end up necessary, one way or another.
But if you want to administer systems, if you want to control all their hardware, then you want the operating system image to expose as much as possible to the network (for authorized and authenticated users, of course). And there's a fairly obvious security hole - it's entirely plausible if you merely have a remote application service, which isn't well tied to the operating system, a remote application could start a local process which the service itself cannot regulate, or even detect, becoming an out-of-band attack vector. And while you may be able to count on normal OS tools to discover and handle such an incident, and you can have distributed administration tools that handle those OS tools, the added complexity makes the situation more precarious.
Meanwhile, because of the way distributed systems work, many nodes in your network may be used for nothing more than hosting remote processes. Modern OS images provide a large number of services and drivers that may not be needed or used; a compute-only network node could be trivially small, and have no exposed filesystem (or an immutable one) for remote applications to attack. While it's possible to set up, eg Linux, systems to run at this rudimentary level, most systems are still expected to be general-purpose, and their OS images reflect that. Systems set up for hosted compute only, especially if they have an out-of-band or self-update mechanism, may be all but untouchable to applications in a distributed system, in addition to being faster and simpler to operate.
The reason, of course, that most OS images today contain so many nuances is because the hardware they are running on top of can be so variable, and it's unclear how much of that hardware a given user might actually need. And that brings me to my last question:
Why Modular Hardware?
When I first envisioned MOS/DCA, adding a processor to a system meant connecting little hardware devices together; the system was made of modules that would daisy-chain infinitely, and each module's hardware included absolutely nothing except its payload, and the backbone networking chip. No general-purpose networking was involved here; it was dedicated pieces slotting together to make, effectively, a single desktop system out of hardware modules.
Hardware drivers under such a system, you understand, would be quite simple. A CPU-only chip has no need for any drivers except the chipsets that support the CPU itself (including a fan and temperature sensor), and the backbone. No generic networking, no printers or keyboards, no serial busses or parallel ports, no USB, no solid-state or spinning platter disk drives, no CD/DVD/Blu-ray drives. A compute-only node not only need not do anything else, it could not. No such accessory hardware existed.
In return for that simplicity, these modules were the very definition of commodity hardware. I imagined they would be churned out by the millions. In the modern world, I imagine them being built off of license-free technologies like RISC-V, driving the costs ever downwards; twenty years ago, I would have hoped for something similar, even if no such were commercially available at the time. Nowadays, Raspberry Pi has proven you can deliver a full computer for $30, as long as you don't mind handling exposed pins and solder.
If you could make every computer you own work better with a single $30 purchase, would you really stop at one? If plugging a raspberry pi into my parents' network was all it took to salvage my mother's failing laptop, she would have taken it in a heartbeat. And the $30 raspberry pi comes with a graphics chip, HDMI out, and other accessories that a dedicated compute node may not need. What really would be the ground floor, if you still want to end up with a unit that's good enough to make your distributed system better?
It goes beyond the compute only nodes. Outside of general-purpose busses like USB, any task-specialized node only needs a few additional drivers. Some will be complex, like a graphics unit with a GPU and multiple output ports, but if you only want to add, say, an AI accelerator chip, or a collection of data drives, the system image that runs that entire node has very few additions on top of the base layer; you will be certain that no other capability exists or is being used, because that's all that the system is.
If computer parts become that level of commodity hardware… well, I'll be honest, it will harm research and development of new chips. That's a nice way of saying that chipmaking companies, which are already on rocky ground nowadays, could lose significant licensing revenue and sales, so long as people can get a cheaper alternative. Some chipmakers, especially in the GPU space, are cagey about letting others understand how their technology works to preserve their competitive advantage - but commodity GPUs don't need to be individually powerful, if they can work together. A no-nonsense alternative that's easier to program for and keep updated over the years, which gets to the same place with quantity instead of density, would certainly make that policy of stonewalling your own developers harder to maintain.
I'm trying hard not to paint those outcomes as purely good for the consumer - really, we've benefitted a lot from chip R&D over the years, in terms of increased capability and lower power consumption. Without looking into it, I take it as given that there's graft and corruption at some levels in those companies big chip, and I don't care one way or the other. Commodity hardware harms anyone who's trying to make high per-unit margins, whether they have good or bad reasons. Likewise, commodity hardware comes with a massive fall in quality, inescapably. My opinion, yours, theirs, or the average consumer's, all make no difference to market capitalism.
What holds us back from that possible future is the need to make and sell full, general-purpose computers, especially ones compatible with Windows. As soon as China can make royalty-free, network-connected processors, that boost the power of your distributed system simply by existing, you'll be able to buy them in bulk. If the drivers for those bulk processors are open source and can be maintained for decades to come, that will transform the computing space irrevocably.
Again - that's not an unqualified good, not if quality and R&D both nosedive. But in our world where we already have oceans of unused hardware sitting around, perhaps it might be for the best that we start to make do with simpler, less wasteful technologies.
So Why MAD?
Everything I've talked about here is tied together by the ultimate goal of having software that expands into multiple pieces of hardware as needed. If you can do that, you can use processors and memory and GPUs on other, even older, computers. You can display the result on any random, network-connected display. You can use any random piece of network-connected hardware with a keyboard as input. You can access your home applications, not just your home files, from wherever you are. Our existing models don't work to do all that.
The future in which they can do that is exciting. The future in which you can control all the devices that you own, all the devices that you can reach, as though they were all in a single logical piece of hardware, that's exciting. The idea of cheaper hardware, even if it is commodity, is exciting. The idea of working or even playing like you're at home, whenever you have access to an internet connection, is exciting.
MAD will undoubtedly be wrong in some of its particulars. It's probably got some fundamental flaws that we'll only discover in implementation, or on thorough review by experts. And in the end, it's possible that MAD, being my concept alone for how distributed systems could work, will simply not be how things do work. And I embrace all of that.
Because the future where we can do so much more, is worth a lot more than me being celebrated for writing some silly blog posts.