Why MAD?

This is another blog post that I've ended up banging my head against, writing and then discarding multiple, relatively long revisions.  Unlike the last several, however, this is not about describing a technical component of Project MAD; it's about the opposite.  The truth is, when you distill a distributed applications model down to its essence, it sounds so simple as to be irrelevant. And indeed, the model - absent proper support - might do little enough.  It really needs to exist inside of a context that makes it more valuable.

And the funny thing is, a properly supported model for distributed applications is insanely valuable.  There are thousands of examples, spread across hundreds or thousands of different domains, where we are having difficulty writing applications that combine the resources of two or more computers together.  In the world today, we are expecting and hoping that companies and open-source projects will each create solutions to each of these examples on a case-by-case basis.

Those are what I call a special-case solutions to the problem of distributed applications.  What I am searching for is the general-case solution, one that fits, if not all of these examples, most of them.  A single technology would make it easier to create solutions for these problems, but alone, a distributed application model still expects - nay, demands - that programmers solve the hard problems involved in making applications work across long distances.  Those hard problems are the kind of thing that need to be solved centrally, on an operating system level, so that each application programmer can go about doing the thing they are there to do.

But it's worth laying out the argument, as best I can, piece by piece.

Why Distributed Applications?

I said above that a properly supported model for distributed application would be insanely valuable.  It's worth running through some examples of things that are wrong with modern computing that come down to, essentially, the lack of general distributed applications.

There are a lot of examples of simply inadequate hardware.  My dear mother has a relatively recent laptop that isn't upgradable, and is now all but useless for a non-technical user.  (I'm sure I could install Linux and it would work fine, but she doesn't want it)  That's an example of something that's too cheap; but modern GPUs are getting too expensive because they are trying to get a single chip to do a massively parallel task all at once.  If someday distributed applications (generically, not counting special cases) can split their GPU load among multiple processors as easily as they can put it on a single one, a lot of people will be a lot happier.  Making applications distributed likewise helps with inadequate memory as much as inadequate CPU threading; you still need the same amount of memory, but today you may already have the excess, perhaps locked up in old laptops or desktops where there's no good way to use it.

It goes beyond computers that are poorly designed.  Smart devices, including TVs, watches, glasses, car dashboards, VR headsets, and so on, are not generally meant to run an entire generic PC for their user, but they have enough hardware to serve as a decent front end - but the process for making good use of them is spotty at best and terrible at worst.  It isn't even, to my knowledge, acknowledged that these are all examples of the same class of problem: finding a good way to make use of compute, display, and sensors distributed across many devices.  That's part of what I'm trying to solve, frankly.

There's also ownership of data.  Home cloud systems exist (I use one), but larger companies put a lot more development work into the proprietary versions, so those end up more advanced and user-friendly.  A large part of making these cloud-compute systems, however, involves solving the fundamental problems of distributed applications, if only in a specific case - finding a free compute node, authenticating users, deploying and updating applications, error handling, and so on.  If there is a general-case solution, and one that users can deploy at home or in small businesses with relative ease, then the value proposition of cloud computing isn't as strong.  This not only relieves end users, who don't have to trust the companies, but it takes companies out of the business of both safeguarding and somehow monetizing other people's data, which is an unfortunate contradiction to be in charge of.

The last example I'll point out for now is administration.  It's not trivial for non-technical users to manage home servers, not least because managing the server is simply a separate task from managing your PC, so it's something that home users need to explicitly remember to do.  Where your PC might (with varying levels of obnoxiousness) force you to update apps, drivers, and services, currently any machine that isn't in the foreground must either do things automatically or… not do things automatically, and leave the user to their own devices.  Distributed administration means that the state of the entire system is always in the foreground; if some part needs updating, it is the same whether that part is local or remote.

In the same vein, because the distributed application model that I'm proposing is about an application expanding to use remote resources, the applications are not split into separate client and server components that must be updated and managed separately.  A unified model means that local and remote pieces are understood as parts of the same whole, and that includes installation, deployment, and updates.  Between distributed OS updates and application updates, the entire environment that is being used - even though it is on multiple pieces of hardware - is all in scope at once, and all managed at once.

That, however, is all about distributed applications.  You could make a distributed application that manages existing operating systems. But MAD is more than just the app model, which raises the next question.

Why a Distributed Operating System?

When I originally created MAD, I took it as granted that we would need a new operating system to handle the management and operation of a distributed system.  Frankly, I merely assumed it; it's fair to wonder whether or not a distributed-first operating system is ultimately required.  Indeed, if you browse my blog, you will find a lot of concepts that seem to be unrelated distractions from the core concept.  And in some cases, that's probably exactly right; you'll never catch me saying that MAD is perfect.  Given the name, that'd be kind of silly.

The distributed operating system concept is not the distributed application model.  They are separate, and must be argued for separately.  And quite frankly, I can envision a future in which distributed applications are simply run on top of existing operating systems with little or no change to those operating system.  Frankly, managing distributed applications comes down to a software service; it doesn't require a clean slate.  I imagine that a lot of people, assuming they managed to figure out what I'm saying among all the stumbling explanations here on my blog, would make that point themselves.

There are two fundamental counter-arguments I'll make.

First, the distributed operating system is a collection of standards, and there should be a collection of standards that applies to distributed applications even if they are being run on top of existing operating systems.  To some extent, when I'm talking about the distributed operating system, I'm talking about the environment that exists inside of the distributed ecosystem itself, rather than the physical disk images on top of which the system itself operates.  It's important that applications and the systems onto which they are deployed, can agree on some terms, like what resources are available, what deployment constraints an application fragment needs fulfilled, how standard functions and hooks work, and so on.  It's astonishingly easy to imagine getting that wrong and creating two or more completely incompatible systems, or ones that will require serious shims and compatibility patches to be made to work later on.

The other argument to be made is that operating systems, even open-source and user-focused operating systems, have become bloated - not because of any technical failing on anyone's part, but because they were developed over time, and different components represent different ideologies regarding how a computer should be programmed.  One needs only familiarize themselves briefly with any one of the many arguments about Linux init systems to see that we are operating a complicated system on top of mixed metaphors.  I'm well aware that any OS I create or am involved in, will not be the final operating system, and assuming it survives, it will tend back towards the current state of chaos over time; thus, that itself can't be my argument.

But the various technologies I have placed generally under the OS category of Project MAD are all distillations.  For example the System API Directory and Application Folders frameworks, together unify files on disk with the abstract concept of data, as provided by drivers, services, and applications.  While I expect to have arguments about this framework with experts someday, the point (as I have made it in my last couple blog posts) is having a consistent metaphor that works for all data in a distributed system, from the smallest bits of application state, to LED light indicators exposed by drivers, to database sockets, to embedded applications and libraries.  One possible interpretation of the SAD/AF framework (not necessarily the one I'll push for) implies that any variable stored persistently in any part of any application is addressable (if not necessarily accessible, or even listed) via the directory, uniquely, so that it would be the same from anywhere in the system.  And at the same time, every compiled-in function would be not only addressable but callable from anywhere, complete with full type safety and crash handling.

It's not about actually doing that.  That would be a safety and security nightmare.  But if you have a data accessibility technology built into your distributed operating system that makes that level of access possible, it can handle anything less than that.  If you can go that far, then you can manage every application process on every node of a distributed system, you can bit-bang every GPIO bus on every piece of hardware, and you can read every last byte from every disk.  And critically, you can do it all from the filesystem, where everything accessible may also be listed publicly for you to peruse, so that you may better understand the system you're working on.

A single consistent metaphor that allows you to do everything that a distributed system can do - that's valuable, and it's a quantity that is not at all guaranteed in modern operating systems.  Building the right constraints on top to prevent abuse… I'm not going to say it's a secondary concern.  But I have some confidence that security experts won't have to strain themselves to find ways to block funny looking remote procedure calls, with suspicious senders and destinations.  If nothing else, IT departments are already wrestling with similar problems.

However, the SAD/AF framework only works for applications within the distributed OS's purview.  It is my concept that even a dedicated MAD OS image will have a number of services and drivers that are not accessible remotely under any circumstances, for security reasons - but if you simply have an ADA-compliant server on a random operating system image, almost none of the applications, services, or drivers of the host system will be exposed to the distributed system.  Rather than an intentional screening of things that do not deserve to be accessed remotely, you simply have the common and average fact of incompatibility.  And that's… fine.  It'll probably end up necessary, one way or another.

But if you want to administer systems, if you want to control all their hardware, then you want the operating system image to expose as much as possible to the network (for authorized and authenticated users, of course).  And there's a fairly obvious security hole - it's entirely plausible if you merely have a remote application service, which isn't well tied to the operating system, a remote application could start a local process which the service itself cannot regulate, or even detect, becoming an out-of-band attack vector.  And while you may be able to count on normal OS tools to discover and handle such an incident, and you can have distributed administration tools that handle those OS tools, the added complexity makes the situation more precarious.

Meanwhile, because of the way distributed systems work, many nodes in your network may be used for nothing more than hosting remote processes.  Modern OS images provide a large number of services and drivers that may not be needed or used; a compute-only network node could be trivially small, and have no exposed filesystem (or an immutable one) for remote applications to attack.  While it's possible to set up, eg Linux, systems to run at this rudimentary level, most systems are still expected to be general-purpose, and their OS images reflect that.  Systems set up for hosted compute only, especially if they have an out-of-band or self-update mechanism, may be all but untouchable to applications in a distributed system, in addition to being faster and simpler to operate.

The reason, of course, that most OS images today contain so many nuances is because the hardware they are running on top of can be so variable, and it's unclear how much of that hardware a given user might actually need.  And that brings me to my last question:

Why Modular Hardware?

When I first envisioned MOS/DCA, adding a processor to a system meant connecting little hardware devices together; the system was made of modules that would daisy-chain infinitely, and each module's hardware included absolutely nothing except its payload, and the backbone networking chip.  No general-purpose networking was involved here; it was dedicated pieces slotting together to make, effectively, a single desktop system out of hardware modules.

Hardware drivers under such a system, you understand, would be quite simple.  A CPU-only chip has no need for any drivers except the chipsets that support the CPU itself (including a fan and temperature sensor), and the backbone.  No generic networking, no printers or keyboards, no serial busses or parallel ports, no USB, no solid-state or spinning platter disk drives, no CD/DVD/Blu-ray drives.  A compute-only node not only need not do anything else, it could not.  No such accessory hardware existed.

In return for that simplicity, these modules were the very definition of commodity hardware.  I imagined they would be churned out by the millions.  In the modern world, I imagine them being built off of license-free technologies like RISC-V, driving the costs ever downwards; twenty years ago, I would have hoped for something similar, even if no such were commercially available at the time.  Nowadays, Raspberry Pi has proven you can deliver a full computer for $30, as long as you don't mind handling exposed pins and solder.

If you could make every computer you own work better with a single $30 purchase, would you really stop at one?  If plugging a raspberry pi into my parents' network was all it took to salvage my mother's failing laptop, she would have taken it in a heartbeat.  And the $30 raspberry pi comes with a graphics chip, HDMI out, and other accessories that a dedicated compute node may not need.  What really would be the ground floor, if you still want to end up with a unit that's good enough to make your distributed system better?

It goes beyond the compute only nodes.  Outside of general-purpose busses like USB, any task-specialized node only needs a few additional drivers.  Some will be complex, like a graphics unit with a GPU and multiple output ports, but if you only want to add, say, an AI accelerator chip, or a collection of data drives, the system image that runs that entire node has very few additions on top of the base layer; you will be certain that no other capability exists or is being used, because that's all that the system is.

If computer parts become that level of commodity hardware… well, I'll be honest, it will harm research and development of new chips.  That's a nice way of saying that chipmaking companies, which are already on rocky ground nowadays, could lose significant licensing revenue and sales, so long as people can get a cheaper alternative.  Some chipmakers, especially in the GPU space, are cagey about letting others understand how their technology works to preserve their competitive advantage - but commodity GPUs don't need to be individually powerful, if they can work together.  A no-nonsense alternative that's easier to program for and keep updated over the years, which gets to the same place with quantity instead of density, would certainly make that policy of stonewalling your own developers harder to maintain.

I'm trying hard not to paint those outcomes as purely good for the consumer - really, we've benefitted a lot from chip R&D over the years, in terms of increased capability and lower power consumption.  Without looking into it, I take it as given that there's graft and corruption at some levels in those companies big chip, and I don't care one way or the other.  Commodity hardware harms anyone who's trying to make high per-unit margins, whether they have good or bad reasons.  Likewise, commodity hardware comes with a massive fall in quality, inescapably.  My opinion, yours, theirs, or the average consumer's, all make no difference to market capitalism.

What holds us back from that possible future is the need to make and sell full, general-purpose computers, especially ones compatible with Windows.  As soon as China can make royalty-free, network-connected processors, that boost the power of your distributed system simply by existing, you'll be able to buy them in bulk.  If the drivers for those bulk processors are open source and can be maintained for decades to come, that will transform the computing space irrevocably.

Again - that's not an unqualified good, not if quality and R&D both nosedive.  But in our world where we already have oceans of unused hardware sitting around, perhaps it might be for the best that we start to make do with simpler, less wasteful technologies.

So Why MAD?

Everything I've talked about here is tied together by the ultimate goal of having software that expands into multiple pieces of hardware as needed.  If you can do that, you can use processors and memory and GPUs on other, even older, computers.  You can display the result on any random, network-connected display.  You can use any random piece of network-connected hardware with a keyboard as input.  You can access your home applications, not just your home files, from wherever you are.  Our existing models don't work to do all that.

The future in which they can do that is exciting.  The future in which you can control all the devices that you own, all the devices that you can reach, as though they were all in a single logical piece of hardware, that's exciting.  The idea of cheaper hardware, even if it is commodity, is exciting.  The idea of working or even playing like you're at home, whenever you have access to an internet connection, is exciting.

MAD will undoubtedly be wrong in some of its particulars.  It's probably got some fundamental flaws that we'll only discover in implementation, or on thorough review by experts.  And in the end, it's possible that MAD, being my concept alone for how distributed systems could work, will simply not be how things do work.  And I embrace all of that.

Because the future where we can do so much more, is worth a lot more than me being celebrated for writing some silly blog posts.

Plan 9: On Resource Distribution

 So I've recently become aware of the Plan 9 family of operating systems and related technologies.  Invented in the ‘90s by the geniuses from Bell Labs, it is a distributed operating system using a distributed filesystem - so it's not unlike in Project MAD, and as such, it's a good idea to discuss the one in terms of the other.  I have revised this summary many times by now, and I’m still not sure I'm happy with it, but here we go anyhow.

There are a lot of different ways to look at the differences between the two; I'll try to avoid cynical or self-serving “analysis” that paints everything I do as right and anyone else's ideas as wrong.  In part so that I can get this blog post out sooner, I'd like to focus in on one particular aspect first, and draw a distinction between the way Plan 9 and MAD handle resources in a distributed system.

I will summarize Plan 9's resource model as a being ‘pull’ model, by which I mean, the resources of the system (eg, devices and running software agents) are made publicly available, and any given program can “pull on them” (which yes, includes pushing data) from wherever the program is on the system.  I'll make an analogy here of two ships, at some arbitrary distance apart in a large body of water.  A ‘pull’ model involves a direct connection between the two ships, analogous to firing a grappling hook across the distance and pulling a line taut; you can then send resources back and forth across the line as needed, but because that rope is the sole point of contact, all negotiations between the two ships must go across it, from the most mundane to the most meaningful.

MAD does the opposite; it uses a “push” model.  Using the same analogy, a MAD ship will send out smaller boats, each towing a line, to go out and meet other ships.  Because these boats contain trusted crewmembers, they can negotiate and handle certain problems and events autonomously; not everything that happens needs to be coordinated with the mothership, meaning the line between the boat and ship can be reserved for important communications and resource transfers.

Stepping back from the analogy, Plan 9's uses a protocol called 9P that creates a shared filesystem, and then uses some filesystem functions for configuration, which selects which devices any given process should be using (eg, console, keyboard/mouse, display, etc), among other uses.  While I can see the vision, the choice to encode system configuration in a way that's hard to read and write is problematic.  Better tooling would help; it would be nice if the canonical configuration tool does what it does in a more user-friendly way.  Many systems have trouble with that, but it's something that really ought to be considered when making your tools.

Plan 9's design works because they chose to make everything in the system a file, and make those files available over the network.  The “Everything is a file” metaphor of course comes from Unix, and I will get into how MAD uses a different core philosophy another time.  Making a distributed filesystem that is available across the network is also a mechanism MAD uses, though differently; I will mostly discuss that more later, too.  The point is, Plan 9 decided on a standard interface for devices and other programs, a way to configure that interface, and a way to make that interface available across a network; that is the core of what a distributed system needs.  What makes it fundamentally a ‘pull’ model is that they didn't do anything more than making those interfaces available; in that way it is little more than a server-client model that would work just as well using the Internet Protocol as it does under 9P, if for whatever reason you chose to build such a network over IP.

Note that under Plan 9, this resource distribution model does not consider ‘the ability to run programs’ a resource analogous to other devices.  Believe me, I sympathize; I spent a lot of time thinking about how you represent the ability to run programs as a consumable resource generally, and while I imagine I have an answer, it is definitely something that can only be borne out through testing.  For certain, though, the ability to run program is not something that you can package into a file and make available on the network, at least not the way Plan 9 has done it.  (It might work in conjunction with the ADA, below, but not alone)

MAD's ‘push’ model also was born directly from the need to consume remote devices.  At first I assumed that, as in Plan 9, you are essentially performing remote procedure calls to reach every device in the system, but I didn't like that.  I decided I wanted to try to minimize network congestion, and negotiating with devices frequently comes with a lot of redundant back-and-forth for what is ostensibly a single function call on the program's side.  As I described in my blog post about the history of the programming model, I got the idea to execute scripts on remote nodes so that you could do several operations on a device at once, before finally settling on a formal, agent-oriented solution.

That solution, the Agentic Distributed Application model, requires that your application be split into several programs known as agents; no agent may assume the existence of any resource that it does not declare as a requirement, and no agent shall be placed on a hardware module that doesn't meet its requirements.  These agents are ‘pushed’ out to some correct hardware machine (the choice of which may be automatic, configured, or manual) when the application is loaded, and then all agents of an application simply communicate with each other, with the network abstracting away the distance between.  Thus each agent becomes a proxy with which to access the resource or resources that are local to that piece of hardware.

Of interest to what I said above, you can consider the ability to run programs to be a resource in the ADA model, because the resources here are understood as constraints.  This is useful, for example, when you have incompatible processor instruction sets present on a single system; you may only have AMD64 agents compiled for your program, and they shall not be run on an ARM or RISC-V processor.  In the same breath, though, you can extend the compatibility of your application by adding extra agents; since most agents will be very simple, the chances that compiling them will be difficult is minimal, and if you have some particular process that is more difficult to adapt to other architectures, it is fine if that specific agent is incompatible, while the rest of the program can be run on different processor types.  For example, using an ARM-based Raspberry Pi as your terminal means that you need input and output agents that compile for that processor (some asterisks here that shall be ignored), but the core of your application can depend on AMD64, which may be run on a full server, your desktop, or a laptop in a closet, whatever's easiest.

The ADA model's primary benefit is giving me, the system designer, confidence that we have enough information to say that applications and their agents can be automatically distributed throughout the system.  Together with software hooks that allow programs to override the default methodology, this resource requirement model filters out places where it would be incorrect to send a given agent, and because the application doesn't require a single machine to meet all of the requirements at once (I include access to files in this), the application as a whole is compatible with the gestalt system so long as all the capabilities it needs exist somewhere.  Similarly, because we control our assumptions, it is vastly more plausible to say that some Agents (especially a CPU-only agent, but really anything that doesn't have local side effects) can be paused and moved from one module to another without impacting the overall performance of the application, if for example the processor you're on becomes busy with some important task, or even if it crashes or disconnects from the system.

Under the ADA, all of these distributed applications have their own place within a shared filesystem, one that works somewhat differently to Plan 9's.  Each ADA application instance has a folder in the directory, and when an ADA agent makes requests for resources, it does so relative to this application-instance root; anything that the application needs should be mounted there.  Where Plan 9 makes the decision to change how any given application sees the global filesystem, MAD customize this private (but publicly accessible) directory of links.  Thus for example, they would look for a console device at ./cons, rather than /dev/cons.

Also in the application directory, of course, are links to the application's agents.  The application never needs to explicitly talk about where their agent is after it is deployed; they act upon the local link to that agent, which will proxy their request to the agent no matter where it is.  In fact the canonical location of these agents is likely to be mounted under the hardware itself, rather than in the application directory, because that's where the process is.  Either way, because these agents are filesystem links, they can be named properly, and thus be human-readable; a small consideration, perhaps, but one that helps programmers and administrators keep their sanity.

There are lots of other things that may or may not belong in this directory, depending on implementation: one good example is the application's user-specific file cache; rather than the application explicitly being told to (or independently deciding to) place that in some “/home/user/.cache/browser” folder, when the app is started, they assume that their own private “./cache” folder will link somewhere, whether that's a user folder or a private ramdisk that will vanish when the program closes.  Configuration files can be similarly linked (though MAD has an explicit configuration mechanism).

These application directories implicitly containerize applications.  If for example, your application depends on a shared library, they should expect that shared object to “be” within their own private directory, even if what is there is only a link.  If your app depends on another program, they should expect that program to “be” within their private directory.  In this way, the application works exactly the same way whether their dependencies are embedded or shared; it works the same way whether the application is “installed” or run from a private directory.  And, if a user ever has a need to forcibly embed an older or modified version of a shared library, for example for compatibility reasons, the mechanism for this is explicitly already there.  (Obviously, this is also a security problem that needs monitoring, but that's not difficult to address, at least in principle)

Much of the above will be modified slightly when I talk, another time, about how MAD doesn't exactly use the “everything is a file” core metaphor, but I think it all stands alone for now.  This is all, ultimately, about how you organize data in a distributed system; it all exists to help minimize the assumptions of the application, and thus increase its compatibility.

I have described Plan 9 as coming in at the 60-80% mark for what I'm looking for in MAD; this blog post is a good example of ideas that are in some ways compatible, but approach things from different directions and can have very different consequences.  It's easy to say in the abstract, for instance, that the push and pull models of resource distribution are not that different, because their goals and therefore nominal effects are the same; however, building a system on top of one or the other is a different task.

Part of what I see as valuable in MAD is this kind of perspective.  While accessing devices remotely (pulling) feels like a perfectly valid model (it is, to be clear, how the internet works), there are some questions inherent to a distributed system that it doesn't provide an answer for or a mechanism to assist with.  This difference is clearest when you are looking to solve all of those questions inherent to a distributed system - but it is a difference that any programmer for a distributed system will feel, when they are obliged to solve any of those questions for their niche application.

Having said that: the two models are not incompatible.  Indeed, while formal ADA applications take some work to implement, you can informally do the same as an app developer, absent proper tooling.  Even without something like Plan 9's distributed filesystem, with user consent, you could login to remote hosts (that are under their control, or to which they have file and network permissions), then copy programs over and run them, then connect to them at their known host and port.  The real goal of the ADA is to do as much of this as possible automatically, including (for some simple apps) dividing your program up into agents.  I'll be happy with the ADA when (if) you can take a program not meant explicitly to be distributed across multiple machines, and run it distributed anyway without effort, because that's what it takes to make a consumer-ready operating system.

That task, automation, takes some advanced thought and planning, and that's part of why I am hesitant to say that I've solved anything “for real” without talking to experts.  It's too easy to find that you have assumed wrong or overlooked something and now have a much more difficult problem to solve than you thought you'd have, a problem that might have been easy if you had done it all a different way.  That I can say MAD makes things easy where Plan 9 makes them difficult, also means that MAD might be doing things the difficult way when someone else can do it easily.  That's simply how having different methodologies, works.

And that's part of what makes this all so much fun.

IP: The Worst RPC Stack

 So I was originally going to write on another topic, and you know, I ended up going off on a side rant about something I otherwise wouldn't have really thought about, and I've come to realize that it deserves to be called out on its own.  That topic is: The IP-Port-Protocol stack that we use to communicate with servers is really just about the worst form of remote procedure call that we could have, isn't it?

Now, when I say that, keep in mind I'm an author at heart - I like to make inflammatory statements to get people riled up, especially when the statement is true at heart but knowingly wrong in its particulars.  You can always imagine, or find, worse ways to call a remote procedure, than this.  But I say it this way because most people wouldn't really think of IP:Port addressing as being a remote procedure call at all.  It is how you consume a service, though, which ultimately means that you are calling one well-defined (well, maybe) function from one program in your own program, frequently over a network.

Just on the merits of success, clearly the IP stack has done a lot for us.  But we already expect more from it than it was designed for, and that's before I start talking about the many and varied requirements that I am coming to expect from an RPC mechanism, after exploring Project MAD.

The IP Suite: A Partisan Overview

So when I say “the IP-Port-Protocol stack”, it's not necessarily clear exactly what I mean.  The Internet Protocol suite, which is the basis of literally the entire internet as we know it today, can be understood a lot of ways and with various levels of depth, but I'd like to break the topic up into two pieces: Addressing, and communication.

The part that I'll summarize as Addressing contains numerous mechanisms, some of which are barely relevant to the point I'd like to make, but broadly, they let you translate a name into a series of numbers (which can be summarized as a single number, if you prefer), and then all of the hardware in between you and the destination know what to do to connect you to that number, and connect others to your number.  There is a service, called DNS, which translates friendly website names into an IP number, which is the official canonical number you use to reach that website.  Depending on how complex the service you're trying to reach is, that IP address may actually send you to a service that distributes packets and queries to a number of different handlers, to ensure that nothing gets overwhelmed or to let you use a service more local to yourself instead of a single global server.  (And I may be getting parts of this a little wrong, because it's complicated)

But it's interesting to talk about IP addresses in a local context, because for many home users, IP addresses aren't reserved for a given user or machine; they're temporary.  And even when they aren't temporary, they're mostly arbitrary, meaning that unless someone tells you what an address is supposed to be, you have no context from the number itself.  But it's inconvenient and therefore rare, for local users to set up tools like DNS for their home network, so that you are dealing with descriptive names rather than numbers.

But the inconvenience goes a bit further: all network packets in an IP switched network are targeted not only at a machine, but at a specific port on that machine.  This port is simply another number, and although there are standards suggesting that several well-known ports should be used for specific protocols and applications, there's not actually anything that's stopping you from using the port normally reserved for websites, to do almost anything else you want.  Oh, it'll be inconvenient and confusing if someone tries to access that server from a webpage, but the system itself doesn't care.  This problem cuts both ways: You can use (almost) any port for (almost) anything you want, and you can't tell what a port is actually being used for just by the number in use.  This becomes an even bigger problem when we start trying to manage complex data spaces with IP address hierarchies, but I'll get back to that in a minute.

The second half of my description for the IP suite is “communication.”  Once you've established a data channel to a remote server, you then need both computers to speak the same language across that channel, and this is known as a protocol.  The Internet Protocol itself, of course, is one such language, one that is used to get your message to the right place; but once your message is in the right place, two pieces of software need to agree on how you make requests and how results are returned.  There are several existing protocols, and the most used ones are well-known, but it can still leave you feeling ignorant of how your own machine works.  If you know the IP number and port of a service, but not the right protocol to use, you can do nothing with it - but, as I said, the IP number and port are in some sense arbitrary.  From a certain perspective, the internet is full of billions of holes, and it is your job to cup your hands and yell code phrases into exactly the right hole.  Yell the wrong code phrase or pick the wrong hole, and nothing (or nothing useful) happens.  Pick the right hole, and you'll hear a code phrased yelled back at you from the other end of a pipe.

The end result of all of this, functionally, is that if you have the right combination of IP number, port number, and protocol, you can call one function on a remote machine.  It is, in other words, the worst possible remote procedure call stack.

If that doesn't sound right to you, it's probably because many servers have come to expose a lot of functionality on a single port.  That's all down to the parameters you pass the service when you make a request; the exposed function that you're calling is itself a directory of other functions.  Perhaps, one might argue, that means that you are really exposing far more than a single function, but that argument is flawed.

In fact, generally speaking, it is the job of anything that wants to provide services to invent a new remote procedure call mechanism.  It is your task to take a data block as input and decide what function to call, with what arguments, and if data is returned, what to do with that return value.  It is one of many places in modern computing where the solution to a problem is, force programmers to find their own solutions to that problem.  One significant consequence: every programmer is forced to find their own ways to parse, validate, and verify incoming method parameters from that data block, which provides a massive and porous surface where programmer mistakes can become exploitable program vulnerabilities - and frequently, those vulnerabilities affect general purpose services that are capable of doing a lot of things.

If you want to build a system on top of remote procedure calls, the first thing you must do is not require everyone to reinvent remote procedure calls with every program they write.  It is in this context that I want to talk about how the MOS/ADA subsystems think about remote procedure calls.

RPCs Require Better Communication

The Agentic Distributed Applications model, and the Modular OS concept built on top of it, both talk about listing what remote procedure calls are available.  The list of procedure calls is used for three purposes: one, it tells remote callers what is available, two, it is used to verify that incoming requests are valid, and three, it is used to direct the final result to the function that will be handling the request.  It's worth noting that the IP suite in fact does not tell people what functions are available to be called on a server, and the IP suite itself doesn't verify or validate any incoming requests; if there is a service which can receive the request, that service handles it.

But the MOS/ADA RPC stack is expected to do more than that.  Ultimately, it is the function of the RPC mechanism and not of the individual service's back end, to convert packed data into arguments for the function to be called, and to convert the function's return value and/or error condition into a packed data type for the return trip.  This is known as not requiring programmers to reinvent remote procedure call mechanisms, or if you prefer shorthand, it is known as an RPC mechanism.

In order to do that, the system must have an awareness of data types, which is why data types are an important part of the system directory.  Not only must you be able to convert incoming arguments to the appropriate data type, but you must be able to verify that the incoming data type is an acceptable match for the type expected by the exposed function.  And by having those types be explicitly listed, the calling procedure can do certain checks before sending the request.

(I suppose you could expect every function to take an undefined, fully variable data type as parameter to every function, do no type checking ahead of time, and expect the end programmer to dedicate the first lines of every function to finding out whether or not they were passed the correct arguments, the way for example C programs do with their command line arguments… or you could compartmentalize those checks by having them done in an explicit precondition function, which may be an overridable hook, allowing exported functions to only contain that function's code.  One of those sounds like requiring programmers to reinvent remote procedure calls to me, while the other sounds like having an RPC mechanism.)

Already, we can see incredibly clear distinctions between what I am calling RPC mechanisms, and raw server sockets that get piped straight into application code.  There is a language to RPC calls, and if you want to understand that language, it can be explicitly queried, finding out exactly what the protocol expects and how to format your requests.  Rather than shouting memorized code phrases at one specifically chosen holes among billions in a wall, there is a piece of paper taped on the wall telling you what to say into which pipe to receive what reply.

Of course, someone developing their own RPC mechanism to respond to queries on a raw server socket can do all this… and they can do it separately for every single application they create, because it isn't a defined mechanism, it is a private standard that they have decided on.  It makes far more sense for this mechanism to be standardized and this directory to be machine generated, because most if not all of the information is already there when you compile your program (or run your script, if the language used doesn't get compiled).  As long as you agree on how to translate the information already required to create and present within your program, into this directory listing, it should be relatively simple to implement.

Relatively simple, of course, does not mean easy, or even simple.  There are a lot of questions about types that need definitive answers and canonical standards.  Today, before any of those questions are answered and those standards canonized, it would be relatively difficult and complex.  To put it another way, Project MAD has always, from the very beginning, been about doing the hard work ahead of time so that other people can simply walk in and do the fun parts.  This is known in some circles as creating an operating system, though perhaps not everyone would agree.

I would argue that once you have all of those standards, you can do everything that the IP-Port-Protocol stack can do, but with names instead of numbers so that you can read what you're doing, and with documentation that tells you what's available and how to use it.  A protocol, in other words, that is not only machine readable, but human readable.  One that not only allows communication at runtime, between processes that already speak the same language, but allows the programmer who created a program to communicate with users and other programmers in the future, making explicit their assumptions, needs, and intentions.

The cost in exchange, of course, is complexity and speed.  As with many aspects of Project MAD, that slowdown might be a real problem, but equally, the speed we have now comes with fragility and ignorance.  We have built a lot of very important systems on top of tools that only barely allow us to communicate; we've discovered that we don't need much structure in order to make something like the Internet work.  But as we have wanted to create more powerful and flexible tools, we've chafed at the restrictions.  Better communication is needed.

A Word or Two on Addressing

So I promised to come back to a bit about the Addressing side of the IP-Port-Protocol stack.  As I said at the beginning, this entire topic originated in a blog post about other things, and specifically, I was trying to list all of the problems that Project MAD tries to address with its solutions.  One of those is the confusing morass that is IP addressing within application containers and private networks.

For those not aware, application containers are used to explicitly contain a piece of software and its dependencies, so that if two applications each require different versions of the same library or tool in order to function, they will not accidentally select the other app's incompatible version in place of their own.  They likewise won't share some system configuration bits, which is useful if two apps each require the same embedded tool but configure them differently.  There is another conversation to be had, here, about configuration, but we'll get back to it in a bit.

Suppose for the moment you had a home server farm, with several hardware servers, each running hypervisors that split the system into multiple virtual servers, and each virtual server used IP-based application containers, with some of these containers incorporating their own private IP networks containing multiple internal containers.  It's fair to say that there would be a lot of IP addresses being used to direct traffic on this distributed system: each hardware server's network adapters have their own, each virtual server's network adapters have their own, each container and sub-container has its own.

Part of the very real problem with this setup is that all of these IP addresses are arbitrary, and due to the way their scopes work, it's entirely plausible that if you leave things to automated processes, random containers on different virtual servers may have the same internal IP address.  They shouldn't be visible to each other unless you've done something wrong, and perhaps no part of the distributed system as a whole may malfunction, but as an administrator, programmer, debugger, or power user of the application containers, it is frustrating that what is supposed to be a unique identifier is neither unique nor identifying.  An IP address alone may tell you neither what is at that address, nor can you be sure that it is used exactly once, even within your own, privately operated server room.  And it may not be quite as simple as an amateur would like to change those internal IPs so that everything is unique, especially if those unique IDs are only useful for reading logs and chasing down problems in a complex and distributed application.

Part of what I hoped to guarantee with the MOS System Directory is that across an entire, theoretically infinite distributed system, it is still plausible to uniquely identify every resource, by having explicit nested scopes.  Because systems list their internals, even privately, as long as you can distinguish peers from each other within their own context, you can create a correct, unique, descriptive identifier for every resource - meaning that at every level of context, you can uniquely identify all children of your peers (even ones you have no access to), and if you have permission to explore the peerages going up the stack and outwards as well as down, you can discover your system's parent's long-lost uncle's step-son's cousin's application container's database, and the fully qualified unique identifier for it and for you will tell you exactly what your relationship is and isn't.

At the same time, because the unique identifier is based on nested scopes, you don't need a fully qualified unique ID to describe anything, not unless you share no relationship except having the same system root.  In fact, if you know the scope you're targeting, you can convert a local directory reference into an absolute one, and vice versa.  It's still wise, of course, to use the fully qualified ID once you start going up in scope, in case there are subtleties you are missing, but it's not necessarily technically required.  Once any two resources share a scope, that scope is the highest you need to go in order for the two to identify each other and communicate across the network.  Even if there is a faster way to communicate, the two fully qualified IDs demonstrate the maximum number of scope changes required for one to reach the other.

It may be somewhat crude for me to say, that I believe an addressing protocol should allow you to give, get, and understand the address of that thing.  It may be a bit rude to suggest that the IP protocol's arbitrary identifiers make it difficult to feel like you are in charge of the layout of your system.  It is unquestionably arrogant of me to suggest that I could do better than the technology that united and changed the world.  But what I don't think is arrogant, rude, or crude, is pointing out the flaws in any technology, especially where those flaws have consequences.  Any system built on my ideals will have its own flaws, and I have to trust that some person smarter than me will someday figure out a way to address them... no pun intended.

And IP has its flaws.  All of this is without getting into the limitations of the still-ubiquitous IPv4, which hasn't had enough addresses in a very long time and has felt for a while like it should just get replaced by its successor, IPv6, to prevent some very real, and some other mostly theoretical problems.  Frankly, that's not my problem with it, not as a power user of home networks, nor as the architect of Project MAD.

My problem is that the IP stack as a whole was a very early (starting in the 1970s) way to do remote procedure calls, and all of computing has advanced and changed since then.  Even if you don't like my solutions, look at the general problems we are facing.  We want to combine systems together, tightly, in ways the designers of the internet protocol did not foresee at the time.  If you don't like my formulation of the general case problem that needs to be solved, find a better one.  But ever since I've had this general case problem that needed solving, my mind has been seeking out solutions that fit it.  You can build a distributed system on top of IP using yet another layer of abstraction, and maybe that'll be necessary.  But for addressing parts of a system, there are better ways than depending on arbitrary and not-guaranteed-unique numeric identifiers.

One Last Topic: Distributed Configuration

So I just barely touched on the topic of distributed configuration when talking about application containers, and it's an interesting side topic, one that deserves its own full blog post later, but I believe I've teased that before, and so it's better to have a brief discussion about it, in case it keeps slipping my mind in future posts.

Configuration in a distributed system is tricky, because at any given time, an application is under multiple, potentially conflicting policy and rule domains.  The user running the application may have rules and policies.  The hardware device that the application is running on may have certain specific configuration rules necessary to make it work.  The distributed system may have an administrator that is enforcing certain rules and policies.  The application itself may have multiple default policy sets, for example, one for power users and one for non-technical ones.  And an application may be embedded in another application, and the parent application may have specific rules that it needs the embedded application to follow.

But also, general rules and policies are not the only items of interest.  Anyone who has any claim over an application, such as the hardware, user, administrator, or parent application, may call out that specific application and make an explicit configuration change to it, in order to ensure proper functioning.  A user may wish to start an instance of the application with a special configuration, whether that's part of a script, or a deliberate choice made by an interactive user in a shell.

It is my intention that the Modular OS, and the Agentic Distributed Applications model specifically, has an explicit mechanism for gathering and resolving all configurations present on the system.  The resolution step may be accomplished by a callback hook, that is, an application can choose to apply configuration changes in a specific way or even may choose to misunderstand or ignore configuration changes, but there needs to be a mechanism that presents the application with a list of all configurations, and there ought to be a standard way to resolve any disputes.

Although I'm not quite happy with it yet, the working name for this mechanism is the Standard Environment Variable Evaluator, or STEVE.  STEVE is an important part of the ADA model, and arguably fundamentally necessary for ADA to work properly, just as STEVE makes little sense outside of a distributed context.

This topic, about how best to handle distributed configuration, is one that will be determined, in the end, by people who know the topic better than I.  No matter what I try to say about it now, it is a topic that will get revisited and revised once good intentions meet practical concerns, and as such, feel free to take much of what I say below with a very large grain of salt.  But naively, I believe that there is a hierarchy of default and explicit desired configuration states.  The least important configuration, for example, is the program default built into the code, which is used only if nothing else is configured.  On the other hand, an explicit decision made by the user when starting the program is the highest priority among its peers in the configuration hierarchy.

Beyond that, though, all defaults yield to all explicit configurations, and both defaults and explicit configurations are ordered by how specific the configuration is.  Configurations are less specific if, for example, they refer to an application category broadly, like Start all windows maximized.  More specific would be a configuration referencing a specific application in general, then a specific application when run by a particular user, and then a specific instance of an application when run by that user, such as when the application is passed a configuration parameter when it is launched.

But there is a major exception to this rule.  One of the things that came out of Project MAD is that, in a distributed system, every component has the ability and authority to refuse service--indeed, it's frankly the only control you have in a distributed system--and as such, everything that has dominion over an application, such as the hardware it's operating on, the user, or an administrator, can kill a misbehaving application if it does not follow policy.  Consequently, when we talk about configuration, there is a distinction between desires and rules.  A program caught not following the rules may be killed, but a program that merely disrespects the desires of others will not be.

In the hierarchy of configuration options, then, any applicable rule should take precedence over a stated desire.  If, say, the Administrator refuses to allow full-screen applications, and the user explicitly tries to start an application in full-screen mode, your options when parsing the configuration are clear: do you allow the user to do something that may attract the administrator's ire, possibly causing the application to be forcefully quit, or do you prevent the user from doing what they have explicitly said they want to do?

Of course, it's even more complicated than that, because we're talking about a distributed system, which may have multiple competing sources for various software and hardware services.  Suppose for example you have your private phone connected to a desktop machine, in order to use the desktop's monitor, CPU, GPU, and input devices to play games.  The system administrator for that desktop machine forbids full-screen applications, but your phone is not subject to that system administrator; it is completely under your control.  Your phone, however, may not be configured to not display desktop-style applications on its own screen, and for good reason - it may be too small, too low-resolution, and in the wrong orientation, and the processors may be lackluster and prone to overheating.  In that context, if you want to launch a full-screen application, do you try to launch it on the desktop screen where that isn't allowed, on the phone screen that it isn't suited to, or neither?

My thoughts on this matter are incomplete, but there is one thing that I am certain of: it should not be down to a programmer to find and read configuration options across a distributed system, nor to determine what the nominal hierarchy is.  An operating system for a distributed system should find and request all relevant configuration options and present them to the application for parsing.  This needs to be done before a distributed application selects what hardware it is going to use, because the rules and policies governing the hardware will be used to determine which hardware is selected.

A bonus that comes from the system handling these matters, is that several lists of configurations can be made explicit: the total configuration on a system with sources, the total configuration as applied to a specific application, and depending on the configuration mechanism, you may get a list of what configuration options the application itself requests, including requests from all internal libraries and embedded utilities, which tells you what configuration changes you can make to influence its behavior.  You can see a list of what configuration changes a user has made, and which of them have and have not been actually queried by applications in the last day, week, month, or year--indicating that they may be useless, malformed, or out of date.

But likewise, a system-wide configuration parser can validate a list of configuration options, providing a list of warnings and errors that indicate some options are malformed, out of date, or nonexistent.  It can simulate how configuration would be applied if you used it on a piece of hardware, on a system you're connected to.  It can tell you how Administrator-applied policies have affected you and your programs, and how your own configurations have affected you.  It can tell you when a configuration option made explicitly simply matches the default.  It can tell you when rules and configuration options have changed, in ways that affect you.

These kind of features are only possible in a system where handling configuration is a standard process.  As with the other mechanisms I've described in this blog post, the current solution is “Bring your own,” or in other words, there is no configuration mechanism.  Even centralized systems like the Windows Registry and Windows Active Directory don't give you all the information you could possibly want, and too many applications have completely internal and private configuration files that will never be explained to anyone, meaning the applications basically cannot be configured even though they were designed to be.

These are all systems that solved specific cases of the problem when they needed to, but nobody tried to solve the problem in the general case.  Perhaps if they had tried, it would have worn out its welcome wherever and whenever application developers wanted something different. I believe that it is laudable to at least examine the problem's general case and search for solutions - but more than that, in the case of a distributed system, I think it will be necessary.  To not have a standard solution to this problem will put undue burden on application developers.

Wrapping Up

I'll consider it fair if I get feedback on a lot of the points I've raised in this blog, not merely this post, as being incorrect on a technical level.  Others know more than me about a lot of the things I've discussed.  But the point of view of this blog, and the reason why it's worth writing, is that I see challenges ahead that we're not ready for.  Building a system on top of remote procedure calls requires a lot, and our current model of clients and servers is a poor excuse for that.  Arguably, the client-server model was never meant to stand in for general RPC mechanisms - but in the modern day, it's been a hammer tasked with far more than driving in and pulling out nails.

I feel like the various web services and web APIs prove this point as well as anything.  Many of these are utilizing the tools meant for web pages as a middleware to enable general remote procedure calls over the internet, and as such, they are awkward, insecure, and suffer from a lot of problems whose primary source is programmers needing to reinvent various wheels.  Tasks that should be mechanical underpinnings of a system are being hacked together.  That's before talking about using webpages as a display in non-web contexts because standards between various operating systems are massively inconsistent, or talking about webpages being the best answer we have for a remote server presenting a local interface to the user.

The tools are already stretched.  Utilizing webpages on nonstandard ports is inconvenient, even if it's the best way to get multiple separate server applications to cooperate, rather than integrating them into a single app.  Trying to understand the flow of data in a container stack is a headache, but it's still easier than trying to manage dependencies on a system not built to properly handle dependencies.  Trying to understand what is going wrong when a client and server communicate over a network can be maddening, but it's the best way we have to make use of another machine's computing resources in parallel with your own machine's.

If you want to merge machines and utilize one's capabilities from another, you need a better mechanism.  If you don't like mine, by all means, show me up.  Do better.  I'd love to hear about it.