Conceptual Walkthrough: Distributed Applications

It may be surprising, since I am here to talk about distributed computing, but I hate process forking - making copies of a running application, each running the same code but each dedicated to a different task.  I'm sure I'm not alone in this, because it's not as though people are running around calling fork() for every little thing; multithreaded and multiprocess apps are not exactly common, not unless you're doing something that really requires such a technology.

Frankly, it's a headache to try to understand.  In theory, it should be simple; there's a copy of your program, but you know which copy is which, so you just do different things with them.  If before that, you open up a pipe, both copies of the application will have access to the same file descriptor for that pipe, and they can communicate through it.  But all of those things being technically possible isn't a plan.  When that pattern is laid out in a book, it leaves you wondering: what am I supposed to do with it?

I talked in several places about programmers being obliged to reinvent the wheel, and process forking is a good example; forking wouldn't be such a terrible thing, if it was paired with a functional remote procedure call mechanism.  When two copies of the same program are connected with a pipe (or two), you can use it to pass data, but that data transfer is mostly pointless without context.  Adding context, labeling the data, makes it essentially a very simple, poorly implemented remote procedure call, where the "function" that is being "called" is simply an index specifying what to do with the data, out of a set of options fixed at compile time.  While process forking would still be a bit confusing if the process forking mechanism came with a built in RPC mechanism, at least the programmer would not have to wrestle with all the complexity of implementing their own RPC stack from scratch while also trying to build program logic around it.

Although I wasn't thinking of it when I designed the system, the ADA distributed applications model is basically exactly this, and to help you understand it, let's walk through the analogy a little more closely.

As I see it, forking has two fundamental problems with it; first, the fork is undifferentiated.  It is easy to overlook the fact that it is a feature of programming languages that we can attach variable names to data in the first place; when I say that forks are undifferentiated, what I mean is that this deliberate feature and mechanism of programming languages no longer functions.  In each half of the program, a large number of symbols exist but point to the wrong data, and it is entirely up to the programmer to recognize this and keep track of which symbols are now incorrect.

This is in fact the reason why I have specified that under the ADA, programs should be written and compiled such that each module which is to be depoloyed separately, shall have its own variables confined to a single namespace.  Namespaces in this context may be nothing more than a collection of variables given a group name, but for our purposes, it draws an explicit distinction between what is, and is not, accessible within a specific context.  You could, of course, do the same thing with a general forked application - but it still leaves us with the problem of needing to create a new remote procedure call mechanism.

Let's take the "remote" part out of the equation for now, and only talk about a forked application communicating with another copy of itself.  A procedure call mechanism in this context is a pipe that you periodically check (or block and wait), looking for a data stream in a fixed format.  That data stream will tell you what function to call, or what variable to write to, or what variable to read from, and it may also include parameters for the function (or an index to an array, etc) and then you return any out-values through the pipe to the other side with a similar mechanism.  Likewise, the calling side of the pipe will send a packet in the expected format down the pipe, and then block and wait for a return value (or merely an acknowledgement that the function is being run, or that it has completed successfully, depending on the context).

You as the programmer will most likely want to select some highly specific subset of functions or variables that you want to use with this mechanism - when building it yourself, you will most likely only accept some very few "commands" that you listen for and respond to from the other half of your application.  Because both halves are the same applicaiton, though, you could also use a lookup table that lets you unambiguously access any variable or function present on your half of the application - there are, after all, a fixed number of those functions and variables, and you know for a fact that all of them have unique identifiers, because you use those unique identifiers while programming.  And once you have selected a function and/or variable to access, any parameters that get sent along by the request can be type-checked, because you know the method signature or variable type.

Doing all of that work by hand would be an awful experience, and most if not all of that hand-done work would be wasted, as you are unlikely to call most of the functions or access most of the variables.  If however, it were a function of the programming language, compiler, and linker - that is eminently possible, because that is, essentially, the compiler's whole job.  Translating an identifier to a function, checking the number and type of the parameters, all of that is necessary to create the program in the first place.

Suppose, then, that you had a compiler that created an indexing function.  These kind of functions exist, when you use reflection tools - the indexing function would take a variable name or function and some arbitrary parameters, and call the function with those parameters, and then the indexing funciton would return the variable or return value when it's done.  Let's suppose at first that exactly one indexing function is created for your whole program.  You as the programmer might simply pass anything that comes down the fork pipe to the indexing function (unless you just called a function, in which case you would wait for the return value and parse that) - in fact, ideally, there would be a built-in function that just reads commands from the pipe in exactly this way, and a paired function that writes to the pipe in exactly the way the reader expects.

This is still relatively unsafe; there is a lot of potential for race conditions and so on, and there is nothing stopping you from requesting data from the fork that is actually stored somewhere else.  But assuming you had some need to fork an application and run it, this reflection-based indexing function would free the programmer from having to figure out how to pass messages back and forth, free them from having to write any middleware, and simply allow the programmer to use the capabilities of the other process, its data and functions.

But let's go back to the namespace thing.  You see, we are still left with the problem I started out on - that a forked process has duplicate identifiers, half of which are simply wrong, because the actual memory location where they are stored is in another process.  But, you know which process they are stored in, and you have a well-established mechanism for requesting data or calling functions in that process.  If you knew for certain at compile time that some subset of those identifiers would be in the forked process, the compiler might translate any requests for those identifiers to instead use the pipe-send function.

But wait, there's a problem: both forks of the application are running the same code, so at compile time, it's impossible to distinguish whether the request is going to be requesting data from the original or forked process.  This is where namespaces come in - you may not know whether a given namespace is in one process or the other, but you should know for certain that the contents of a specific namespace will all be in the same place.  It's only when you exit the namespace that there is any chance you will be trying to reach data that exists in another process.  That's simple, then: in the forking function, allow the programmer to specify which namespaces are being detached, resulting in two collections of namespaces, one associated with the original process, and one with the fork.  This will be a runtime check, in case there is any reason to change which namespaces get separated in different circumstances - because it is a runtime check, a checking function will be called every time any bit of code requests data or functions outside its own namespace.  (There are ways to reduce the overhead, but ignore that for now.)

Well, if you can do all this once, why not do it multiple times?  Say you have a program with five total namespaces and you want each in its own process.  There's no reason why that should be difficult, should it?  You would simply need the runtime check, the one that knows which process contains a given piece of data, to also select the appropriate pipe to use.  There is, admittedly, one complication - making sure that all forked processes receive updated routing tables - but since you've built a robust framework for sharing data already, that hardly seems like much of an imposition.

It's worth pointing out, though, that if this system works correctly, it should also work correctly if you never fork the application at all.  In that case, there is no confusion about which process owns which variable - all of them are stored in the same process.  So, your program will truly not care whether it is currently running in a single process, two processes, or five processes; when you as the programmer insist on calling a function or fetching data, the function will be called or the data fetched, no matter which process currently stores that particular piece of data.

Now, forking a process is necessarily something that happens in the same place where your application currently is, which sharply limits the usefulness of everything I've just described.  Not to say that it's useless - I think anyone who is interested in forking processes would appreciate having all of this functionality - but what I've just described is not actually the functionality that I want from the system.  A distributed application, as I've described it, is one where this forking process happens between two computers - where you can run part of the program logic on an entirely separate machine.  And conceptually, once you've done everything I've said above, that's not actually a difficult proposition... to an extent, at least.  You need some permissions, and you need to make certain that dependencies and the like are synchronized, but the technicalities of starting a process somewhere else, and binding a pipe that connects the two processes over the network, are not particularly difficult.

I've said this in other places, but to reiterate here, the value of doing this fork over the network is allowing your program to place specific bits of code where they are needed, without the program itself needing to care that those bits are not on the same machine.  Generally, these "specific bits" are in one of three categories: input, output, or computation.  The benefit I'm talking about is not accelerating your program with parallelization (you can do that if you have a whole lot of computing to do).  No, the benefit I'm talking about is a kind of program that we don't currently possess - one that replaces the idea of remote desktops and SSH tunneling with running an application in one place, and handling the input and output in other places.

This category of applications is something that should be simple.  If you want to edit files, you generally want the file editor to be where the files are, because it will involve a lot of reading from and writing to the disk; compared to that, GUI updates and keyboard and mouse inputs are much less sensitive.  I for instance have an NFS volume that holds many of my files attached to this computer, but if I want to unzip files on that machine, I tend to open a remote shell and perform the action local to where the files are, rather than having my desktop wait on network traffic in both directions for every file operation.  Likewise, some tasks that depend on accelerators, such as video editing, are best done when the data and the accelerator are in one place, but that doesn't necessarily require that the user is in that same place.  As long as input and output latency is within acceptable parameters, you can do a lot remotely - and it's only easier if the entire output stack is being run on the machine closest to the output itself, for the same reason.  A lot of system GUI updates, especially, are highly redundant; you can summarize the changes with a few bytes, while a single new image frame may be several kilobytes, and sending every single frame in an easily-reproduced GUI animation may be millions of times more expensive than doing it locally.

Instead of asking what is required to live in a world where this is possible, it is better to ask why we don't.  The legacy of computing simply assumes that everything will be in one place, and comparatively little of our development has been focused on splitting a program up into pieces distributed across a larger system.  We are encountering this need more and more, and so more and more people are getting exposed to it, but until the default way we write programs allows this kind of cross-computer access natively, accessing a program running somewhere else will continue to be a tedious affair.

And to be fair, this "conceptual walkthrough" is more difficult than it sounds, in that it requires tweaks to compilers and programming languages, on top of providing a mechanism to deploy those forked programs on another machine.  Those things happen, and there are experts in those fields, but if anyone is wondering why I haven't written my own code... rewriting program languages is simply a task that's beyond me!  Tasks that are much simpler for the compiler and linker would be an enormous headache to write by hand, but that doesn't mean that it's easy to change how compliers work.

But for the people who actually know how to do all these things... I imagine that if they just knew what they were trying to accomplish, they could change the world.  There is a lot that's possible, but which simply won't happen without the right support and tooling.

History of the MOS/ADA Programming Model

 This topic is tricky to introduce.  One of the central tenets to Project MAD as it is today, is a whole new model for applications, and I've said that it was the last things that came together, and it tied everything together in a neat package.  What you won't know is that my early concepts of the MOS/DCA project also included a modification to standard programming models, and that legacy is still very obvious once you know to look for it.

It's difficult to ferret out exactly what I meant by it if you're just reading my notes from back then, but it was there: I wanted to expose the internals of executables and shared libraries so that, say, a shell command could execute an arbitrary function.  Important to making that work is having an interactive shell that can capture typed data - more like an interpreted language's REPL loop than something familiar like Bash or Windows Command shell.  And this has to be a structured tool, in some fashion; in modern applications programming, you pass nothing to an application when it starts except strings, and the application has to figure out literally everything from that.

This ties back nicely to my complains in the last couple blog posts about programmers being obliged to reinvent the wheel.

At the time, what I had wasn't anything like a plan.  At most, I had a complaint.  And that sparked a wish.  After all, especially within the GNU toolset, how many applications are actually just wrappers around library functions?  Complex wrappers, sometimes; that complexity is frequently needed so that this one tool can interact with a complex programming library function in multiple ways, or so that common data pipelines (piping a list of files through grep, as an easy example) operate smoothly and the way you would expect them to.  As with a lot in programming, things that are simple to say are frequently hard to do, and if you want to actually do them, there is a lot of nuance and complexity that must be navigated.

The other side of that, though, is that the nuance and complexity spawns more nuance and complexity.  Your complex tool needs to be interacted with in complex ways.  Add a third layer on top, that assumes access to tools that use libraries; then add a fourth layer and more, tools on top of tools.  As long as nothing changes, eventually you get a level of tooling where you can issue a one-word command to do exactly what you mean to.

But every level in between the core library and the final level of tooling can shift.  Needs to shift.  Every layer must be updated whenever any of its dependencies update, for security reasons if nothing else, and sometimes bigger shifts are called for.  To a certain extent, there's no getting away from that - but if it were possible to have the library function that you actually care about, be executable in its own right without an application wrapping it, there can suddenly be whole levels of redundant tooling that are at least simplified, if not eliminated.

And of course there will probably be several other layers of tooling that crop up around this new concept; I won't lie.  Ultimately, we create tools because we have specific needs, and as long as the tools fulfill specific rather than general needs, they will diversify.  But many of those pain points that we are removing are redundant ones.  Every application needs to transform text input into data streams.  Applications that read from and write to data files have to pick or create libraries to that work with the specific protocols and data streams.  And if you're creating a workflow that involves data files being processed, you need individual applications whose only role is to be a part of that workflow, applications that parse files and put them in a form other applications can take as input.

Many of these niche applications, that only exist to take one data format and make it available, or take text input and render it back into a data file, are at once important to someone's workflow and possibly security, but also are tools nobody wants to maintain.  Nobody wants to need to maintain them.  It's important that someone creates a tool that parlays between data files and workflows, but it's a thankless job, and a product nobody wants to pay for.

In short, a shell that understands data objects and types, and can use that to call a function built into an application or library directly, instead of needing to build one application after another to handle the workflow… that would solve a lot of problems.

The steely-eyed veterans in the audience are saying, “Well, we have exactly that with interpreted languages like Python.  In Python, you can open a library and call a specific function out of it.  If you have a python file that is meant to be an executable, you can also load that as though it were a library, and call functions out of it, assuming it's designed right.”  And this is, of course correct.  (For the record, I had these thoughts before I was introduced to Python.  Probably.  Around that time, at the latest.  Again, my notes are not great.)

The difference is having that be an assumption built into the entire operating system, into the fundamental programming method, so that the entire system is built on one foundation instead of a shifting mix of dependencies from various legacies.  That is, of course, making a virtue of a vice: it means starting over and doing everything from scratch, which is... painful.  And these initial thoughts of mine were missing critical parts of the picture, which is why for many years I set them aside as irrelevant and a waste of time.  At the time, all I was thinking was that, doggone it, we shouldn't have to reinvent the wheel over and over.

And in truth… there are still today whole categories of errors that only exist because library functions are sealed away.  For example, languages like Python, PHP, Go, and many others can't easily use functions buried in libraries for other languages like C; people have to write new versions of old libraries, and each needs to be maintained separately, by different people, using different language constructs and different dependencies.  Ultimately, it becomes such a different problem to maintain each of them that even if one person was willing to maintain every copy of the same algorithm, it would become untenable for some fraction of them.

All this despite the fact that ultimately, all you want is to pass parameters in and get results out.  If there were no worry about dependencies or languages, if all you had to do was call the function and capture the return, it would be simple to do from any language.  Of course, from a certain point of view… that's always been all you have to do, if your language is willing to parse library headers or code from other languages.  But the library formats in use, Linux's .so and Windows' .dll, neither comes with a human- or machine-readable directory of functions meant to inform programmers of how to use them.  If you want to know how to use them, you need specific files intended for developers to use.  That means if you have an entirely different programming language and they want to use existing library functions, they need to read those files, …despite them being in another language.

In short, there's a reason why we are where we are.

When I talk about Project MAD as a change in the entirety of computing, it can sound like I'm vastly overestimating my own cleverness - and well, maybe.  But these are real problems.  My first concept almost sounds promising on its own: just call library functions.  But that concept does nothing about compatibility and discovery problems.  It hardly matters whether or not the code exists on your system if you don't know that it's there, where it is, what it does, how to call it, what side effects to expect, and what the return values and errors mean - and that assumes that errors get propagated correctly.  And the details need to be exact; one bit out of place changes everything.

Just like there's a reason why we got here, there's also a litany of reasons why we haven't leaped forward.  Even if we all agreed on one bright idea that we would work together on, it would be a massive effort with a lot of problems that need to be solved.  And if the bright idea that we all agreed on happens to have massive holes inherent to it, because it's not as bright an idea as we thought… that's a lot of work for nothing.

That's to say, I do understand, have always understood, and will always understand, that none of this is simple, that I can be wrong, and that ultimately, even if my idea was core to anything, it would be the people who actually achieve it that deserve the credit, because it will be very difficult.

But also…

But also, various subsystems that are a part of the MAD concept (here, specifically, the Modular OS and Agentic Distributed Application model) are there specifically to solve these problems.  I've described the system directory in this post on the Unified System API, but not all parts of the solution came around at the same time.  Part of it, as I've just described in this post, was there from near the beginning.  Many years later (five to ten, maybe) I started thinking about the directory, and specifically, having types understood system-wide and having them be a central part of the system directory.  Before this, all I really had was “Call a function inside a library, so we don't have to reinvent wheels.”  But it didn't make sense, quite.

I said in my first post that the ADA came about in the process of writing blog posts about the MOS/DCA concept.  I was trying to explain, and things still felt incomplete.  I had deduced that functions and hardware drivers needed to be sorted by function, which fed into the type hierarchy in the system directory, and I was also thinking about embedded libraries and applications, and how that made application deployment and dependency management neater.  I was writing about wrapping functions with scripts so that some logic was happening on remote nodes, but that wasn't working for me.

The reason why I like the ADA goes beyond the model itself.  The model slotted well into the weirdly shaped hole that the whole “software model” side of the project had.  Yes, you could call functions directly instead of only being able to run applications - but why?  What made that so compelling, and not merely one person ranting at the sky?  Sure, it would be nice to have a directory of all the functions and types within a system, but what made that so critical to the operation of an OS that it has to be a fundamental component?

But within the ADA, you may be calling functions within your own application agents and within others'.  You will be deploying agents to hardware modules specifically because there is software on that module that is required by that Agent.  And it goes beyond functions and code (or, well, it doesn't, but it helps to think of this way): the ADA is also about exposing data so that applications can be monitored and debugged, but that requires types to be in some sense objective, even if that only means a type connected to the application and not a system-wide one.

Equally, within ADA, an Agent exists to handle the tricky problems of accountability.  There is an entirely reasonable question when you try to, for example, perform GUI API calls remotely.  The memory being used by your GUI application can't leave the display controller, or at the very least, there will be a copy of the output in the display controller, which must be managed by the GUI stack itself--while being owned by an external party.  Without an agent such as the ADA's, there are massive, massive questions to be answered about how this is not an absolutely terrible idea.  But if you can't do things like call GUI functions remotely, then... what even is the plan?

But perhaps most importantly, in the ADA, deploying Agents and routing messages is a system function, one attached to the ADA servers and not one implemented by applications themselves.  That means that knowledge of functions, data, and types needs to be standardized at a system level.  If the ADA model were implemented separately by each application, then each application would need to separately track and manage type and API version compatibility.  Implementing this as a system function makes things not only possible, but easy for programmers to handle, by doing the hard work ahead of time, in the operating system.

I like to, perhaps arrogantly, think that I am touching on fundamental truths with some of these analyses.  It's why I keep hammering, for example, on don't force every programmer to reinvent the wheel.  It is hubris to say I alone know best (I've demurred enough that you know I'm not that hubristic), but it's still satisfying to line up a whole bunch of problems in a row, pointing at them, and say, Example one, example two, example three.  Tying things up with a bow on top is satisfying, and the point of feeling genuinely excited about the ADA and Project MAD in general is that a stack of problems are placed in a box and wrapped neatly up.

It also feels good because this isn't all the work of a day, month, or year.  While this talk of modifying programming models may not have happened in the first year of the project (the first part was mostly the DCA, honestly), it's at least 15 years in the making.  Tricky problems that have tickled the back of my mind for years are coming together.  These are problems that weren't just tricky in practice.  They were tricky conceptually; it wasn't clear that there even was an answer.  It wasn't hard to imagine that I was wasting my time, because there was no clear vision of the end.

There's still a lot rough, and there's still lots of room for me to be wrong.  But it's astonishing how the last few pieces have made the puzzle come together.  At least from where I stand, it looks promising, like there's really an answer.  Not an easy one, but one that leaves us all much better off.

And even if I'm wrong, perhaps getting more eyes on the collection of problems I've been working through will help.