Conceptual Walkthrough: Distributed Applications

It may be surprising, since I am here to talk about distributed computing, but I hate process forking - making copies of a running application, each running the same code but each dedicated to a different task.  I'm sure I'm not alone in this, because it's not as though people are running around calling fork() for every little thing; multithreaded and multiprocess apps are not exactly common, not unless you're doing something that really requires such a technology.

Frankly, it's a headache to try to understand.  In theory, it should be simple; there's a copy of your program, but you know which copy is which, so you just do different things with them.  If before that, you open up a pipe, both copies of the application will have access to the same file descriptor for that pipe, and they can communicate through it.  But all of those things being technically possible isn't a plan.  When that pattern is laid out in a book, it leaves you wondering: what am I supposed to do with it?

I talked in several places about programmers being obliged to reinvent the wheel, and process forking is a good example; forking wouldn't be such a terrible thing, if it was paired with a functional remote procedure call mechanism.  When two copies of the same program are connected with a pipe (or two), you can use it to pass data, but that data transfer is mostly pointless without context.  Adding context, labeling the data, makes it essentially a very simple, poorly implemented remote procedure call, where the "function" that is being "called" is simply an index specifying what to do with the data, out of a set of options fixed at compile time.  While process forking would still be a bit confusing if the process forking mechanism came with a built in RPC mechanism, at least the programmer would not have to wrestle with all the complexity of implementing their own RPC stack from scratch while also trying to build program logic around it.

Although I wasn't thinking of it when I designed the system, the ADA distributed applications model is basically exactly this, and to help you understand it, let's walk through the analogy a little more closely.

As I see it, forking has two fundamental problems with it; first, the fork is undifferentiated.  It is easy to overlook the fact that it is a feature of programming languages that we can attach variable names to data in the first place; when I say that forks are undifferentiated, what I mean is that this deliberate feature and mechanism of programming languages no longer functions.  In each half of the program, a large number of symbols exist but point to the wrong data, and it is entirely up to the programmer to recognize this and keep track of which symbols are now incorrect.

This is in fact the reason why I have specified that under the ADA, programs should be written and compiled such that each module which is to be depoloyed separately, shall have its own variables confined to a single namespace.  Namespaces in this context may be nothing more than a collection of variables given a group name, but for our purposes, it draws an explicit distinction between what is, and is not, accessible within a specific context.  You could, of course, do the same thing with a general forked application - but it still leaves us with the problem of needing to create a new remote procedure call mechanism.

Let's take the "remote" part out of the equation for now, and only talk about a forked application communicating with another copy of itself.  A procedure call mechanism in this context is a pipe that you periodically check (or block and wait), looking for a data stream in a fixed format.  That data stream will tell you what function to call, or what variable to write to, or what variable to read from, and it may also include parameters for the function (or an index to an array, etc) and then you return any out-values through the pipe to the other side with a similar mechanism.  Likewise, the calling side of the pipe will send a packet in the expected format down the pipe, and then block and wait for a return value (or merely an acknowledgement that the function is being run, or that it has completed successfully, depending on the context).

You as the programmer will most likely want to select some highly specific subset of functions or variables that you want to use with this mechanism - when building it yourself, you will most likely only accept some very few "commands" that you listen for and respond to from the other half of your application.  Because both halves are the same applicaiton, though, you could also use a lookup table that lets you unambiguously access any variable or function present on your half of the application - there are, after all, a fixed number of those functions and variables, and you know for a fact that all of them have unique identifiers, because you use those unique identifiers while programming.  And once you have selected a function and/or variable to access, any parameters that get sent along by the request can be type-checked, because you know the method signature or variable type.

Doing all of that work by hand would be an awful experience, and most if not all of that hand-done work would be wasted, as you are unlikely to call most of the functions or access most of the variables.  If however, it were a function of the programming language, compiler, and linker - that is eminently possible, because that is, essentially, the compiler's whole job.  Translating an identifier to a function, checking the number and type of the parameters, all of that is necessary to create the program in the first place.

Suppose, then, that you had a compiler that created an indexing function.  These kind of functions exist, when you use reflection tools - the indexing function would take a variable name or function and some arbitrary parameters, and call the function with those parameters, and then the indexing funciton would return the variable or return value when it's done.  Let's suppose at first that exactly one indexing function is created for your whole program.  You as the programmer might simply pass anything that comes down the fork pipe to the indexing function (unless you just called a function, in which case you would wait for the return value and parse that) - in fact, ideally, there would be a built-in function that just reads commands from the pipe in exactly this way, and a paired function that writes to the pipe in exactly the way the reader expects.

This is still relatively unsafe; there is a lot of potential for race conditions and so on, and there is nothing stopping you from requesting data from the fork that is actually stored somewhere else.  But assuming you had some need to fork an application and run it, this reflection-based indexing function would free the programmer from having to figure out how to pass messages back and forth, free them from having to write any middleware, and simply allow the programmer to use the capabilities of the other process, its data and functions.

But let's go back to the namespace thing.  You see, we are still left with the problem I started out on - that a forked process has duplicate identifiers, half of which are simply wrong, because the actual memory location where they are stored is in another process.  But, you know which process they are stored in, and you have a well-established mechanism for requesting data or calling functions in that process.  If you knew for certain at compile time that some subset of those identifiers would be in the forked process, the compiler might translate any requests for those identifiers to instead use the pipe-send function.

But wait, there's a problem: both forks of the application are running the same code, so at compile time, it's impossible to distinguish whether the request is going to be requesting data from the original or forked process.  This is where namespaces come in - you may not know whether a given namespace is in one process or the other, but you should know for certain that the contents of a specific namespace will all be in the same place.  It's only when you exit the namespace that there is any chance you will be trying to reach data that exists in another process.  That's simple, then: in the forking function, allow the programmer to specify which namespaces are being detached, resulting in two collections of namespaces, one associated with the original process, and one with the fork.  This will be a runtime check, in case there is any reason to change which namespaces get separated in different circumstances - because it is a runtime check, a checking function will be called every time any bit of code requests data or functions outside its own namespace.  (There are ways to reduce the overhead, but ignore that for now.)

Well, if you can do all this once, why not do it multiple times?  Say you have a program with five total namespaces and you want each in its own process.  There's no reason why that should be difficult, should it?  You would simply need the runtime check, the one that knows which process contains a given piece of data, to also select the appropriate pipe to use.  There is, admittedly, one complication - making sure that all forked processes receive updated routing tables - but since you've built a robust framework for sharing data already, that hardly seems like much of an imposition.

It's worth pointing out, though, that if this system works correctly, it should also work correctly if you never fork the application at all.  In that case, there is no confusion about which process owns which variable - all of them are stored in the same process.  So, your program will truly not care whether it is currently running in a single process, two processes, or five processes; when you as the programmer insist on calling a function or fetching data, the function will be called or the data fetched, no matter which process currently stores that particular piece of data.

Now, forking a process is necessarily something that happens in the same place where your application currently is, which sharply limits the usefulness of everything I've just described.  Not to say that it's useless - I think anyone who is interested in forking processes would appreciate having all of this functionality - but what I've just described is not actually the functionality that I want from the system.  A distributed application, as I've described it, is one where this forking process happens between two computers - where you can run part of the program logic on an entirely separate machine.  And conceptually, once you've done everything I've said above, that's not actually a difficult proposition... to an extent, at least.  You need some permissions, and you need to make certain that dependencies and the like are synchronized, but the technicalities of starting a process somewhere else, and binding a pipe that connects the two processes over the network, are not particularly difficult.

I've said this in other places, but to reiterate here, the value of doing this fork over the network is allowing your program to place specific bits of code where they are needed, without the program itself needing to care that those bits are not on the same machine.  Generally, these "specific bits" are in one of three categories: input, output, or computation.  The benefit I'm talking about is not accelerating your program with parallelization (you can do that if you have a whole lot of computing to do).  No, the benefit I'm talking about is a kind of program that we don't currently possess - one that replaces the idea of remote desktops and SSH tunneling with running an application in one place, and handling the input and output in other places.

This category of applications is something that should be simple.  If you want to edit files, you generally want the file editor to be where the files are, because it will involve a lot of reading from and writing to the disk; compared to that, GUI updates and keyboard and mouse inputs are much less sensitive.  I for instance have an NFS volume that holds many of my files attached to this computer, but if I want to unzip files on that machine, I tend to open a remote shell and perform the action local to where the files are, rather than having my desktop wait on network traffic in both directions for every file operation.  Likewise, some tasks that depend on accelerators, such as video editing, are best done when the data and the accelerator are in one place, but that doesn't necessarily require that the user is in that same place.  As long as input and output latency is within acceptable parameters, you can do a lot remotely - and it's only easier if the entire output stack is being run on the machine closest to the output itself, for the same reason.  A lot of system GUI updates, especially, are highly redundant; you can summarize the changes with a few bytes, while a single new image frame may be several kilobytes, and sending every single frame in an easily-reproduced GUI animation may be millions of times more expensive than doing it locally.

Instead of asking what is required to live in a world where this is possible, it is better to ask why we don't.  The legacy of computing simply assumes that everything will be in one place, and comparatively little of our development has been focused on splitting a program up into pieces distributed across a larger system.  We are encountering this need more and more, and so more and more people are getting exposed to it, but until the default way we write programs allows this kind of cross-computer access natively, accessing a program running somewhere else will continue to be a tedious affair.

And to be fair, this "conceptual walkthrough" is more difficult than it sounds, in that it requires tweaks to compilers and programming languages, on top of providing a mechanism to deploy those forked programs on another machine.  Those things happen, and there are experts in those fields, but if anyone is wondering why I haven't written my own code... rewriting program languages is simply a task that's beyond me!  Tasks that are much simpler for the compiler and linker would be an enormous headache to write by hand, but that doesn't mean that it's easy to change how compliers work.

But for the people who actually know how to do all these things... I imagine that if they just knew what they were trying to accomplish, they could change the world.  There is a lot that's possible, but which simply won't happen without the right support and tooling.

Generic Runtime Linking

 My last post, about a Distributed Application Model, was an attempt to separate one of several good ideas buried in Project MAD in hopes of making them more accessible to others.  This post is similar; in many ways it is reframing what I said in the last time, with a specific focus on explaining the core mechanism and how it flavors everything else that is going on in the MAD OS and MAD/ADA system.

We are talking, today, about dynamic linking under Project MAD, or if you prefer, MAD/Libs.  And to begin, let's talk about how this normally works, as I understand it.

Preface: Standard Runtime Linking

Programs (let's leave aside scripts for now) are normally written in a human-readable language and then translated into machine code in a process called compiling; frequently, there will be multiple, separate bits of machine code generated when you compile a program, as programmers like to split things up into logical segments.  Because the machine code is still structurally similar to the layout of the human-readable program, you can still recognize that in the resulting machine code, for example, a particular function starts here, and a particular bit of data is there.  An index of these facts about the machine code - where things are and what each of them is - is included with this compiled object.

When one of these compiled objects wants to reference another, you must necessarily be very precise and specific, in two parts: you must know what other object file contains the thing you want, and then know how to find that specific thing among all the other things inside of that file.  During the process of compilation, it's easy to keep track of this all automatically; a mapping is made between an identifier and where exactly to find it, in both human-readable and machine code forms.  Thus if you want to combine several objects together, you simply go through the object's dependencies, and find which file to look at, and when you combine the objects together, you ensure that the function call that was previously left as a dependency is translated into an actual function call within the final object.  This process is known as linking.

There is a form of linking at compile-time which depends on code that the user themselves did not create (which is called a library); this process is known as static linking, and it is, ultimately, the same as it is when you link together objects that the user created, except that the library isn't something that you just created and so you have to know where it is.  The process of specifying and finding these libraries is standard, and the details get into operating system and filesystem standards; common libraries are usually stored in standard places, and you only need to reference them unambiguously.

If, at compile time, the library you say you need cannot be found, the compiler makes a fuss and everything stops.  Mostly, the list of these library files is stored in configuration files along with all the human-readable program source files, and the linker program just checks with the OS to see if they can find the files you want.  And of course, if someone else wants to compile your program (assuming you share the source files), they can reference this list and go looking for exactly the libraries they need to compile the program, but assuming they have or get all those libraries, they will resolve all the dependencies your program has and come away with a finished, working executable.

A dynamic or run-time linking model alters this last bit.  It allows you to create an executable that still has dependencies; the executable is considered “finished”, but when you go to run it, the process is interrupted until the system can find a copy of the libraries it depends on and link them in.  Because this process happens on computers that the programmer doesn't control, if anything goes wrong at this stage, it can be a little harder to figure out and fix, but we generally have the process figured out nowadays.  (In theory.  Mostly.  Kind of.)

Other than that, dynamic linking is only slightly different than static linking.  Because these links are designed to be made when the program is loaded rather than having the canonical on-disk file altered, they are often done with function pointers - which treat the concept of where to find the function as data which can be modified, as opposed to a static linking model where these offsets may be metaphorically set in stone.  Even then, though, the system must find and load the file containing those functions, before it's possible to update those function pointers so that they will work correctly for the program that depends on them.

There are a few added complications, but by and large, this is representative of the process that we call dynamic linking, and to summarize one last time, it requires you to keep track of identifiers (as originally written in human-readable language) and what file to look for them in, and it requires those files to provide a listing that translates the identifiers into pointers for functions and data which can be stuck into the program to allow it to run.  Ultimately, this is what it takes for a program to call a compiled function stored in another file; to put it in a word, it's just recordkeeping.

Problems With the Extant Model

The model as we use it today assumes that linking is and must indefinitely remain highly specific.  It simply does not do for someone to link any other object file in place of the requested one; it is assumed that you are referencing some complex bit of code that works on every client machine in exactly the same way as it worked on the developer's machine, and nothing else will do.  This is not without reason; it's really problematic for a program to work differently on some random computer as it works for the programmer.

The problem is, because the linking process is very specific, operating systems and programming tools lack the features that would be necessary to facilitate generic linking.  The concept does appear in scripting languages, where linking isn't a thing, and there are concepts in programming that do the same thing within a given project (using dynamic function and data tables, again), but it remains a presumption that everything will be done exactly the way the programmer originally intended - and again, this is not without reason.

There problem is, this non-generic structure is remarkably flimsy.  Too many things are assumed and never tested; there is a lack of formal structure that allows you to describe things in unambiguous, but still flexible, ways.  As a result, there are tons and tons of small, specific, things, many of which depend on other small specific things, and if any of those small specific things break, much larger things that you would not expect to be fragile may also break.

Anyone who uses Linux (and related systems) to any degree has encountered situations where they go to update their systems and receive a deluge of small updates.  If one were to examine many of these updates, it can be alarming just how many are redundant; there are libraries that do the same or similar things but have different interfaces, libraries that do the same thing in different programming languages (especially scripts, which are wholly incompatible with one another), thin front-end executables that expose library functionality to shells (usually exposing only one specific library per executable), services that just sit around exposing a library's functionality, and libraries that compete with one another and may have some incentive to be incompatible.

What makes this system fragile is that none of these copies or similar functions can be easily swapped out.  If a given library used by a given program becomes a problem for whatever reason, everything that depends on that specific library fails, because there is no easy way to transition from using that library to using an equivalent library.  You can roll back to an older version… but that's only a patchwork, a temporary fix.

Why are they all incompatible?  The easy answer is because nothing is standardized, and that's correct, but I will say it differently.  There is no standard language for describing the dependencies of a program.  Enumerating them, yes; any given executable will tell you that it depends on function X in library Y.  But that enumeration of the dependencies does not describe anything.

So long as we cannot describe our dependencies, it is impossible to offer a generic alternative.

A Standard API Is A Generic Dependency

It's probably clear to existing readers of my blog that I have been describing for a bit, the idea of standardizing APIs.  Anyone who is new will come to know this concept as the System API Directory, also known as the MAD/SAD subproject.  But what specifically am I advocating for?

Suppose that your program wants to play a sound file; suppose that sound file is provided by the user, and you do not know in advance how the sound is encoded.  When I talk about describing a dependency, I mean that your program wants to be able to say, “I want to link to a library that will play a sound, as it was stored in a file.”  There are extant libraries that provide programmers a “one-stop shop” for various sound file formats, but depending on such a library is not the same as describing the intent of your program.  You remain dependent on that one specific thing, which itself has many additional dependencies, some of which your own program will never use.

Just as your program cannot be linked with just any library, the library was not written to be used in just any arbitrary way.  The library was designed to be used some specific way, and that specific way becomes the structure of your program.  Once your program is compiled for one library, there is no going back; even if you had another library with identically named functions, it is impossible to know for certain that they are compatible.  (Technically, yes, you could try to fool the linker, and perhaps even succeed - but it is not how the system is designed.)

When I talk about a standard API, I am talking about something that exists to guide the library programmer as much or more than it guides the applications programmer that will depend on it.  It does more than simply telling the library programmer what names to call their functions, and the applications programmer what the function name to call is.  It informs the structure of both projects; it describes when memory is allocated and when it is released, what guarantees can be made, what may or may not be taken for granted, and how errors should be handled.  Only when both sides agree on these kind of terms can a specific library be replaced with a generic alternative.

The language of APIs was always meant to describe exactly these sorts of problems, but to date it has largely been left to creators to decide what's in them.  And the point I'd like to make right now, is that until standards exist, libraries cannot be tested against and held to those standards.  And so long as libraries cannot be held to standards, you cannot use them generically, only in the highly specific ways they were designed to be used.

When you look at the state of code libraries, you will see that they must, will, and have created their own ad hoc standards for almost every task.  What is perhaps most distressing, however, is how commonly these ad hoc standards are literally the first thing that came to the creator's mind.  Once the library has been written, and applications linked against it, it becomes painful to change the API and force all the applications using it to rewrite their own code.  Even before it sees any kind of widespread usage, altering the API may mean making large structural changes to how the library was originally laid out.  If there is no impetus to do so, it is most likely that the programmers never will, and whatever form the idea first had, it will continue to have indefinitely.

I choose to place the blame for the “fragility” of non-generic interfaces and linking schemes on this problem: too many have unique and bespoke interfaces, with no agreement and no impetus to agree on what those interfaces should look like.  Programs that depend on some arbitrary set of libraries may find that in different places in their code, the same task is performed by different sub-dependencies, each working slightly differently, and those differences may require different structure around each function call.

I do not intend to disparage individuality in programming methodology; having multiple ways of doing a similar task is valuable, and arguably critical for long-term success.  However, having these multiple methods be fundamentally incompatible, especially where it need not be so, is problematic.  Indeed, I have long thought that one of the benefits of this kind of generic linking is giving users (not merely programmers) the ability to swap in a more optimized version of some algorithm when it is discovered, or to swap in a more transparent version if debugging or monitoring a program.  (There is a security argument to be had here at another time.)

I am also not necessarily suggesting that there be one ultimate and irrevocable interface for some given class of problems moving forward.  Rather, I am talking about deciding upon some minimum reasonable standard which defines the very concept of utilizing a technique, algorithm, process, resource, or device.  For sounds, for example, this minimum standard may include playing the file, pausing and stopping, playing repeatedly, getting or setting the current playback position, and changing the volume.  This list by no means describes everything you may want to do with a sound file, but it is a fair minimum standard (and I would gladly listen to arguments for expanding or shrinking it).

Of course, going beyond the minimum is a matter which brings its own concerns.

Expanding the Language of APIs

It was never my intent, and it remains not my intent, to claim to know how best to organize APIs, or to lay down firm standards without discussion.  As such I hope you will take the following section as informal, tentative, and conceptual.

I have talked in the past about the MAD System API Directory as containing a hierarchical tree of types, from most generic to most specific, and to this concept I still hold.  The nature of this kind of tree makes it technically possible that all possible interfaces can be categorized into rough categories, but this alone may not prove good enough for the task of describing a large, diverse, and robust system.

Originally, I envisioned that the first few layers of this tree were broad categorizations, and once you pass a certain point, then programmers are free to create various child standards with whatever additions to the parent standard they wish to add.  This has the problem, however, of not providing granular extensions to an interface; part of the intent behind the broad categorization is to establish simple baseline APIs that allow programmers to avoid interacting with unused features, for example when reading documentation or when experimenting with the functions of a library.

If you don't have some method for granular extensions, then you are obliged to create complex child standards almost immediately.  For audio files, for example, one program might have a specific need to pitch-shift an audio file (for example, to tune a musical instrument sample to different pitches), but otherwise, you might only use basic capabilities of an audio playback library.  Another program might need to cleanly loop audio, potentially with defined entry and exit audio segments (for example background music, with an intro and coda).  If these functions are not part of the baseline, in a child-only model, it stands to reason that someone will immediately create a “everything and the kitchen sink” child API that includes both of these features and tons more.  Worse, two similar but unequal child APIs may be mostly but not completely compatible, and there is no language to describe their intersection or extremities.

Although my experience with API design is minimal, I cannot help but think that the language of API design itself needs to be extended.  The above problem, for example, seems like it is begging for a concept of “features” which describe granular additions to an API; a child API may provide, or a program may require, some set of those features, but more to the point, the baseline API standard can be read and understood absent technical discussion of advanced features.  Whether or not features in this sense are the right way to accomplish this, I believe that finding some solution to this problem is laudable.

I have other similar thoughts - enough that it would complicate this section and blog post.  Some are already discussed elsewhere on this site, others will be described at some point.

But to summarize this for right now, remember that we are talking about making libraries into something predictable and testable.  That's all that's really required for generic linking; as long as you know what will happen when your program calls a function, then you can move away from depending only and very specifically on the same file from the same vendor as you originally designed the program to use.

But for all that I've talked so far about the problems with what we have, I haven't talked much (in this post at least) about the benefits of a modular, genericized library structure.

I Did Call It MADLibs

I talked in the last post about making devices accessible using inter-process communication and remote procedure calls.  Without rehashing (or if I'm being honest, rereading) that whole argument, I'd like to point out that the idea of generic linking fits directly into the idea that devices (in the OS sense, which may include things like files) have inherent capabilities.

Generic linking suggests that anyone writing library code knows how to expose functionality in such a way that if a program depends on some specific capability (in the API sense), it can know whether your library provides that specific capability.  Devices necessarily have some kind of driver, even boring devices like files; the driver describes the capabilities of that device.  Thus it stands to reason that a library should be able to make the claim, that it provides a device specific capability, when paired with some device of that type.

At this point, we have returned to this post about assigning types to every filesystem object.  And while that kinda is the point, I'd like to phrase it a certain way right now:

Libraries which provide device-specific capabilities are verbs that act upon a noun.  Any given program may start with a specific verb and allow the users to fill in a noun, but you can also create a list of all device-specific capabilities that anyone claims to offer for a specific class of device.  In other words, a highly generic program such as a user shell (file browser, etc) can start with a noun and allow the user to select any appropriate verb, provided by any library that is available on the system.  The user, of course, has a lot of files and devices available to them; they have a lot of nouns, each with a comprehensive list of verbs.

I said in my last post that “it may surprise people who don't look at how OSes work that an average programmer probably doesn't know how to get a list of all devices present on a given machine, can't tell what a random device is capable of, and can't figure out how to control that device.”  This is the culmination of that indictment of modern computing: A Madlib-style system, mine or otherwise, that allows you to cultivate a list of verbs that work with nouns and nouns that work with verbs is absolutely possible, it just doesn't exist in systems as we have them today.  Certainly not in the generic case, where you expect the same methodology to work as well with services and hardware devices as it does with documents and media files; the “file associations” structure that modern operating systems use is primitive in comparison.

It is also important that the Madlib-style system provides a rigorous way for the system itself to know how to provide a given capability for a given device.  Remember that Project MAD is distributed; if you want to consume a capability on a remote piece of hardware, you need to know what, if anything, you need to bring with you when you send a program agent out to consume that capability, and you need to know what to expect when you get there.  If for example a remote node has a camera, and some nearby tertiary node contains a library that knows can provide a specific capability for that camera, you may need to copy the library over to the camera host and load/run it, in order to get the required capability.

It is important to consider this, because the library you're talking about may have multiple dependencies.  The camera capability you want to consume, for example, may also require an AI accelerator chip, which may not be present on the camera node, but it may exist in the larger system.  If you want to consume this AI analysis of the camera feed, a separate setup process must first begin which connects the camera to the AI accelerator, and then your program to the output of the AI accelerator.

Under MAD, this scenario makes sense.  It is reasonable to expect this stack of programs, libraries, and drivers to deploy automatically when you ask to consume what appears to be merely the capability of the camera.  It makes sense that you can simply use a verb on a noun.

If someone else finds a way to do the same… I welcome it.  Ultimately, I really just want computers to be better.  In the meantime... I hope this sparks at least some thought on the subject.

Why MAD?

This is another blog post that I've ended up banging my head against, writing and then discarding multiple, relatively long revisions.  Unlike the last several, however, this is not about describing a technical component of Project MAD; it's about the opposite.  The truth is, when you distill a distributed applications model down to its essence, it sounds so simple as to be irrelevant. And indeed, the model - absent proper support - might do little enough.  It really needs to exist inside of a context that makes it more valuable.

And the funny thing is, a properly supported model for distributed applications is insanely valuable.  There are thousands of examples, spread across hundreds or thousands of different domains, where we are having difficulty writing applications that combine the resources of two or more computers together.  In the world today, we are expecting and hoping that companies and open-source projects will each create solutions to each of these examples on a case-by-case basis.

Those are what I call a special-case solutions to the problem of distributed applications.  What I am searching for is the general-case solution, one that fits, if not all of these examples, most of them.  A single technology would make it easier to create solutions for these problems, but alone, a distributed application model still expects - nay, demands - that programmers solve the hard problems involved in making applications work across long distances.  Those hard problems are the kind of thing that need to be solved centrally, on an operating system level, so that each application programmer can go about doing the thing they are there to do.

But it's worth laying out the argument, as best I can, piece by piece.

Why Distributed Applications?

I said above that a properly supported model for distributed application would be insanely valuable.  It's worth running through some examples of things that are wrong with modern computing that come down to, essentially, the lack of general distributed applications.

There are a lot of examples of simply inadequate hardware.  My dear mother has a relatively recent laptop that isn't upgradable, and is now all but useless for a non-technical user.  (I'm sure I could install Linux and it would work fine, but she doesn't want it)  That's an example of something that's too cheap; but modern GPUs are getting too expensive because they are trying to get a single chip to do a massively parallel task all at once.  If someday distributed applications (generically, not counting special cases) can split their GPU load among multiple processors as easily as they can put it on a single one, a lot of people will be a lot happier.  Making applications distributed likewise helps with inadequate memory as much as inadequate CPU threading; you still need the same amount of memory, but today you may already have the excess, perhaps locked up in old laptops or desktops where there's no good way to use it.

It goes beyond computers that are poorly designed.  Smart devices, including TVs, watches, glasses, car dashboards, VR headsets, and so on, are not generally meant to run an entire generic PC for their user, but they have enough hardware to serve as a decent front end - but the process for making good use of them is spotty at best and terrible at worst.  It isn't even, to my knowledge, acknowledged that these are all examples of the same class of problem: finding a good way to make use of compute, display, and sensors distributed across many devices.  That's part of what I'm trying to solve, frankly.

There's also ownership of data.  Home cloud systems exist (I use one), but larger companies put a lot more development work into the proprietary versions, so those end up more advanced and user-friendly.  A large part of making these cloud-compute systems, however, involves solving the fundamental problems of distributed applications, if only in a specific case - finding a free compute node, authenticating users, deploying and updating applications, error handling, and so on.  If there is a general-case solution, and one that users can deploy at home or in small businesses with relative ease, then the value proposition of cloud computing isn't as strong.  This not only relieves end users, who don't have to trust the companies, but it takes companies out of the business of both safeguarding and somehow monetizing other people's data, which is an unfortunate contradiction to be in charge of.

The last example I'll point out for now is administration.  It's not trivial for non-technical users to manage home servers, not least because managing the server is simply a separate task from managing your PC, so it's something that home users need to explicitly remember to do.  Where your PC might (with varying levels of obnoxiousness) force you to update apps, drivers, and services, currently any machine that isn't in the foreground must either do things automatically or… not do things automatically, and leave the user to their own devices.  Distributed administration means that the state of the entire system is always in the foreground; if some part needs updating, it is the same whether that part is local or remote.

In the same vein, because the distributed application model that I'm proposing is about an application expanding to use remote resources, the applications are not split into separate client and server components that must be updated and managed separately.  A unified model means that local and remote pieces are understood as parts of the same whole, and that includes installation, deployment, and updates.  Between distributed OS updates and application updates, the entire environment that is being used - even though it is on multiple pieces of hardware - is all in scope at once, and all managed at once.

That, however, is all about distributed applications.  You could make a distributed application that manages existing operating systems. But MAD is more than just the app model, which raises the next question.

Why a Distributed Operating System?

When I originally created MAD, I took it as granted that we would need a new operating system to handle the management and operation of a distributed system.  Frankly, I merely assumed it; it's fair to wonder whether or not a distributed-first operating system is ultimately required.  Indeed, if you browse my blog, you will find a lot of concepts that seem to be unrelated distractions from the core concept.  And in some cases, that's probably exactly right; you'll never catch me saying that MAD is perfect.  Given the name, that'd be kind of silly.

The distributed operating system concept is not the distributed application model.  They are separate, and must be argued for separately.  And quite frankly, I can envision a future in which distributed applications are simply run on top of existing operating systems with little or no change to those operating system.  Frankly, managing distributed applications comes down to a software service; it doesn't require a clean slate.  I imagine that a lot of people, assuming they managed to figure out what I'm saying among all the stumbling explanations here on my blog, would make that point themselves.

There are two fundamental counter-arguments I'll make.

First, the distributed operating system is a collection of standards, and there should be a collection of standards that applies to distributed applications even if they are being run on top of existing operating systems.  To some extent, when I'm talking about the distributed operating system, I'm talking about the environment that exists inside of the distributed ecosystem itself, rather than the physical disk images on top of which the system itself operates.  It's important that applications and the systems onto which they are deployed, can agree on some terms, like what resources are available, what deployment constraints an application fragment needs fulfilled, how standard functions and hooks work, and so on.  It's astonishingly easy to imagine getting that wrong and creating two or more completely incompatible systems, or ones that will require serious shims and compatibility patches to be made to work later on.

The other argument to be made is that operating systems, even open-source and user-focused operating systems, have become bloated - not because of any technical failing on anyone's part, but because they were developed over time, and different components represent different ideologies regarding how a computer should be programmed.  One needs only familiarize themselves briefly with any one of the many arguments about Linux init systems to see that we are operating a complicated system on top of mixed metaphors.  I'm well aware that any OS I create or am involved in, will not be the final operating system, and assuming it survives, it will tend back towards the current state of chaos over time; thus, that itself can't be my argument.

But the various technologies I have placed generally under the OS category of Project MAD are all distillations.  For example the System API Directory and Application Folders frameworks, together unify files on disk with the abstract concept of data, as provided by drivers, services, and applications.  While I expect to have arguments about this framework with experts someday, the point (as I have made it in my last couple blog posts) is having a consistent metaphor that works for all data in a distributed system, from the smallest bits of application state, to LED light indicators exposed by drivers, to database sockets, to embedded applications and libraries.  One possible interpretation of the SAD/AF framework (not necessarily the one I'll push for) implies that any variable stored persistently in any part of any application is addressable (if not necessarily accessible, or even listed) via the directory, uniquely, so that it would be the same from anywhere in the system.  And at the same time, every compiled-in function would be not only addressable but callable from anywhere, complete with full type safety and crash handling.

It's not about actually doing that.  That would be a safety and security nightmare.  But if you have a data accessibility technology built into your distributed operating system that makes that level of access possible, it can handle anything less than that.  If you can go that far, then you can manage every application process on every node of a distributed system, you can bit-bang every GPIO bus on every piece of hardware, and you can read every last byte from every disk.  And critically, you can do it all from the filesystem, where everything accessible may also be listed publicly for you to peruse, so that you may better understand the system you're working on.

A single consistent metaphor that allows you to do everything that a distributed system can do - that's valuable, and it's a quantity that is not at all guaranteed in modern operating systems.  Building the right constraints on top to prevent abuse… I'm not going to say it's a secondary concern.  But I have some confidence that security experts won't have to strain themselves to find ways to block funny looking remote procedure calls, with suspicious senders and destinations.  If nothing else, IT departments are already wrestling with similar problems.

However, the SAD/AF framework only works for applications within the distributed OS's purview.  It is my concept that even a dedicated MAD OS image will have a number of services and drivers that are not accessible remotely under any circumstances, for security reasons - but if you simply have an ADA-compliant server on a random operating system image, almost none of the applications, services, or drivers of the host system will be exposed to the distributed system.  Rather than an intentional screening of things that do not deserve to be accessed remotely, you simply have the common and average fact of incompatibility.  And that's… fine.  It'll probably end up necessary, one way or another.

But if you want to administer systems, if you want to control all their hardware, then you want the operating system image to expose as much as possible to the network (for authorized and authenticated users, of course).  And there's a fairly obvious security hole - it's entirely plausible if you merely have a remote application service, which isn't well tied to the operating system, a remote application could start a local process which the service itself cannot regulate, or even detect, becoming an out-of-band attack vector.  And while you may be able to count on normal OS tools to discover and handle such an incident, and you can have distributed administration tools that handle those OS tools, the added complexity makes the situation more precarious.

Meanwhile, because of the way distributed systems work, many nodes in your network may be used for nothing more than hosting remote processes.  Modern OS images provide a large number of services and drivers that may not be needed or used; a compute-only network node could be trivially small, and have no exposed filesystem (or an immutable one) for remote applications to attack.  While it's possible to set up, eg Linux, systems to run at this rudimentary level, most systems are still expected to be general-purpose, and their OS images reflect that.  Systems set up for hosted compute only, especially if they have an out-of-band or self-update mechanism, may be all but untouchable to applications in a distributed system, in addition to being faster and simpler to operate.

The reason, of course, that most OS images today contain so many nuances is because the hardware they are running on top of can be so variable, and it's unclear how much of that hardware a given user might actually need.  And that brings me to my last question:

Why Modular Hardware?

When I first envisioned MOS/DCA, adding a processor to a system meant connecting little hardware devices together; the system was made of modules that would daisy-chain infinitely, and each module's hardware included absolutely nothing except its payload, and the backbone networking chip.  No general-purpose networking was involved here; it was dedicated pieces slotting together to make, effectively, a single desktop system out of hardware modules.

Hardware drivers under such a system, you understand, would be quite simple.  A CPU-only chip has no need for any drivers except the chipsets that support the CPU itself (including a fan and temperature sensor), and the backbone.  No generic networking, no printers or keyboards, no serial busses or parallel ports, no USB, no solid-state or spinning platter disk drives, no CD/DVD/Blu-ray drives.  A compute-only node not only need not do anything else, it could not.  No such accessory hardware existed.

In return for that simplicity, these modules were the very definition of commodity hardware.  I imagined they would be churned out by the millions.  In the modern world, I imagine them being built off of license-free technologies like RISC-V, driving the costs ever downwards; twenty years ago, I would have hoped for something similar, even if no such were commercially available at the time.  Nowadays, Raspberry Pi has proven you can deliver a full computer for $30, as long as you don't mind handling exposed pins and solder.

If you could make every computer you own work better with a single $30 purchase, would you really stop at one?  If plugging a raspberry pi into my parents' network was all it took to salvage my mother's failing laptop, she would have taken it in a heartbeat.  And the $30 raspberry pi comes with a graphics chip, HDMI out, and other accessories that a dedicated compute node may not need.  What really would be the ground floor, if you still want to end up with a unit that's good enough to make your distributed system better?

It goes beyond the compute only nodes.  Outside of general-purpose busses like USB, any task-specialized node only needs a few additional drivers.  Some will be complex, like a graphics unit with a GPU and multiple output ports, but if you only want to add, say, an AI accelerator chip, or a collection of data drives, the system image that runs that entire node has very few additions on top of the base layer; you will be certain that no other capability exists or is being used, because that's all that the system is.

If computer parts become that level of commodity hardware… well, I'll be honest, it will harm research and development of new chips.  That's a nice way of saying that chipmaking companies, which are already on rocky ground nowadays, could lose significant licensing revenue and sales, so long as people can get a cheaper alternative.  Some chipmakers, especially in the GPU space, are cagey about letting others understand how their technology works to preserve their competitive advantage - but commodity GPUs don't need to be individually powerful, if they can work together.  A no-nonsense alternative that's easier to program for and keep updated over the years, which gets to the same place with quantity instead of density, would certainly make that policy of stonewalling your own developers harder to maintain.

I'm trying hard not to paint those outcomes as purely good for the consumer - really, we've benefitted a lot from chip R&D over the years, in terms of increased capability and lower power consumption.  Without looking into it, I take it as given that there's graft and corruption at some levels in those companies big chip, and I don't care one way or the other.  Commodity hardware harms anyone who's trying to make high per-unit margins, whether they have good or bad reasons.  Likewise, commodity hardware comes with a massive fall in quality, inescapably.  My opinion, yours, theirs, or the average consumer's, all make no difference to market capitalism.

What holds us back from that possible future is the need to make and sell full, general-purpose computers, especially ones compatible with Windows.  As soon as China can make royalty-free, network-connected processors, that boost the power of your distributed system simply by existing, you'll be able to buy them in bulk.  If the drivers for those bulk processors are open source and can be maintained for decades to come, that will transform the computing space irrevocably.

Again - that's not an unqualified good, not if quality and R&D both nosedive.  But in our world where we already have oceans of unused hardware sitting around, perhaps it might be for the best that we start to make do with simpler, less wasteful technologies.

So Why MAD?

Everything I've talked about here is tied together by the ultimate goal of having software that expands into multiple pieces of hardware as needed.  If you can do that, you can use processors and memory and GPUs on other, even older, computers.  You can display the result on any random, network-connected display.  You can use any random piece of network-connected hardware with a keyboard as input.  You can access your home applications, not just your home files, from wherever you are.  Our existing models don't work to do all that.

The future in which they can do that is exciting.  The future in which you can control all the devices that you own, all the devices that you can reach, as though they were all in a single logical piece of hardware, that's exciting.  The idea of cheaper hardware, even if it is commodity, is exciting.  The idea of working or even playing like you're at home, whenever you have access to an internet connection, is exciting.

MAD will undoubtedly be wrong in some of its particulars.  It's probably got some fundamental flaws that we'll only discover in implementation, or on thorough review by experts.  And in the end, it's possible that MAD, being my concept alone for how distributed systems could work, will simply not be how things do work.  And I embrace all of that.

Because the future where we can do so much more, is worth a lot more than me being celebrated for writing some silly blog posts.

Beyond Unix: Application Folders

 Originally, as with the last blog post, I wanted this to be part of a longer post, but it seems wiser to separate posts by topic.  I… honestly wrote a long thing and then realized that it became even harder to understand than my average post, because it was just all over the place.  Part of the reason for this, as I realized after writing the last draft, is that this, like many parts of the MAD concept, can really stand on its own, and jamming concepts together, even if they would work together in MAD, is not a great way to treat your audience.

So for now, let's talk a little bit about MAD-style application folders.  In truth, I already started talking about this in my last post about Plan 9, towards the end, but it definitely needs some more detailed exposition.  But first, I'd better justify that post title.

Unix File Collections

I first got to play with a Unix-style OS when I went to college and installed a copy of Red Hat linux on my (one and only, tower) PC, circa 2003.  I got that copy of Linux from a CD provided in the back of a big book about the operating system.  To this day, Linux derivatives have been my only real point of contact for the Unix philosophy, and of course famously, Linux Is Not UniX.  Even so, many of their central mechanisms are (in my understanding) derived from the Unix fundamentals.

I have vivid memories of prowling through the /usr/bin directory in that first Linux box, trying to run every executable in order to figure out what it was and how to use it.  Naturally, I left this exercise disappointed, if still starry-eyed and naively believing that someday I would understand it all.  Today the consolidated bin directories on my (relatively fresh) Arch Linux box have over 5500 executables in them, and I would never imagine trying to understand them all.  There are too many applications that I didn't want or explicitly install registered in the GUI list, let alone the various helper commands and components that make them up or are tied to various system services, libraries, and …others.

I guess that sense of wonder and exploration has been beaten out of me, because it really is my instinct to want to understand and have control over my own system as best I can - what I can do with it, how to configure it, and how it works.  Unsurprisingly, MAD reflects that philosophy, and to wit: we're going to be talking about breaking up that /bin collection.

If it were put to me, I would divide programs up into six general categories: device drivers, services, applications, extensions, commands, and functions.  Of these, services are arguably background applications, and drivers are arguably services.  These three persist and wait loop on something, a user input, device event, or other external impetus.  Commands are an interface into another system (usually a service or driver); functions generally transform input into output, for as long as there is input to transform.  Extensions are applications, services, commands, or functions that are only meant to be consumed within a particular context (another service or application); there is no point to them being accessible outside of that context.  They may be embedded, or they may be separately installed, but they are useless without the parent application or service.

The point of making these distinctions, is that I consider it virtuous to collect all your system programs, but I do not consider it virtuous to keep them all in the same bucket.  If nothing else, commands and functions should be understood as part of a workflow (and extensions, within their context), while the rest should not be.  (If you have a workflow that ends up launching an application or starting a service, …well, you've basically made an extension, haven't you?  If you aren't extending the app, you're extending your shell.)  But rather than really being about the collection, this is about the system treating all of these executables the same, as nothing more than executables generally.

Among the reasons to draw a distinction: generally, commands and functions do not have their own application files, and they are less likely to have a configuration file; they may be nothing more than a single executable and any shared dependencies.  It makes sense to have a collection of these independent executables, because you will browse and search that collection in order to find commands to use in your workflow.  Likewise, it makes sense to have a collection of applications, a collections of services, and a collections of drivers.  And to a certain extent, Unix has all of these… just really not in a useful format.  It continues to grind my gears that there isn't simply a global folder in a standard location that just has all the system services and nothing else in it (and I don't mean under /etc).  Likewise, GUIs have to go out of their way to create collections of applications, and they get to decide on how they do that and where they keep it; it's not a central, standard directory.

Another problems with apps and services in Linux is that, by default, they are assumed to have a single, canonical configuration, plus (possibly) one additional configuration per user.  Anything more than that, and you get into the realm of passing in configuration files on the command line, or tweaking the environment before running an application.  Both of which work… but both are abusing more general mechanisms without actually addressing the problem.  This global-centric system has another consequence; network ports are a global and limited resource, and being numerical rather than name-based, reserving and targeting ports becomes problem when multiple copies of a service exist, or you want to run a service but hide its port, among other things.  These problems have solutions… but they all arise from old and faulty (or at least suboptimal) assumptions.

Over and over again when trying to write about MAD I come back to the idea of assumptions.  MAD/ADA, the distributed app model, is one tool to help you control the assumptions of a given application, but I'd like to talk about a different tool today: the MAD Application Folder.  The App Folder framework tends to be assumed to exist by ADA, but neither truly needs the other, as you'll see.

Application Folders For Fun and Profit

This topic is going to need a little technical overview before I get into the why's (and you know I love getting into the why's), simply because if I don't say this explicitly, I'm going to try to say in the middle of some other paragraph and it won't come out well.

The App Folder framework is fundamentally three pieces.  First, it is a standard for packaging applications for distribution, as a folder; but unlike many packaging schema, the application files are meant to stay together and installed just as they were received.  This folder format is intended to be eminently portable; the same format works whether the application comes bundled with its dependencies or whether they are separated for more efficient distribution.  Likewise, the same format works whether the application is installed system-wide, stored in some arbitrary filesystem folder (say a user Download folder), or embedded in another application.  We'll get to that.

Second, the framework is a standard for configuring applications, by editing, replacing, moving, adding, or deleting files within the application folder.  This configuration can be kept separately and applied at runtime, so that the application distribution folder is kept unmodified - we'll get to that, too.

Third, it is a standard for representing running applications and services as live folders (similar to the Unix procfs), with a specific emphasis on representing dependencies, imports, and exports as files in this live directory.

If it helps you make sense of what I'm trying to say, the application live folder is modeled directly off of the distribution folder, as modified by the configuration.  The line between these three concepts is straight and direct - once you understand the purpose of the live folders.

One main point of live folders is to keep most non-shell applications from ever directly accessing the global filesystem - their dependencies, exports and outputs, inputs and configuration, some internal program state, and their collection of shared, user, and system files, all are represented here.  If they would be going out to the filesystem to reach, for example, a hardware device, the application loader does it instead, mounting the file in the live folder, and the application looks for the hardware device to be mounted in that specific position.  The same is true of configuration files, application extensions like themes, and so on.  The application distribution file tells the app loader what files to collect; when finally the application is running, everything it needs should all be collected in one place (even if only in link form).

The other major part of the framework is meant to expose data in a way that the system can use.  Suppose for example you have a database which is nominally configured to use a network port.  In normal C code (and pardon me if I've misremembered or misunderstood), this is done as a two stage process, by opening a socket and then binding that socket to the network port, each with different system calls.  What the MAD application folders concept considers ideal in this situation, is for the application to open a socket and mount it in their live application folder, as for example Exports/database.sock.  Separately, a workflow that lives embedded in the application folder and managed by the application launcher (or core) gets triggered by the creation of this file.  Suppose that workflow lives in the distro folder as Exports/Port/3306.  Prior to the socket being created, the application live folder contains nothing in the Exports/Port folder; if the port binding succeeds, the same workflow creates that 3306 file in the live folder, making it point to the database.sock file, which is the real workhorse.  If you wanted to change the port number, in this example, you would only need to rename the 3306 file, and if you wanted to prevent the database from opening any port (we'll get back to that in a moment), you would only need to remove it entirely.

If you establish all of the above as the way things should work, then the system could maintain a collection of open network ports as files, each of which points to the appropriate application's live folder.  Incoming network requests on that port are forwarded to the database application's socket, using the filesystem to store the routing data.  That doesn't necessarily mean that the filesystem must be involved every time a packet crosses the network; it may suffice for “filesystem changed” notifications to cascade through the system, forcing updates in routing tables.  But this collection of ports as files would make for a very intuitive interface, allowing system administrators to use the filesystem itself to query what application is managing what port.

It goes beyond that, however.  Assume for the moment that the exported database.sock file can be used just like a network port, but without involving the networking stack.  Thus if you had, for example, a program that wanted to embed this database application, it could configure the embedded database to open its socket but not bind it to a port, and then your program uses the database's live folder to attach to the socket; the networking stack isn't even consulted, much less used as an intermediary.  That's clean and efficient - but what if I said that the same mechanism could let you connect to any other database in the same system without using its port number, even if the port is open, just by doing the same thing with the database service's live folder?  What if I said that the same mechanism could let you connect to a remote database by mounting a database client in the same place that you might have mounted the database server, in your database-consuming application's live folder?  In all three cases, your application will expect to find an application live folder at for example, Imports/Database, and in all three cases, it will specifically target Imports/Database/Exports/database.sock (though perhaps this could be simplified) as the socket it wishes to connect to.  Suddenly, your own application is much more portable, no matter whether it gets configured by the end user to use an embedded database, a system database, or a remote database.  More than that, any application living in that location which provides a compatible database.sock (or indeed, anything else you do to put a link to a database socket at that location) could be used by your application, even if it's not the exact vendor of database you were expecting (obviously, this is a little problematic for databases, which may have variances in their query language, but you get the general idea).

This one example shows the application live folder being used to export resources, import resources, configure those imports and exports, represent application state (eg, whether the socket/port has been opened), make the application embedable, automatically trigger workflows in response to events, provide a compatibility layer between competing service vendors, and make a collection of system resources.

But we aren't done.  Where does your database store its files?  Without googling, I was honestly unsure where in my system a, for example MySQL database file would actually reside, according to its default configuration (and frankly, MySQL's default location annoys me - not that it matters).  But Project MAD is about a distributed system, and there's a problem with storage in a distributed system: there isn't guaranteed to be a single canonical file store.  In fact, there may not be any general-purpose persistent storage anywhere in the system (though in that case you probably wouldn't be creating a database).

Categorically, in a MAD distributed system, a given application will have three datapoints to start from when looking for storage: the location of the application distribution files (or application configuration; if this is embedded, it will be another application's files, and you can go up the stack if need be), the private folder for the user running the application, and a generic system collection or index.  Under MAD, none of these are guaranteed to provide you with a writable storage volume (it's a reasonable guess, but not guaranteed).  Disk access is a resource in the same sense as any other device driver, though it may be hard to put some of the constraints into words.  MAD generally assumes that arbitrary system components can be removed from the system at any moment - and that includes both the long-term storage you might otherwise use to store the database files, and the module on which the application files themselves are stored.  It would be unlikely to be a problem in practice, but it is a burden I chose to undertake.

That means developing a language to describe what you want to store and why.  For example, a “System Service-persistent storage” disk space request might be different from an “Embedded Service-Application Data” disk space request.  In the former, you would generally ask the system whether there are any disks configured for storing service data first, and if not, fall back on asking the user or storing the database files with the database configuration or application files.  In the latter, you would generally want to ask the parent application where it's storing its data, and failing that, you would ask the user running the application where they want your data (more properly their data, assuming there is a user running your application), then try the application configuration location, and only last would you want to query the system for a global storage location, because your embedded database and the application files may go separate ways without warning.

Whichever way you choose a storage location, once one is selected, a data storage folder will be mounted in your application's (or the database's) live folder, and the application targets that data folder within its own process space with all subsequent filesystem requests.  And to reiterate: the application should not care where the files go.  A distributed system cannot expect applications to understand the system topology.  Your database should not try to learn where it should keep files, and the people maintaining the database code shouldn't need to decide.  It should not be the developer's job to find some clever, out-of-the-way place to put it, even by default.  It suffices that the database asks for a place to keep its files, and one is provided (assuming of course, that if you ask again tomorrow, you will get access to the same volume as you got today).  If the data needs to be encrypted, add that constraint to the request.  Everything else is up to the user and system; either they provide what you want, or they don't.  And if they don't, well then, it's time to get users involved.

Now, you as the user or system administrator?  You get to care.  You set up a file access server and client so that the database can use it.  But the tyranny of the default remains; it matters that the database can install itself without user intervention.  It must, if it can.

I'll mostly finish this part of the blog post for now, though there is one quick aside to make.  When categorizing executables, I said that commands, distinct from functions, depend on services or applications.  Services and applications have managed exports from their live app directory.  Thus it stands to reason, that if those commands are exported from an application or service live directory, then the system can create and manage a dynamic collection of commands, depending on what services are currently running.  Any services that are not running, need not contribute their commands to the collection.

Now, not everyone may find this dynamism valuable; some people would prefer to know what all the potential commands are, irrespective of whether services are running or stopped.  But the whole point of separating the different kinds of executables is managing scope.  Going back to my 5500-file /bin folder on my home machine, it would take some legitimate effort to sort that cluttered heap of random words into categories that help me, the user and administrator, know what I can do with my own damn machine.  Making that collection easier to manage is virtuous in itself.

Where Does This Leave Us?

So I've described application folders.  What about this is really going beyond Unix's philosophy?  If anything, we're depending more on the “everything is a file” philosophy, which you could argue is more of a back-to-your-roots movement.  Well, let's go back and look at what I've said so far.

Suppose we have an ideal Linux box or similar, based on the above.  Instead of having a mixed /bin directory, it is split into /cmd, /app, and /svc folders.  The command directory is a dynamic collection of functions and commands that have been installed separately, or exposed by currently running apps and services.  It does not contain the entry points for applications and services; those are split into their respective directories, but those entry points can still be collected automatically.

The application and service directories contain all applications and services installed at the system level, as opposed to installed in user storage.  It contains distribution folders for each of these applications, and under that distribution folder, you will also find a list of currently running instances of the application, if any, as live folders.  (Alternately, they could be neighbors, eg, AppFolder and AppFolder.ProcID, though that sounds sloppy to me.  Don't worry, MAD does this part differently, but we aren't talking about that for now.)

Given everything I've said, the /svc folder becomes a list of all system services that have been installed, and a list of all services currently running, and the configuration of those services, and each service provides a collection of commands that exist specifically to interact with that service, and each service may provide direct exports that you interact with, such as sockets, and each service points to any system resources that they are consuming, and each service can collect a list of any applications currently using this service's resources.

Thus for example, if the port management I talked about before was run by a service, its live folder would contain a collection of all currently opened ports on a given net interface, and would contain a link pointing to which net interface device it is regulating, among other bits of information.  Each of the opened ports in the collection would be a filesystem link to the live folder of the system service or application that has requested that port.  Because it links to a live folder, if you wanted to kill the process currently managing a specific port, you could very simply kill /svc/net.eth0/port/80, and the kill command would have no trouble getting its process ID implicitly.  And if you want to know who's using that socket or port, the app using the port should maintain a list of open connections in its live folder - because that is a portion of the application state that can and probably should be represented in the filesystem, once you've gotten all the rest of this up and running.

Oh… and as a nice side effect, you get a system collection of applications.  Almost seems like an afterthought now, doesn't it?  Even though distributed applications are the whole reason the entire system was invented…

Anyway, unless I grossly misunderstand the Unix specifications, all of this goes well beyond them, and it goes beyond Linux and other operating systems that I've heard of.  It still inherits some of the general attitude of Unix; I'm not looking to make it more like Windows, for example.  But it goes well beyond Unix, which is all that I claimed in the post title.

Okay now, take a deep breath.  Almost done with this one, I promise.

If the above were to be read by any serious veterans in the operating system design space, I suspect they would say among other things that app folders are all unnecessary and inefficient.  Even disregarding the, “What we already have, works” argument (which is kind of a lame argument, even when it is totally correct)… they still have a point.  Reconfiguring an entire Linux or other *nix based distribution to run on top of this service-collection-folder instead of non-filesystem-based services would be a massive undertaking, and there are absolute loads of questions that would need to be answered about how all the standards around these services should be set up: how app distro folders and live folders work, how the configuration mechanisms work, how drivers work - all of that before you start talking about what exports specific services should have, or whether and how the underlying architecture of the system should be reworked to use ports less and direct sockets more, plus more and more questions that bubble up as we move forward.  And trust me, everything I've said in this blog post - everything - is simplified, with pieces left off because I don't want to overload anyone, or trip over my own metaphorical tongue any more than necessary.

Honestly, you could multiply the whole collection of questions that are rattling around in your head by ten or a hundred and probably still not come close to all the various problems and questions involved in understanding and/or creating Project MAD.  The MAD App Folder framework is itself just a fraction of the Agentic Distributed App model, which itself is a fraction of the operating system, which was always meant to be attached to a whole hardware project that will probably never come to be - but that hardware project informs the software design and the operating system.  It is the nominal model, the stress test; it is important, not something I intend to just set aside or forget about.

When I say that Project MAD is a tangled mess of thoughts in my head, that is what I'm talking about.  It is “Take this problem that should probably take an entire community to sort out, multiply it by a hundred, cram it into my mind, and then try to reach in with chopsticks to untangle and remove it piece by piece” levels of tangled.  Again: the whole point of Application Folders was to help create distributed applications, but by golly gee whiz it sure would really make it easier to keep track of system services in a Linux box.  And again, this entire blog post is a trimmed down version of the application folders framework.  It's meant to be integrated into the directory and used alongside other concepts.

Weird how when you have a good idea, you start seeing uses for it all over the place, even though many of those “good ideas” won't be when you actually test them.  And yes - some parts of this won't be, and goodness knows I can't tell which from here.  But I think that it's a worthy idea that needs exploring, even if the MAD/AF framework is the only part of the system you port out.  Maybe it wouldn't be the best single piece, out of all the pieces that you could chop off and use separately… but you could.  And it would be interesting.

History of the MOS/ADA Programming Model

 This topic is tricky to introduce.  One of the central tenets to Project MAD as it is today, is a whole new model for applications, and I've said that it was the last things that came together, and it tied everything together in a neat package.  What you won't know is that my early concepts of the MOS/DCA project also included a modification to standard programming models, and that legacy is still very obvious once you know to look for it.

It's difficult to ferret out exactly what I meant by it if you're just reading my notes from back then, but it was there: I wanted to expose the internals of executables and shared libraries so that, say, a shell command could execute an arbitrary function.  Important to making that work is having an interactive shell that can capture typed data - more like an interpreted language's REPL loop than something familiar like Bash or Windows Command shell.  And this has to be a structured tool, in some fashion; in modern applications programming, you pass nothing to an application when it starts except strings, and the application has to figure out literally everything from that.

This ties back nicely to my complains in the last couple blog posts about programmers being obliged to reinvent the wheel.

At the time, what I had wasn't anything like a plan.  At most, I had a complaint.  And that sparked a wish.  After all, especially within the GNU toolset, how many applications are actually just wrappers around library functions?  Complex wrappers, sometimes; that complexity is frequently needed so that this one tool can interact with a complex programming library function in multiple ways, or so that common data pipelines (piping a list of files through grep, as an easy example) operate smoothly and the way you would expect them to.  As with a lot in programming, things that are simple to say are frequently hard to do, and if you want to actually do them, there is a lot of nuance and complexity that must be navigated.

The other side of that, though, is that the nuance and complexity spawns more nuance and complexity.  Your complex tool needs to be interacted with in complex ways.  Add a third layer on top, that assumes access to tools that use libraries; then add a fourth layer and more, tools on top of tools.  As long as nothing changes, eventually you get a level of tooling where you can issue a one-word command to do exactly what you mean to.

But every level in between the core library and the final level of tooling can shift.  Needs to shift.  Every layer must be updated whenever any of its dependencies update, for security reasons if nothing else, and sometimes bigger shifts are called for.  To a certain extent, there's no getting away from that - but if it were possible to have the library function that you actually care about, be executable in its own right without an application wrapping it, there can suddenly be whole levels of redundant tooling that are at least simplified, if not eliminated.

And of course there will probably be several other layers of tooling that crop up around this new concept; I won't lie.  Ultimately, we create tools because we have specific needs, and as long as the tools fulfill specific rather than general needs, they will diversify.  But many of those pain points that we are removing are redundant ones.  Every application needs to transform text input into data streams.  Applications that read from and write to data files have to pick or create libraries to that work with the specific protocols and data streams.  And if you're creating a workflow that involves data files being processed, you need individual applications whose only role is to be a part of that workflow, applications that parse files and put them in a form other applications can take as input.

Many of these niche applications, that only exist to take one data format and make it available, or take text input and render it back into a data file, are at once important to someone's workflow and possibly security, but also are tools nobody wants to maintain.  Nobody wants to need to maintain them.  It's important that someone creates a tool that parlays between data files and workflows, but it's a thankless job, and a product nobody wants to pay for.

In short, a shell that understands data objects and types, and can use that to call a function built into an application or library directly, instead of needing to build one application after another to handle the workflow… that would solve a lot of problems.

The steely-eyed veterans in the audience are saying, “Well, we have exactly that with interpreted languages like Python.  In Python, you can open a library and call a specific function out of it.  If you have a python file that is meant to be an executable, you can also load that as though it were a library, and call functions out of it, assuming it's designed right.”  And this is, of course correct.  (For the record, I had these thoughts before I was introduced to Python.  Probably.  Around that time, at the latest.  Again, my notes are not great.)

The difference is having that be an assumption built into the entire operating system, into the fundamental programming method, so that the entire system is built on one foundation instead of a shifting mix of dependencies from various legacies.  That is, of course, making a virtue of a vice: it means starting over and doing everything from scratch, which is... painful.  And these initial thoughts of mine were missing critical parts of the picture, which is why for many years I set them aside as irrelevant and a waste of time.  At the time, all I was thinking was that, doggone it, we shouldn't have to reinvent the wheel over and over.

And in truth… there are still today whole categories of errors that only exist because library functions are sealed away.  For example, languages like Python, PHP, Go, and many others can't easily use functions buried in libraries for other languages like C; people have to write new versions of old libraries, and each needs to be maintained separately, by different people, using different language constructs and different dependencies.  Ultimately, it becomes such a different problem to maintain each of them that even if one person was willing to maintain every copy of the same algorithm, it would become untenable for some fraction of them.

All this despite the fact that ultimately, all you want is to pass parameters in and get results out.  If there were no worry about dependencies or languages, if all you had to do was call the function and capture the return, it would be simple to do from any language.  Of course, from a certain point of view… that's always been all you have to do, if your language is willing to parse library headers or code from other languages.  But the library formats in use, Linux's .so and Windows' .dll, neither comes with a human- or machine-readable directory of functions meant to inform programmers of how to use them.  If you want to know how to use them, you need specific files intended for developers to use.  That means if you have an entirely different programming language and they want to use existing library functions, they need to read those files, …despite them being in another language.

In short, there's a reason why we are where we are.

When I talk about Project MAD as a change in the entirety of computing, it can sound like I'm vastly overestimating my own cleverness - and well, maybe.  But these are real problems.  My first concept almost sounds promising on its own: just call library functions.  But that concept does nothing about compatibility and discovery problems.  It hardly matters whether or not the code exists on your system if you don't know that it's there, where it is, what it does, how to call it, what side effects to expect, and what the return values and errors mean - and that assumes that errors get propagated correctly.  And the details need to be exact; one bit out of place changes everything.

Just like there's a reason why we got here, there's also a litany of reasons why we haven't leaped forward.  Even if we all agreed on one bright idea that we would work together on, it would be a massive effort with a lot of problems that need to be solved.  And if the bright idea that we all agreed on happens to have massive holes inherent to it, because it's not as bright an idea as we thought… that's a lot of work for nothing.

That's to say, I do understand, have always understood, and will always understand, that none of this is simple, that I can be wrong, and that ultimately, even if my idea was core to anything, it would be the people who actually achieve it that deserve the credit, because it will be very difficult.

But also…

But also, various subsystems that are a part of the MAD concept (here, specifically, the Modular OS and Agentic Distributed Application model) are there specifically to solve these problems.  I've described the system directory in this post on the Unified System API, but not all parts of the solution came around at the same time.  Part of it, as I've just described in this post, was there from near the beginning.  Many years later (five to ten, maybe) I started thinking about the directory, and specifically, having types understood system-wide and having them be a central part of the system directory.  Before this, all I really had was “Call a function inside a library, so we don't have to reinvent wheels.”  But it didn't make sense, quite.

I said in my first post that the ADA came about in the process of writing blog posts about the MOS/DCA concept.  I was trying to explain, and things still felt incomplete.  I had deduced that functions and hardware drivers needed to be sorted by function, which fed into the type hierarchy in the system directory, and I was also thinking about embedded libraries and applications, and how that made application deployment and dependency management neater.  I was writing about wrapping functions with scripts so that some logic was happening on remote nodes, but that wasn't working for me.

The reason why I like the ADA goes beyond the model itself.  The model slotted well into the weirdly shaped hole that the whole “software model” side of the project had.  Yes, you could call functions directly instead of only being able to run applications - but why?  What made that so compelling, and not merely one person ranting at the sky?  Sure, it would be nice to have a directory of all the functions and types within a system, but what made that so critical to the operation of an OS that it has to be a fundamental component?

But within the ADA, you may be calling functions within your own application agents and within others'.  You will be deploying agents to hardware modules specifically because there is software on that module that is required by that Agent.  And it goes beyond functions and code (or, well, it doesn't, but it helps to think of this way): the ADA is also about exposing data so that applications can be monitored and debugged, but that requires types to be in some sense objective, even if that only means a type connected to the application and not a system-wide one.

Equally, within ADA, an Agent exists to handle the tricky problems of accountability.  There is an entirely reasonable question when you try to, for example, perform GUI API calls remotely.  The memory being used by your GUI application can't leave the display controller, or at the very least, there will be a copy of the output in the display controller, which must be managed by the GUI stack itself--while being owned by an external party.  Without an agent such as the ADA's, there are massive, massive questions to be answered about how this is not an absolutely terrible idea.  But if you can't do things like call GUI functions remotely, then... what even is the plan?

But perhaps most importantly, in the ADA, deploying Agents and routing messages is a system function, one attached to the ADA servers and not one implemented by applications themselves.  That means that knowledge of functions, data, and types needs to be standardized at a system level.  If the ADA model were implemented separately by each application, then each application would need to separately track and manage type and API version compatibility.  Implementing this as a system function makes things not only possible, but easy for programmers to handle, by doing the hard work ahead of time, in the operating system.

I like to, perhaps arrogantly, think that I am touching on fundamental truths with some of these analyses.  It's why I keep hammering, for example, on don't force every programmer to reinvent the wheel.  It is hubris to say I alone know best (I've demurred enough that you know I'm not that hubristic), but it's still satisfying to line up a whole bunch of problems in a row, pointing at them, and say, Example one, example two, example three.  Tying things up with a bow on top is satisfying, and the point of feeling genuinely excited about the ADA and Project MAD in general is that a stack of problems are placed in a box and wrapped neatly up.

It also feels good because this isn't all the work of a day, month, or year.  While this talk of modifying programming models may not have happened in the first year of the project (the first part was mostly the DCA, honestly), it's at least 15 years in the making.  Tricky problems that have tickled the back of my mind for years are coming together.  These are problems that weren't just tricky in practice.  They were tricky conceptually; it wasn't clear that there even was an answer.  It wasn't hard to imagine that I was wasting my time, because there was no clear vision of the end.

There's still a lot rough, and there's still lots of room for me to be wrong.  But it's astonishing how the last few pieces have made the puzzle come together.  At least from where I stand, it looks promising, like there's really an answer.  Not an easy one, but one that leaves us all much better off.

And even if I'm wrong, perhaps getting more eyes on the collection of problems I've been working through will help.