Beyond Unix: The System Directory

 I have more rewrites of this post than usual, which tends to mean that I'm trying to say too many things and they're all jamming together and making things more complicated than I wanted them to be.  It's why the last couple blog posts had to get split out instead of being unified into a larger ongoing discussion.  It's incredibly frustrating, because I feel like I have a very simple thing that I want to explain, and thousands of words later I realize it's been buried in a pile of other things.

Of course, for me, “very simple” tends to mean that I want to talk about a combination of systems each with moving parts that just so happen to mesh nicely together.  So, you know, it's not necessarily anything that's actually simple.  Writing this blog has been… tricky.

The point is, I've been trying to say something about the Unix “everything is a file” philosophy, and how MAD is going further in that direction than Unix and all its derivatives, which is itself such a hubristic and complicated statement that there are very real chances of merely coming off like an asshole instead of a smart asshole, which is at least closer to the goal.  But then I go and wreck that already complicated thing by discussing other complicated things and it all just gets confused.

So let me try this again, from yet another angle that kind of feels sideways to me.  But fair warning, I'm not entirely sure I succeed.

Everything is a (File)System

The true point and the true nature of Unix's “everything is a file” philosophy isn't actually about files - and that's good, because files kind of suck.  In my blog post about how the Internet Protocol makes for a lousy RPC mechanism, I pointed out that its main disadvantage is how it depends on typeless data streams and has no real plan for anything beyond that.  If you want to build on top of a typeless data streams, as a system conceit, you are expecting programmers to invent and reinvent methods of typechecking, conversion, validation, verification, bounds checking, error detection, error correction, and error handling - not to mention adding any self-documentation mechanisms they might wish to have, so that people can query the options available or understand why errors happened and what they mean.  It's not a terrible place to start, when designing literally the first operating systems humanity has ever had, but it's… incomplete.  Primitive.  And I want to argue, it's fundamentally insufficient, something that shouldn't make its way unchanged into a more industrial operating system, one that deserves to be used across the globe for decades or centuries to come.

That complaint applies to IP, and it applies to files and IPC data pipes, and it applies to program entry points, all of which are forced to work on top of highly generic data streams.  So it's good that the “file” part of “everything is a file” isn't really the point.  I know it seems like it is; it's between 20-25% of the phrase depending on how you count it.  But no, it's about the system, the underlying metaphor that goes beyond merely organizing the system and starts to feel like a comfortable home for users to play around in.

The filesystem was Unix's answer to this problem - but that philosophy has not really been embraced by a lot of its descendants.  Most of the people and projects who came in after Unix or Linux was already a thing, simply created the best mechanism they could think of for their own particular problem and ran with it, Unix philosophy be damned.  The windowing system, for example; I can understand them not thinking, especially at first, that there's any reason to involve the filesystem for concepts like desktops, app windows, widgets, and so on.  You'd need a compelling reason for them to build that feature into their system, and “it's the philosophy” isn't enough.

The consequences of a system not being cohesive only come out after many years of people depending on it.  A whole ecosystem of tools springs up that exist to fill in the gaps left behind by other tools, to make it more convenient to depend on third-party tools and systems, that take the edge off of two systems grinding together, and so on.  Eventually the original project lays blanketed under a layer of tools and patches that completely obscure what was originally there.  The result works, and it can be made to work well, but only when you plan around the complications, the rough edges and incompatibilities.  Any time that things aren't going to plan, suddenly the system that looked so good from afar can become a hellscape of gears refusing to mesh, wires left unconnected, and pipes spilling everywhere.

Preventing complex systems from becoming a disaster is a topic other people understand better than I - but without question it's a matter of technical leadership.  Some companies want to be tyrants that force everything to work a certain way, and that can work - so long as people think your product is worth the trouble, and the more trouble you are, the more likely they'll leave someday.  Others like the open source community depend on standards bodies, community cohesion, and the willingness of people to take on difficult tasks for the sake of others - and again, the more difficult the task, the less willing people will be.  Either way, the ideal result is some plan or system that handles any discrepancy, either forcing or guiding people to do the right thing.

It's foolish to pretend that you'll ever reach that ideal, but it's still what an engineer should aim for, at least one in my exact position.  It is, to paraphrase, kind of the job.  It will be other people's job to invent the best piece that fits a certain slot or task - it's my job to design an overall system that they'll want to be part of, foresee complications and make sure that there are tolerances built in to handle mistakes and idiosyncrasies.  And to a certain extent… the philosophy of the open source movement, which is totally okay with components bolted onto the side of a machine and wires hanging loose, that's a system that's nice to be a part of.  It's nice to know you can tinker, nice to know that that's kind of the plan, even if it's a messy one.

When I say, as I've said many times, that maybe MAD won't win but I still think I'm on to something… I'm saying is that I know I'm not a genius system designer.  Creating something clever is one thing - making something that everyone will agree with is another.  It's disagreements that lead to pieces bolted on the sides, people going their own way instead of following along - or worse, they lead to them never joining in the first place, projects never getting off the ground.  Agreement is the heart of technical leadership - agreeing that things are right, because they're right, and because the process isn't too big of a hassle to manage.

I think if people saw the problems the way I do, we would at least all be moving in the general direction I'm pointing.  And if I'm wrong… well, I won't find that out without at least explaining the system for others to react to.  And that, ultimately, is why we're here.

A Universe in your Pocket: The SAD

This blog post is part of the “Beyond Unix” set and I've been talking about the filesystem, so it shouldn't be much of a surprise that I'm here to talk about its replacement in the MAD paradigm.  The “System API Directory” or SAD is defines the MAD operating system, and it's important to understand that it is a protocol, not a place.  (I've explained the API part of it before and I'll revisit it again soon, but this blog post is already long enough as-is.)  Under Unix, the root filesystem is in fact stored on disk, meaning that if they wanted and had the authority, a user could simply do whatever they wanted with it.  User here, being distinct as a concept from OS designer, or from anyone who knows what they're doing.  Any random person with the root password can break the system just by renaming folders in the root directory; indeed, if you want to prevent that, you have to add features, not remove them, because under Unix the root directory is a place, not a protocol.

There is a good reason why the root of the MAD/SAD is not a literal filesystem directory - the OS is designed to be distributed among many computers, and it's incorrect to try to say that any one of the computers is the “root” or “home” for the whole system.  In a distributed system there are many “roots”; the term becomes ambiguous.  The root filesystem operator in principle is simply a command to start your filesystem query in one specific place, and under MAD, the correct place to start a generic query is in an abstract space that exposes important concepts as quickly as possible.  (There are some more specific queries you will want to start in different places, but that's for another time.)

Part of the point of a filesystem, part of what makes it valuable, is that it is in some ways self-documenting.  Listing a directory tells you what's in it; the names of files and folders tell you something about them.  A file path has meaning, especially starting from the root; you know something about the file you're trying to reach from where it is, not just from its own name is.  More to the point, you know where something should belong by knowing what it is, and who it belong to.  As much as everyday users brush up against these problems when organizing their documents and/or desktop, the stakes are much higher for anyone whose work will be used by others.

In a way, this philosophy comes directly from Unix; the /etc directory doesn't simply give you a predictable place to put configuration files, it also makes the filepath for system configuration files as short as possible, which is as much about semantics as string length.  You don't have to go looking for system configuration at the tail end of some involved filesystem tree; it is one of the most important things you may go looking for, so it is indexed near the root.  (If you've ever had to go looking for some specific configuration file buried under /usr/share/someThirdPartyLibrary/config, or had to edit the C:\Windows\System32\drivers\etc\hosts file, you understand.  Even if you can understand the filepath once you see it, you might not be able to guess it, and quite frankly there are frequently multiple “correct” place it might be.  All of this obscurity can sometimes make a necessary task harder.)

Under MAD the root directory contains dynamic collections of concepts deemed important; a collection of actual hardware modules, for one, and a collection of users for another, though there are more.  All those modules and all those users are themselves collections of other important things, and many of those collections are themselves full of other collections.  Importantly, as you go down these trees, you will frequently be changing who exactly you are making filesystem requests to.  Indeed, a large part of the root filesystem is about pointing you towards whomever you need to ask the rest of your question - the more of that can be cached, the fewer questions need to be asked and fewer answers awaited with bated breath… but that's not always practical.  The root protocol handler can be done locally (must be, because the system is decentralized), and the top-level contents of those modules might be cached, but if you want to get a list of what applications a user has running, for example, that will be something you have to ask, as it may change moment to moment.  

This is one of many parts of a distributed system that is less optimized than its counterpart in a monolithic system, but it's not possible for a distributed system to have a central authority that can answer all your questions at once, if for no other reason than scalability.  A distributed system has a theoretically infinite maximum scope, insofar as the back-end allows; a large enough system could have problems simply indexing the hardware (in terms of memory and computing requirements, if nothing else), to say nothing of the many devices and software endpoints each one may export.  In an infinitely large system, it's important that the scope narrows down to only what you need to know; for an application, that means narrowing the entire universe down to only the machines it is running on, and the machine its authorizing user is running on.  (Well, unless I've missed something, plus there's at least one asterisk not worth going into now)

In my last two posts I talked about the ADA distributed application model, and application folders; for our purposes here, these concepts are expanding on the same fundamental data abstraction.  Under the ADA, when you want to access a remote resource (including files), your application sends an Agent to the piece of hardware where that resource resides, and then the application Agent acts as a proxy for access to that resource.  This model fundamentally means that applications, once deployed, can focus only on the machines they are already on instead of understanding the system as a whole; even if they accept client connections, those clients are obliged to be present locally, and so aren't remote from the server's perspective.  Application Folders push this chain of custody into the system directory; the exposure makes the relationships easier for developers and users to understand and audit.

A simple example.  Suppose there are two hardware modules; your application lives on one, /module/somemod.123/apps/myApp, and it wants to access a file on another module, /module/othermod.234/files/data.json.  Your application will deploy a file-access agent to module 234; this deployment process also involves the filesystem, so your agent will have some canonical, physical url, eg /modules/othermod.234/agents/myApp/fileAccess; the agent will also be mounted under your app, as /module/somemod.123/apps/myApp/agents/fileAccess.  This agent negotiates with the file server on module 234, and if it is allowed to open the file, it exposes that file to the rest of your application as /modules/othermod.234/agents/myApp/fileAccess/data.json.  And because the agent is mounted under your application, the same file is also mounted as ./agents/fileAccess/data.json.

This alone may seem like a comfortable level of abstraction - but what if you don't necessarily know what agent will be retrieving that data file, or what the agent will be named?  You may have, for instance, may instances of the fileAccess agent, each of which is present on a different module, each exposing some other file; some of those other files may even be titled data.json on the native filesystem.  Yet another layer of abstraction wouldn't be hard; simply mount the same data file one more time at, for example ./imports/data.json or ./imports/renamedDataFile.json, depending on exactly how fancy you want to get.  This last file mount contains enough information in the filesystem itself that you can reach the actual file, but the url that you actually use to access it (as stored in and exposed by the filesystem) becomes descriptive.

Is that important?  Well maybe.

Consider if you are debugging an application.  It opens a file, and the file doesn't contain what they expect, perhaps because the write process got corrupted by another instance or another application.  The actual file that you are reading has the canonical url /module/othermod.234/files/data.json, but that url may not explain its role within your application even a little bit.  If it's not a default part of your application, it's probably being opened due to a configuration or user input; you may only understand the file's purpose when you find that particular line of configuration or where in the code the input occurred.

The source of the file doesn't change its role in your program - it's just that with existing programming models, the role of a file is often stored entirely in variable names, which at best is data stored in debug information.  The application folders framework gives us the option of having that naming convention persist when the application is running; you don't always want that, for some secure applications, but it's a useful abstraction for anything you may want to understand and reconfigure.  Choosing to cleverly rename the file gives you information that's available at a glance.  If it's named ./imports/previousWindowPosition.json or ./imports/customIcons.json, for example, you'll know what's supposed to be in that file in a heartbeat, even if it turns up empty or malformed.

After all of this fussing about the specifics, it almost feels weird to come back around to where this started: making a theoretically infinite system shrink down until all you need to focus on is the application itself.  But the truth is that most applications have fixed needs that are determined by their configuration.  Once those needs are filled, it only needs to look inwards, and the infinite system beyond its borders is no longer relevant.  If that raises the question about how you fill those needs, I invite you to go back to the post on Plan 9 and resource distribution - but the short answer is that you use configured defaults where set, and then use a list of constraints to narrow down the infinite system to only those modules which might be useful, and pick arbitrarily among them, or have the user pick, whichever is appropriate.

Constraining the list of modules by their capabilities, is part of the reason why the list of modules is a system root directory, and why capabilities are root directories of each module.  The root protocol handler for the SAD gets handled locally, which means that each client must get enough information, and quickly, to build a list of what modules provide what capabilities.  Much of this can be cached, and may be provided at intervals to ensure everyone is on the same page - but the point is that the SAD plants a flag saying that these details are important and central to the model, thus they are available immediately.  If you were in an infinite system, at least each query you made to a nearby machine would be short and over with quickly, all while still using a directory structure that users themselves can investigate, explore, and audit, in order to understand how their own machines work.

Having the directory be browseable is part of what makes it self-documenting, part of what makes the whole system come together, and part of what makes users feel like it is under their control, able to be made use of and modified to better suit their needs.  And exposing those pieces of a given module that are important, according to the system, also helps users and developers understand.  As you come to understand how adding capabilities to your system makes it more powerful, you can come to understand which capabilities you want to add, and if you are a business, what capabilities you can make money selling to people.

Keep in mind that under MAD, that's a large part of the point - the hardware gestalt gets stronger when you add to it, which means that businesses are incentivized to provide you the hardware capabilities you want added to your gestalt in a useful package, cheaply.  MAD's hardware schema assumes people will simply package processors that exist for no other purpose than to be attached to a generic system - it's the reason why we can't assume anything exists on a given machine, unless the machine tells us that it does.  These systems will be cheap compared to full PCs - they will have fewer support chips, no USB, no HDMI, and no SATA; there will be much less to license, especially if open projects like RISC-V gain traction.  But open projects depend on support, and support depends on popularity; absent that popularity, it's a gamble to get behind new technologies, open or proprietary.  The ability to add capabilities to other machines for dirt cheap sounds like a pretty good reason to gamble on an open technology, though, doesn't it?

I admit I haven't gone into the hardware side of MAD pretty much at all, and maybe I'll get to that soon.  My Introduction and Index post has a very sad, empty space where posts on that topic should be.

But there's at least one more topic in this “Beyond Unix” chain, one that this post was supposed to be on, but that conversation will be easier when I can just reference this blog post to say that the point of “everything is a file” is about the system.  Because we'll be changing that system, for the better - and more than anything else I've talked about it, that one's going to be a doozy.  Though, again… I've tried to say it all before.  It's already out there... if you can decode my ramblings.  I'm hoping this string of blog posts will be much more comprehensible, and as such, I'll try to keep a thread of logic going from post to post.

These posts may end up being redundant - but if they help anyone understand, then it's worth it.  At least, once people understand what I have and what I'm doing, then we can start talking about the actual concepts, debating them on their technical merits.  Until I can explain them well… there aren't any technical merits to debate, as far as anyone else is concerned.

Which is… kind of a lonely place to be, but that's just life.

Beyond Unix: Application Folders

 Originally, as with the last blog post, I wanted this to be part of a longer post, but it seems wiser to separate posts by topic.  I… honestly wrote a long thing and then realized that it became even harder to understand than my average post, because it was just all over the place.  Part of the reason for this, as I realized after writing the last draft, is that this, like many parts of the MAD concept, can really stand on its own, and jamming concepts together, even if they would work together in MAD, is not a great way to treat your audience.

So for now, let's talk a little bit about MAD-style application folders.  In truth, I already started talking about this in my last post about Plan 9, towards the end, but it definitely needs some more detailed exposition.  But first, I'd better justify that post title.

Unix File Collections

I first got to play with a Unix-style OS when I went to college and installed a copy of Red Hat linux on my (one and only, tower) PC, circa 2003.  I got that copy of Linux from a CD provided in the back of a big book about the operating system.  To this day, Linux derivatives have been my only real point of contact for the Unix philosophy, and of course famously, Linux Is Not UniX.  Even so, many of their central mechanisms are (in my understanding) derived from the Unix fundamentals.

I have vivid memories of prowling through the /usr/bin directory in that first Linux box, trying to run every executable in order to figure out what it was and how to use it.  Naturally, I left this exercise disappointed, if still starry-eyed and naively believing that someday I would understand it all.  Today the consolidated bin directories on my (relatively fresh) Arch Linux box have over 5500 executables in them, and I would never imagine trying to understand them all.  There are too many applications that I didn't want or explicitly install registered in the GUI list, let alone the various helper commands and components that make them up or are tied to various system services, libraries, and …others.

I guess that sense of wonder and exploration has been beaten out of me, because it really is my instinct to want to understand and have control over my own system as best I can - what I can do with it, how to configure it, and how it works.  Unsurprisingly, MAD reflects that philosophy, and to wit: we're going to be talking about breaking up that /bin collection.

If it were put to me, I would divide programs up into six general categories: device drivers, services, applications, extensions, commands, and functions.  Of these, services are arguably background applications, and drivers are arguably services.  These three persist and wait loop on something, a user input, device event, or other external impetus.  Commands are an interface into another system (usually a service or driver); functions generally transform input into output, for as long as there is input to transform.  Extensions are applications, services, commands, or functions that are only meant to be consumed within a particular context (another service or application); there is no point to them being accessible outside of that context.  They may be embedded, or they may be separately installed, but they are useless without the parent application or service.

The point of making these distinctions, is that I consider it virtuous to collect all your system programs, but I do not consider it virtuous to keep them all in the same bucket.  If nothing else, commands and functions should be understood as part of a workflow (and extensions, within their context), while the rest should not be.  (If you have a workflow that ends up launching an application or starting a service, …well, you've basically made an extension, haven't you?  If you aren't extending the app, you're extending your shell.)  But rather than really being about the collection, this is about the system treating all of these executables the same, as nothing more than executables generally.

Among the reasons to draw a distinction: generally, commands and functions do not have their own application files, and they are less likely to have a configuration file; they may be nothing more than a single executable and any shared dependencies.  It makes sense to have a collection of these independent executables, because you will browse and search that collection in order to find commands to use in your workflow.  Likewise, it makes sense to have a collection of applications, a collections of services, and a collections of drivers.  And to a certain extent, Unix has all of these… just really not in a useful format.  It continues to grind my gears that there isn't simply a global folder in a standard location that just has all the system services and nothing else in it (and I don't mean under /etc).  Likewise, GUIs have to go out of their way to create collections of applications, and they get to decide on how they do that and where they keep it; it's not a central, standard directory.

Another problems with apps and services in Linux is that, by default, they are assumed to have a single, canonical configuration, plus (possibly) one additional configuration per user.  Anything more than that, and you get into the realm of passing in configuration files on the command line, or tweaking the environment before running an application.  Both of which work… but both are abusing more general mechanisms without actually addressing the problem.  This global-centric system has another consequence; network ports are a global and limited resource, and being numerical rather than name-based, reserving and targeting ports becomes problem when multiple copies of a service exist, or you want to run a service but hide its port, among other things.  These problems have solutions… but they all arise from old and faulty (or at least suboptimal) assumptions.

Over and over again when trying to write about MAD I come back to the idea of assumptions.  MAD/ADA, the distributed app model, is one tool to help you control the assumptions of a given application, but I'd like to talk about a different tool today: the MAD Application Folder.  The App Folder framework tends to be assumed to exist by ADA, but neither truly needs the other, as you'll see.

Application Folders For Fun and Profit

This topic is going to need a little technical overview before I get into the why's (and you know I love getting into the why's), simply because if I don't say this explicitly, I'm going to try to say in the middle of some other paragraph and it won't come out well.

The App Folder framework is fundamentally three pieces.  First, it is a standard for packaging applications for distribution, as a folder; but unlike many packaging schema, the application files are meant to stay together and installed just as they were received.  This folder format is intended to be eminently portable; the same format works whether the application comes bundled with its dependencies or whether they are separated for more efficient distribution.  Likewise, the same format works whether the application is installed system-wide, stored in some arbitrary filesystem folder (say a user Download folder), or embedded in another application.  We'll get to that.

Second, the framework is a standard for configuring applications, by editing, replacing, moving, adding, or deleting files within the application folder.  This configuration can be kept separately and applied at runtime, so that the application distribution folder is kept unmodified - we'll get to that, too.

Third, it is a standard for representing running applications and services as live folders (similar to the Unix procfs), with a specific emphasis on representing dependencies, imports, and exports as files in this live directory.

If it helps you make sense of what I'm trying to say, the application live folder is modeled directly off of the distribution folder, as modified by the configuration.  The line between these three concepts is straight and direct - once you understand the purpose of the live folders.

One main point of live folders is to keep most non-shell applications from ever directly accessing the global filesystem - their dependencies, exports and outputs, inputs and configuration, some internal program state, and their collection of shared, user, and system files, all are represented here.  If they would be going out to the filesystem to reach, for example, a hardware device, the application loader does it instead, mounting the file in the live folder, and the application looks for the hardware device to be mounted in that specific position.  The same is true of configuration files, application extensions like themes, and so on.  The application distribution file tells the app loader what files to collect; when finally the application is running, everything it needs should all be collected in one place (even if only in link form).

The other major part of the framework is meant to expose data in a way that the system can use.  Suppose for example you have a database which is nominally configured to use a network port.  In normal C code (and pardon me if I've misremembered or misunderstood), this is done as a two stage process, by opening a socket and then binding that socket to the network port, each with different system calls.  What the MAD application folders concept considers ideal in this situation, is for the application to open a socket and mount it in their live application folder, as for example Exports/database.sock.  Separately, a workflow that lives embedded in the application folder and managed by the application launcher (or core) gets triggered by the creation of this file.  Suppose that workflow lives in the distro folder as Exports/Port/3306.  Prior to the socket being created, the application live folder contains nothing in the Exports/Port folder; if the port binding succeeds, the same workflow creates that 3306 file in the live folder, making it point to the database.sock file, which is the real workhorse.  If you wanted to change the port number, in this example, you would only need to rename the 3306 file, and if you wanted to prevent the database from opening any port (we'll get back to that in a moment), you would only need to remove it entirely.

If you establish all of the above as the way things should work, then the system could maintain a collection of open network ports as files, each of which points to the appropriate application's live folder.  Incoming network requests on that port are forwarded to the database application's socket, using the filesystem to store the routing data.  That doesn't necessarily mean that the filesystem must be involved every time a packet crosses the network; it may suffice for “filesystem changed” notifications to cascade through the system, forcing updates in routing tables.  But this collection of ports as files would make for a very intuitive interface, allowing system administrators to use the filesystem itself to query what application is managing what port.

It goes beyond that, however.  Assume for the moment that the exported database.sock file can be used just like a network port, but without involving the networking stack.  Thus if you had, for example, a program that wanted to embed this database application, it could configure the embedded database to open its socket but not bind it to a port, and then your program uses the database's live folder to attach to the socket; the networking stack isn't even consulted, much less used as an intermediary.  That's clean and efficient - but what if I said that the same mechanism could let you connect to any other database in the same system without using its port number, even if the port is open, just by doing the same thing with the database service's live folder?  What if I said that the same mechanism could let you connect to a remote database by mounting a database client in the same place that you might have mounted the database server, in your database-consuming application's live folder?  In all three cases, your application will expect to find an application live folder at for example, Imports/Database, and in all three cases, it will specifically target Imports/Database/Exports/database.sock (though perhaps this could be simplified) as the socket it wishes to connect to.  Suddenly, your own application is much more portable, no matter whether it gets configured by the end user to use an embedded database, a system database, or a remote database.  More than that, any application living in that location which provides a compatible database.sock (or indeed, anything else you do to put a link to a database socket at that location) could be used by your application, even if it's not the exact vendor of database you were expecting (obviously, this is a little problematic for databases, which may have variances in their query language, but you get the general idea).

This one example shows the application live folder being used to export resources, import resources, configure those imports and exports, represent application state (eg, whether the socket/port has been opened), make the application embedable, automatically trigger workflows in response to events, provide a compatibility layer between competing service vendors, and make a collection of system resources.

But we aren't done.  Where does your database store its files?  Without googling, I was honestly unsure where in my system a, for example MySQL database file would actually reside, according to its default configuration (and frankly, MySQL's default location annoys me - not that it matters).  But Project MAD is about a distributed system, and there's a problem with storage in a distributed system: there isn't guaranteed to be a single canonical file store.  In fact, there may not be any general-purpose persistent storage anywhere in the system (though in that case you probably wouldn't be creating a database).

Categorically, in a MAD distributed system, a given application will have three datapoints to start from when looking for storage: the location of the application distribution files (or application configuration; if this is embedded, it will be another application's files, and you can go up the stack if need be), the private folder for the user running the application, and a generic system collection or index.  Under MAD, none of these are guaranteed to provide you with a writable storage volume (it's a reasonable guess, but not guaranteed).  Disk access is a resource in the same sense as any other device driver, though it may be hard to put some of the constraints into words.  MAD generally assumes that arbitrary system components can be removed from the system at any moment - and that includes both the long-term storage you might otherwise use to store the database files, and the module on which the application files themselves are stored.  It would be unlikely to be a problem in practice, but it is a burden I chose to undertake.

That means developing a language to describe what you want to store and why.  For example, a “System Service-persistent storage” disk space request might be different from an “Embedded Service-Application Data” disk space request.  In the former, you would generally ask the system whether there are any disks configured for storing service data first, and if not, fall back on asking the user or storing the database files with the database configuration or application files.  In the latter, you would generally want to ask the parent application where it's storing its data, and failing that, you would ask the user running the application where they want your data (more properly their data, assuming there is a user running your application), then try the application configuration location, and only last would you want to query the system for a global storage location, because your embedded database and the application files may go separate ways without warning.

Whichever way you choose a storage location, once one is selected, a data storage folder will be mounted in your application's (or the database's) live folder, and the application targets that data folder within its own process space with all subsequent filesystem requests.  And to reiterate: the application should not care where the files go.  A distributed system cannot expect applications to understand the system topology.  Your database should not try to learn where it should keep files, and the people maintaining the database code shouldn't need to decide.  It should not be the developer's job to find some clever, out-of-the-way place to put it, even by default.  It suffices that the database asks for a place to keep its files, and one is provided (assuming of course, that if you ask again tomorrow, you will get access to the same volume as you got today).  If the data needs to be encrypted, add that constraint to the request.  Everything else is up to the user and system; either they provide what you want, or they don't.  And if they don't, well then, it's time to get users involved.

Now, you as the user or system administrator?  You get to care.  You set up a file access server and client so that the database can use it.  But the tyranny of the default remains; it matters that the database can install itself without user intervention.  It must, if it can.

I'll mostly finish this part of the blog post for now, though there is one quick aside to make.  When categorizing executables, I said that commands, distinct from functions, depend on services or applications.  Services and applications have managed exports from their live app directory.  Thus it stands to reason, that if those commands are exported from an application or service live directory, then the system can create and manage a dynamic collection of commands, depending on what services are currently running.  Any services that are not running, need not contribute their commands to the collection.

Now, not everyone may find this dynamism valuable; some people would prefer to know what all the potential commands are, irrespective of whether services are running or stopped.  But the whole point of separating the different kinds of executables is managing scope.  Going back to my 5500-file /bin folder on my home machine, it would take some legitimate effort to sort that cluttered heap of random words into categories that help me, the user and administrator, know what I can do with my own damn machine.  Making that collection easier to manage is virtuous in itself.

Where Does This Leave Us?

So I've described application folders.  What about this is really going beyond Unix's philosophy?  If anything, we're depending more on the “everything is a file” philosophy, which you could argue is more of a back-to-your-roots movement.  Well, let's go back and look at what I've said so far.

Suppose we have an ideal Linux box or similar, based on the above.  Instead of having a mixed /bin directory, it is split into /cmd, /app, and /svc folders.  The command directory is a dynamic collection of functions and commands that have been installed separately, or exposed by currently running apps and services.  It does not contain the entry points for applications and services; those are split into their respective directories, but those entry points can still be collected automatically.

The application and service directories contain all applications and services installed at the system level, as opposed to installed in user storage.  It contains distribution folders for each of these applications, and under that distribution folder, you will also find a list of currently running instances of the application, if any, as live folders.  (Alternately, they could be neighbors, eg, AppFolder and AppFolder.ProcID, though that sounds sloppy to me.  Don't worry, MAD does this part differently, but we aren't talking about that for now.)

Given everything I've said, the /svc folder becomes a list of all system services that have been installed, and a list of all services currently running, and the configuration of those services, and each service provides a collection of commands that exist specifically to interact with that service, and each service may provide direct exports that you interact with, such as sockets, and each service points to any system resources that they are consuming, and each service can collect a list of any applications currently using this service's resources.

Thus for example, if the port management I talked about before was run by a service, its live folder would contain a collection of all currently opened ports on a given net interface, and would contain a link pointing to which net interface device it is regulating, among other bits of information.  Each of the opened ports in the collection would be a filesystem link to the live folder of the system service or application that has requested that port.  Because it links to a live folder, if you wanted to kill the process currently managing a specific port, you could very simply kill /svc/net.eth0/port/80, and the kill command would have no trouble getting its process ID implicitly.  And if you want to know who's using that socket or port, the app using the port should maintain a list of open connections in its live folder - because that is a portion of the application state that can and probably should be represented in the filesystem, once you've gotten all the rest of this up and running.

Oh… and as a nice side effect, you get a system collection of applications.  Almost seems like an afterthought now, doesn't it?  Even though distributed applications are the whole reason the entire system was invented…

Anyway, unless I grossly misunderstand the Unix specifications, all of this goes well beyond them, and it goes beyond Linux and other operating systems that I've heard of.  It still inherits some of the general attitude of Unix; I'm not looking to make it more like Windows, for example.  But it goes well beyond Unix, which is all that I claimed in the post title.

Okay now, take a deep breath.  Almost done with this one, I promise.

If the above were to be read by any serious veterans in the operating system design space, I suspect they would say among other things that app folders are all unnecessary and inefficient.  Even disregarding the, “What we already have, works” argument (which is kind of a lame argument, even when it is totally correct)… they still have a point.  Reconfiguring an entire Linux or other *nix based distribution to run on top of this service-collection-folder instead of non-filesystem-based services would be a massive undertaking, and there are absolute loads of questions that would need to be answered about how all the standards around these services should be set up: how app distro folders and live folders work, how the configuration mechanisms work, how drivers work - all of that before you start talking about what exports specific services should have, or whether and how the underlying architecture of the system should be reworked to use ports less and direct sockets more, plus more and more questions that bubble up as we move forward.  And trust me, everything I've said in this blog post - everything - is simplified, with pieces left off because I don't want to overload anyone, or trip over my own metaphorical tongue any more than necessary.

Honestly, you could multiply the whole collection of questions that are rattling around in your head by ten or a hundred and probably still not come close to all the various problems and questions involved in understanding and/or creating Project MAD.  The MAD App Folder framework is itself just a fraction of the Agentic Distributed App model, which itself is a fraction of the operating system, which was always meant to be attached to a whole hardware project that will probably never come to be - but that hardware project informs the software design and the operating system.  It is the nominal model, the stress test; it is important, not something I intend to just set aside or forget about.

When I say that Project MAD is a tangled mess of thoughts in my head, that is what I'm talking about.  It is “Take this problem that should probably take an entire community to sort out, multiply it by a hundred, cram it into my mind, and then try to reach in with chopsticks to untangle and remove it piece by piece” levels of tangled.  Again: the whole point of Application Folders was to help create distributed applications, but by golly gee whiz it sure would really make it easier to keep track of system services in a Linux box.  And again, this entire blog post is a trimmed down version of the application folders framework.  It's meant to be integrated into the directory and used alongside other concepts.

Weird how when you have a good idea, you start seeing uses for it all over the place, even though many of those “good ideas” won't be when you actually test them.  And yes - some parts of this won't be, and goodness knows I can't tell which from here.  But I think that it's a worthy idea that needs exploring, even if the MAD/AF framework is the only part of the system you port out.  Maybe it wouldn't be the best single piece, out of all the pieces that you could chop off and use separately… but you could.  And it would be interesting.

Plan 9: On Resource Distribution

 So I've recently become aware of the Plan 9 family of operating systems and related technologies.  Invented in the ‘90s by the geniuses from Bell Labs, it is a distributed operating system using a distributed filesystem - so it's not unlike in Project MAD, and as such, it's a good idea to discuss the one in terms of the other.  I have revised this summary many times by now, and I’m still not sure I'm happy with it, but here we go anyhow.

There are a lot of different ways to look at the differences between the two; I'll try to avoid cynical or self-serving “analysis” that paints everything I do as right and anyone else's ideas as wrong.  In part so that I can get this blog post out sooner, I'd like to focus in on one particular aspect first, and draw a distinction between the way Plan 9 and MAD handle resources in a distributed system.

I will summarize Plan 9's resource model as a being ‘pull’ model, by which I mean, the resources of the system (eg, devices and running software agents) are made publicly available, and any given program can “pull on them” (which yes, includes pushing data) from wherever the program is on the system.  I'll make an analogy here of two ships, at some arbitrary distance apart in a large body of water.  A ‘pull’ model involves a direct connection between the two ships, analogous to firing a grappling hook across the distance and pulling a line taut; you can then send resources back and forth across the line as needed, but because that rope is the sole point of contact, all negotiations between the two ships must go across it, from the most mundane to the most meaningful.

MAD does the opposite; it uses a “push” model.  Using the same analogy, a MAD ship will send out smaller boats, each towing a line, to go out and meet other ships.  Because these boats contain trusted crewmembers, they can negotiate and handle certain problems and events autonomously; not everything that happens needs to be coordinated with the mothership, meaning the line between the boat and ship can be reserved for important communications and resource transfers.

Stepping back from the analogy, Plan 9's uses a protocol called 9P that creates a shared filesystem, and then uses some filesystem functions for configuration, which selects which devices any given process should be using (eg, console, keyboard/mouse, display, etc), among other uses.  While I can see the vision, the choice to encode system configuration in a way that's hard to read and write is problematic.  Better tooling would help; it would be nice if the canonical configuration tool does what it does in a more user-friendly way.  Many systems have trouble with that, but it's something that really ought to be considered when making your tools.

Plan 9's design works because they chose to make everything in the system a file, and make those files available over the network.  The “Everything is a file” metaphor of course comes from Unix, and I will get into how MAD uses a different core philosophy another time.  Making a distributed filesystem that is available across the network is also a mechanism MAD uses, though differently; I will mostly discuss that more later, too.  The point is, Plan 9 decided on a standard interface for devices and other programs, a way to configure that interface, and a way to make that interface available across a network; that is the core of what a distributed system needs.  What makes it fundamentally a ‘pull’ model is that they didn't do anything more than making those interfaces available; in that way it is little more than a server-client model that would work just as well using the Internet Protocol as it does under 9P, if for whatever reason you chose to build such a network over IP.

Note that under Plan 9, this resource distribution model does not consider ‘the ability to run programs’ a resource analogous to other devices.  Believe me, I sympathize; I spent a lot of time thinking about how you represent the ability to run programs as a consumable resource generally, and while I imagine I have an answer, it is definitely something that can only be borne out through testing.  For certain, though, the ability to run program is not something that you can package into a file and make available on the network, at least not the way Plan 9 has done it.  (It might work in conjunction with the ADA, below, but not alone)

MAD's ‘push’ model also was born directly from the need to consume remote devices.  At first I assumed that, as in Plan 9, you are essentially performing remote procedure calls to reach every device in the system, but I didn't like that.  I decided I wanted to try to minimize network congestion, and negotiating with devices frequently comes with a lot of redundant back-and-forth for what is ostensibly a single function call on the program's side.  As I described in my blog post about the history of the programming model, I got the idea to execute scripts on remote nodes so that you could do several operations on a device at once, before finally settling on a formal, agent-oriented solution.

That solution, the Agentic Distributed Application model, requires that your application be split into several programs known as agents; no agent may assume the existence of any resource that it does not declare as a requirement, and no agent shall be placed on a hardware module that doesn't meet its requirements.  These agents are ‘pushed’ out to some correct hardware machine (the choice of which may be automatic, configured, or manual) when the application is loaded, and then all agents of an application simply communicate with each other, with the network abstracting away the distance between.  Thus each agent becomes a proxy with which to access the resource or resources that are local to that piece of hardware.

Of interest to what I said above, you can consider the ability to run programs to be a resource in the ADA model, because the resources here are understood as constraints.  This is useful, for example, when you have incompatible processor instruction sets present on a single system; you may only have AMD64 agents compiled for your program, and they shall not be run on an ARM or RISC-V processor.  In the same breath, though, you can extend the compatibility of your application by adding extra agents; since most agents will be very simple, the chances that compiling them will be difficult is minimal, and if you have some particular process that is more difficult to adapt to other architectures, it is fine if that specific agent is incompatible, while the rest of the program can be run on different processor types.  For example, using an ARM-based Raspberry Pi as your terminal means that you need input and output agents that compile for that processor (some asterisks here that shall be ignored), but the core of your application can depend on AMD64, which may be run on a full server, your desktop, or a laptop in a closet, whatever's easiest.

The ADA model's primary benefit is giving me, the system designer, confidence that we have enough information to say that applications and their agents can be automatically distributed throughout the system.  Together with software hooks that allow programs to override the default methodology, this resource requirement model filters out places where it would be incorrect to send a given agent, and because the application doesn't require a single machine to meet all of the requirements at once (I include access to files in this), the application as a whole is compatible with the gestalt system so long as all the capabilities it needs exist somewhere.  Similarly, because we control our assumptions, it is vastly more plausible to say that some Agents (especially a CPU-only agent, but really anything that doesn't have local side effects) can be paused and moved from one module to another without impacting the overall performance of the application, if for example the processor you're on becomes busy with some important task, or even if it crashes or disconnects from the system.

Under the ADA, all of these distributed applications have their own place within a shared filesystem, one that works somewhat differently to Plan 9's.  Each ADA application instance has a folder in the directory, and when an ADA agent makes requests for resources, it does so relative to this application-instance root; anything that the application needs should be mounted there.  Where Plan 9 makes the decision to change how any given application sees the global filesystem, MAD customize this private (but publicly accessible) directory of links.  Thus for example, they would look for a console device at ./cons, rather than /dev/cons.

Also in the application directory, of course, are links to the application's agents.  The application never needs to explicitly talk about where their agent is after it is deployed; they act upon the local link to that agent, which will proxy their request to the agent no matter where it is.  In fact the canonical location of these agents is likely to be mounted under the hardware itself, rather than in the application directory, because that's where the process is.  Either way, because these agents are filesystem links, they can be named properly, and thus be human-readable; a small consideration, perhaps, but one that helps programmers and administrators keep their sanity.

There are lots of other things that may or may not belong in this directory, depending on implementation: one good example is the application's user-specific file cache; rather than the application explicitly being told to (or independently deciding to) place that in some “/home/user/.cache/browser” folder, when the app is started, they assume that their own private “./cache” folder will link somewhere, whether that's a user folder or a private ramdisk that will vanish when the program closes.  Configuration files can be similarly linked (though MAD has an explicit configuration mechanism).

These application directories implicitly containerize applications.  If for example, your application depends on a shared library, they should expect that shared object to “be” within their own private directory, even if what is there is only a link.  If your app depends on another program, they should expect that program to “be” within their private directory.  In this way, the application works exactly the same way whether their dependencies are embedded or shared; it works the same way whether the application is “installed” or run from a private directory.  And, if a user ever has a need to forcibly embed an older or modified version of a shared library, for example for compatibility reasons, the mechanism for this is explicitly already there.  (Obviously, this is also a security problem that needs monitoring, but that's not difficult to address, at least in principle)

Much of the above will be modified slightly when I talk, another time, about how MAD doesn't exactly use the “everything is a file” core metaphor, but I think it all stands alone for now.  This is all, ultimately, about how you organize data in a distributed system; it all exists to help minimize the assumptions of the application, and thus increase its compatibility.

I have described Plan 9 as coming in at the 60-80% mark for what I'm looking for in MAD; this blog post is a good example of ideas that are in some ways compatible, but approach things from different directions and can have very different consequences.  It's easy to say in the abstract, for instance, that the push and pull models of resource distribution are not that different, because their goals and therefore nominal effects are the same; however, building a system on top of one or the other is a different task.

Part of what I see as valuable in MAD is this kind of perspective.  While accessing devices remotely (pulling) feels like a perfectly valid model (it is, to be clear, how the internet works), there are some questions inherent to a distributed system that it doesn't provide an answer for or a mechanism to assist with.  This difference is clearest when you are looking to solve all of those questions inherent to a distributed system - but it is a difference that any programmer for a distributed system will feel, when they are obliged to solve any of those questions for their niche application.

Having said that: the two models are not incompatible.  Indeed, while formal ADA applications take some work to implement, you can informally do the same as an app developer, absent proper tooling.  Even without something like Plan 9's distributed filesystem, with user consent, you could login to remote hosts (that are under their control, or to which they have file and network permissions), then copy programs over and run them, then connect to them at their known host and port.  The real goal of the ADA is to do as much of this as possible automatically, including (for some simple apps) dividing your program up into agents.  I'll be happy with the ADA when (if) you can take a program not meant explicitly to be distributed across multiple machines, and run it distributed anyway without effort, because that's what it takes to make a consumer-ready operating system.

That task, automation, takes some advanced thought and planning, and that's part of why I am hesitant to say that I've solved anything “for real” without talking to experts.  It's too easy to find that you have assumed wrong or overlooked something and now have a much more difficult problem to solve than you thought you'd have, a problem that might have been easy if you had done it all a different way.  That I can say MAD makes things easy where Plan 9 makes them difficult, also means that MAD might be doing things the difficult way when someone else can do it easily.  That's simply how having different methodologies, works.

And that's part of what makes this all so much fun.

Programming MAD

One of the important things about the Agentic Distributed App model, as I've already said in a separate blog post, is that we need an application model that just anyone can jump in and create programs that span the gap between multiple machines.  There is, naturally, a lot wrong with merely saying that this is possible, simply leaving it dangling out there with no questions asked nor answered.

After all, when people talk about distributed computing research now, they are talking among other things about problems that would stymie novice programmers: things like memory coherency, race conditions, partition tolerance, reliability, security, and of course system topology.  I think it's fair to say that most programmers, to say nothing of new ones, prefer to think about and work with single-threaded logic and not address the various dangers and brain-benders involved in keeping memory synchronized between two independent machines, either of which might create a sudden shift in the status quo.

While I do have some thoughts about multithreading, and Project MAD's place in that morass, I'd like to start somewhere simpler.

Distributed Single Thread Programs

One of the things that might not be obvious from everything I've said so far, is that the ADA is supposed to among other things empower single-threaded programs that just so happen to be distributed across multiple machines.  Even if these machines and the software agents on them are executing code at the same time, it remains important to create applications with centralized logic, where all events pass through a central filter, ensuring that events never conspire to have two parts of the same program working at cross purposes.  This is not the only logical model that an ADA application can have, but it is the most reasonable model for many applications to take, or to put it more bluntly, it is the only route many application programmers will ever want to take.  Understanding and dealing with parallel operations can get complicated very fast.

If this still seems odd, understand that many Agents under the ADA model are nothing more than handlers for specific resources, translating internal application events to external events and vice versa.  If you want to process keyboard inputs, something needs to be waiting for those keyboard input events on the machine that actually receives the event, and perhaps some basic processing can be done there, but the active logic that responds to the event should be a part of the main logic loop, at least under the single-thread model.  Likewise, if you are performing GUI operations as part of your program, the functions that actually get placed in a GUI Agent may be little more than batch commands that draw on the screen and handle windowing events.  They may have little or no program logic involved; that may mostly happen in a centralized location, most likely the program core.

The ADA is not specifically about making multiple threads cooperating in parallel easy, much less trivial.  It is about making sharing of resources across device boundaries easy.  It is also about creating an application model that explains how you deploy an application into such a system, with minimal assumptions.  It is about handling edge cases like only having disk access in one remote location, while most of the computation and devices used are elsewhere.  It is about adding new resources in such a way that others how what you have and how to use it, even if that new resource is wildly different from everything you have so far connected.  None of that obliges an application programmer to take classes on networking and distributed systems.

Granted, the ADA is more useful when you are tackling parallel or parallelizable loads.  A single threaded app in a distributed system will be slower than the same on a monolithic system, but a massively parallel task can use the ADA model to natively farm out tasks to any and all processors within a system (or just to some of them) in order to make use of spare capacity that might otherwise be unused.  And while it would be unkind to suggest that the ADA simply will have a load balancer built in for parallel loads, it unquestionably has everything you might need to make a good load balancer.  Metrics like system load are important to consider when determining where you will deploy new agents, or which agents to deploy a workload to, and the ADA shall be able to dynamically add and remove agents anywhere in the system.  Thus, for things like supercompute workloads and web service load balancing, the ADA should give you everything you need for an ideal solution, whether that ideal solution comes built in or not.

It should be emphasized that this comes without needing to design and build custom hardware.  Many data center companies invest heavily in computers designed to pack as much computing power and/or storage into as small a chassis as possible, and while that remains a possibility, the MAD computing paradigm will let you create a machine that has the same capability as one of these advanced systems out of spare parts.  Where space or efficiency is critical, it still behooves one to invest in specialty hardware, but it sure would be nice to know that wasn't the only good option.

The App Streaming Model (For All Applications)

It's also worth noting that there are present-day use cases for the MAD paradigm that have little to do with parallel loads.  One, for example, is the game streaming model that several games companies have attempted, and to some extent succeeded at, in the past decade or so.  It is sometimes very inconvenient to have multiple game consoles tied to your television, or to only be able to play PC games while sitting at your computer.  The standard application model we all grew up with had no solution to this problem, and so several companies developed edge-case solutions to attempt to fill the gap.  Some, such as Google Stadia, collapsed relatively quickly, while others, such as the game streaming engine built into Valve's Steam platform, have continued.

Likewise, mobile app platforms have attempted over the years to design a “streaming app model” that doesn't require mobile applications to be installed, while still having all of the same benefits to the user.  This is designed mostly to remove the need for web apps, which are far from an ideal way to design and distribute applications.  Web apps persist even today because they are a model that is completely understood and completely under the programmers' control.  Web apps don't need much special training to create if you already have web programmers, and can flexibly handle different mobile operating systems and even desktops - but they are more limited than native apps.  At the same time, from a user's perspective, it's nice not to need applications persistently installed on your phone, especially if they insist on taking up background compute resources, need constant updates, or like sending you unwanted notifications.  It's nice to simply run an app without giving anything up to the people who created that app, and without going through a store and tedious install process.

And I'd be remiss not to mention interactive video peripherals such as car entertainment displays, or like Chromecast.  In a way, these solutions prove the fundamental premise of Project MAD: once you get used to the idea of simply connecting your phone to a thing, it becomes difficult to understand why this has been difficult for so many years.  Of course you can trivially stream video from a phone to a TV; of course you can use your phone's navigation in your car.  The problem isn't complicated, and the moment these technologies appeared, we wondered why it took so long for them to arrive.

But each of these solutions is a step to the side.  They are specific-case solutions to the general problem: Input, Output, and Processing are on different machines, but they are all part of the same application.  Trying to take one of these existing, specific-case solutions and turn it into a general-case solution will not work well, because the assumptions do not transfer.  Certainly, for the examples I've listed, you can understand that really only input, output, and processing are involved, but for example: where are your files stored?  In most cases, they are stored where you do the processing, but what happens when that isn't true?  For Chromecast, for instance, the dongle that attaches to your TV doesn't always stream from your phone; it may stream from the original source (Youtube, etc) and bypass your phone completely, which is good, but if you need to store preferences, that will end up being done on your phone, not the dongle nor in the cloud.  (Of course, the cloud service will automatically update your history in most cases, so there is storage there, too)

So now, the model must account for input, output, processing, storage, and media source.  That growing list only serve to change the question: what have we not thought of yet?  What else might be a resource that we need to deal with?  How do we deal with all of these existing devices, and how will we deal with any new ones?  Do they each need special consideration?  In which ways will we need to alter the fundamental model to incorporate each?

And that is the core advantage of a general case solution: the model doesn't change as you add new resources.  You don't need to treat adding remote inputs differently from adding remote outputs.  You don't need to treat adding writable storage different from adding media sources.  Each Agent that you add improves the capabilities of the Application, but that doesn't complicate the model.  The ADA handles game streaming, app streaming, chromecast, car displays, home media libraries, and many other edge cases with the same fundamental underlying model.  Products that were headline news when they first appeared become obvious applications of the underlying technology.

Similarly, the Internet of Things device paradigm has been waiting for a model where either software on your network takes charge of the peripherals, or the peripherals can seamlessly install a service or application on a piece of hardware you control, which you can then use to control it.  The former describes the ADA in general, while the latter is similar to app streaming.  In either case, software must bridge the gap between devices; the question is whether the user brings their own software (which controls the devices via ADA agents) or whether the devices supply it (your phone acts as an ADA server and runs an app stored on the IOT device).  As a bonus, the ADA as a framework would suggest that IOT devices with their own control apps could be controlled by ‘thin clients’ consisting of nothing more than an interface, such as a wall-mounted display.  If you already have an interactive display with a wireless interface, perhaps on your thermostat, refrigerator, toaster, washing machine, oven, shower, water bottle, toothbrush, hair brush, stuffed animal, or sex toy, that display could control your other IOT devices in exactly the same way that your phone or computer could.  This, too, has nothing to do with parallel computation; it only has to do with connecting devices.

Granted, I am being perhaps a touch sensationalist, but only because it is frustratingly difficult to do things that should be easy.  Without the perspective of MAD/ADA, I can agree, the problems look tangled and difficult.  Every addition to the model is something new, and each might add some new complication.  And as I've said multiple times, if my solutions turn out to be inadequate, I still believe that finding a general-case solution to the problems created by something like Project MAD will have most if not all of the benefits above.

One way or another, I hope for a future where we can just do fun things with our devices.

History of the MOS/ADA Programming Model

 This topic is tricky to introduce.  One of the central tenets to Project MAD as it is today, is a whole new model for applications, and I've said that it was the last things that came together, and it tied everything together in a neat package.  What you won't know is that my early concepts of the MOS/DCA project also included a modification to standard programming models, and that legacy is still very obvious once you know to look for it.

It's difficult to ferret out exactly what I meant by it if you're just reading my notes from back then, but it was there: I wanted to expose the internals of executables and shared libraries so that, say, a shell command could execute an arbitrary function.  Important to making that work is having an interactive shell that can capture typed data - more like an interpreted language's REPL loop than something familiar like Bash or Windows Command shell.  And this has to be a structured tool, in some fashion; in modern applications programming, you pass nothing to an application when it starts except strings, and the application has to figure out literally everything from that.

This ties back nicely to my complains in the last couple blog posts about programmers being obliged to reinvent the wheel.

At the time, what I had wasn't anything like a plan.  At most, I had a complaint.  And that sparked a wish.  After all, especially within the GNU toolset, how many applications are actually just wrappers around library functions?  Complex wrappers, sometimes; that complexity is frequently needed so that this one tool can interact with a complex programming library function in multiple ways, or so that common data pipelines (piping a list of files through grep, as an easy example) operate smoothly and the way you would expect them to.  As with a lot in programming, things that are simple to say are frequently hard to do, and if you want to actually do them, there is a lot of nuance and complexity that must be navigated.

The other side of that, though, is that the nuance and complexity spawns more nuance and complexity.  Your complex tool needs to be interacted with in complex ways.  Add a third layer on top, that assumes access to tools that use libraries; then add a fourth layer and more, tools on top of tools.  As long as nothing changes, eventually you get a level of tooling where you can issue a one-word command to do exactly what you mean to.

But every level in between the core library and the final level of tooling can shift.  Needs to shift.  Every layer must be updated whenever any of its dependencies update, for security reasons if nothing else, and sometimes bigger shifts are called for.  To a certain extent, there's no getting away from that - but if it were possible to have the library function that you actually care about, be executable in its own right without an application wrapping it, there can suddenly be whole levels of redundant tooling that are at least simplified, if not eliminated.

And of course there will probably be several other layers of tooling that crop up around this new concept; I won't lie.  Ultimately, we create tools because we have specific needs, and as long as the tools fulfill specific rather than general needs, they will diversify.  But many of those pain points that we are removing are redundant ones.  Every application needs to transform text input into data streams.  Applications that read from and write to data files have to pick or create libraries to that work with the specific protocols and data streams.  And if you're creating a workflow that involves data files being processed, you need individual applications whose only role is to be a part of that workflow, applications that parse files and put them in a form other applications can take as input.

Many of these niche applications, that only exist to take one data format and make it available, or take text input and render it back into a data file, are at once important to someone's workflow and possibly security, but also are tools nobody wants to maintain.  Nobody wants to need to maintain them.  It's important that someone creates a tool that parlays between data files and workflows, but it's a thankless job, and a product nobody wants to pay for.

In short, a shell that understands data objects and types, and can use that to call a function built into an application or library directly, instead of needing to build one application after another to handle the workflow… that would solve a lot of problems.

The steely-eyed veterans in the audience are saying, “Well, we have exactly that with interpreted languages like Python.  In Python, you can open a library and call a specific function out of it.  If you have a python file that is meant to be an executable, you can also load that as though it were a library, and call functions out of it, assuming it's designed right.”  And this is, of course correct.  (For the record, I had these thoughts before I was introduced to Python.  Probably.  Around that time, at the latest.  Again, my notes are not great.)

The difference is having that be an assumption built into the entire operating system, into the fundamental programming method, so that the entire system is built on one foundation instead of a shifting mix of dependencies from various legacies.  That is, of course, making a virtue of a vice: it means starting over and doing everything from scratch, which is... painful.  And these initial thoughts of mine were missing critical parts of the picture, which is why for many years I set them aside as irrelevant and a waste of time.  At the time, all I was thinking was that, doggone it, we shouldn't have to reinvent the wheel over and over.

And in truth… there are still today whole categories of errors that only exist because library functions are sealed away.  For example, languages like Python, PHP, Go, and many others can't easily use functions buried in libraries for other languages like C; people have to write new versions of old libraries, and each needs to be maintained separately, by different people, using different language constructs and different dependencies.  Ultimately, it becomes such a different problem to maintain each of them that even if one person was willing to maintain every copy of the same algorithm, it would become untenable for some fraction of them.

All this despite the fact that ultimately, all you want is to pass parameters in and get results out.  If there were no worry about dependencies or languages, if all you had to do was call the function and capture the return, it would be simple to do from any language.  Of course, from a certain point of view… that's always been all you have to do, if your language is willing to parse library headers or code from other languages.  But the library formats in use, Linux's .so and Windows' .dll, neither comes with a human- or machine-readable directory of functions meant to inform programmers of how to use them.  If you want to know how to use them, you need specific files intended for developers to use.  That means if you have an entirely different programming language and they want to use existing library functions, they need to read those files, …despite them being in another language.

In short, there's a reason why we are where we are.

When I talk about Project MAD as a change in the entirety of computing, it can sound like I'm vastly overestimating my own cleverness - and well, maybe.  But these are real problems.  My first concept almost sounds promising on its own: just call library functions.  But that concept does nothing about compatibility and discovery problems.  It hardly matters whether or not the code exists on your system if you don't know that it's there, where it is, what it does, how to call it, what side effects to expect, and what the return values and errors mean - and that assumes that errors get propagated correctly.  And the details need to be exact; one bit out of place changes everything.

Just like there's a reason why we got here, there's also a litany of reasons why we haven't leaped forward.  Even if we all agreed on one bright idea that we would work together on, it would be a massive effort with a lot of problems that need to be solved.  And if the bright idea that we all agreed on happens to have massive holes inherent to it, because it's not as bright an idea as we thought… that's a lot of work for nothing.

That's to say, I do understand, have always understood, and will always understand, that none of this is simple, that I can be wrong, and that ultimately, even if my idea was core to anything, it would be the people who actually achieve it that deserve the credit, because it will be very difficult.

But also…

But also, various subsystems that are a part of the MAD concept (here, specifically, the Modular OS and Agentic Distributed Application model) are there specifically to solve these problems.  I've described the system directory in this post on the Unified System API, but not all parts of the solution came around at the same time.  Part of it, as I've just described in this post, was there from near the beginning.  Many years later (five to ten, maybe) I started thinking about the directory, and specifically, having types understood system-wide and having them be a central part of the system directory.  Before this, all I really had was “Call a function inside a library, so we don't have to reinvent wheels.”  But it didn't make sense, quite.

I said in my first post that the ADA came about in the process of writing blog posts about the MOS/DCA concept.  I was trying to explain, and things still felt incomplete.  I had deduced that functions and hardware drivers needed to be sorted by function, which fed into the type hierarchy in the system directory, and I was also thinking about embedded libraries and applications, and how that made application deployment and dependency management neater.  I was writing about wrapping functions with scripts so that some logic was happening on remote nodes, but that wasn't working for me.

The reason why I like the ADA goes beyond the model itself.  The model slotted well into the weirdly shaped hole that the whole “software model” side of the project had.  Yes, you could call functions directly instead of only being able to run applications - but why?  What made that so compelling, and not merely one person ranting at the sky?  Sure, it would be nice to have a directory of all the functions and types within a system, but what made that so critical to the operation of an OS that it has to be a fundamental component?

But within the ADA, you may be calling functions within your own application agents and within others'.  You will be deploying agents to hardware modules specifically because there is software on that module that is required by that Agent.  And it goes beyond functions and code (or, well, it doesn't, but it helps to think of this way): the ADA is also about exposing data so that applications can be monitored and debugged, but that requires types to be in some sense objective, even if that only means a type connected to the application and not a system-wide one.

Equally, within ADA, an Agent exists to handle the tricky problems of accountability.  There is an entirely reasonable question when you try to, for example, perform GUI API calls remotely.  The memory being used by your GUI application can't leave the display controller, or at the very least, there will be a copy of the output in the display controller, which must be managed by the GUI stack itself--while being owned by an external party.  Without an agent such as the ADA's, there are massive, massive questions to be answered about how this is not an absolutely terrible idea.  But if you can't do things like call GUI functions remotely, then... what even is the plan?

But perhaps most importantly, in the ADA, deploying Agents and routing messages is a system function, one attached to the ADA servers and not one implemented by applications themselves.  That means that knowledge of functions, data, and types needs to be standardized at a system level.  If the ADA model were implemented separately by each application, then each application would need to separately track and manage type and API version compatibility.  Implementing this as a system function makes things not only possible, but easy for programmers to handle, by doing the hard work ahead of time, in the operating system.

I like to, perhaps arrogantly, think that I am touching on fundamental truths with some of these analyses.  It's why I keep hammering, for example, on don't force every programmer to reinvent the wheel.  It is hubris to say I alone know best (I've demurred enough that you know I'm not that hubristic), but it's still satisfying to line up a whole bunch of problems in a row, pointing at them, and say, Example one, example two, example three.  Tying things up with a bow on top is satisfying, and the point of feeling genuinely excited about the ADA and Project MAD in general is that a stack of problems are placed in a box and wrapped neatly up.

It also feels good because this isn't all the work of a day, month, or year.  While this talk of modifying programming models may not have happened in the first year of the project (the first part was mostly the DCA, honestly), it's at least 15 years in the making.  Tricky problems that have tickled the back of my mind for years are coming together.  These are problems that weren't just tricky in practice.  They were tricky conceptually; it wasn't clear that there even was an answer.  It wasn't hard to imagine that I was wasting my time, because there was no clear vision of the end.

There's still a lot rough, and there's still lots of room for me to be wrong.  But it's astonishing how the last few pieces have made the puzzle come together.  At least from where I stand, it looks promising, like there's really an answer.  Not an easy one, but one that leaves us all much better off.

And even if I'm wrong, perhaps getting more eyes on the collection of problems I've been working through will help.

The Problem with Modern Computing

It's really hard to summarize all of Project MAD; it's one of the reasons why I didn't shy away from that particular three letter acronym.  Perhaps more important, it's hard to explain why I feel so darn sure that I have stumbled onto something meaningfully important.  My descriptions of things can get all tangled up, so it's easy to think that the core of the idea is equally tangled, equally lost.  But I genuinely, truly, believe that there is something fundamental here.

Our world has many more computers than ever before, but each computer we sit down at can feel just as confining as the last.  We're chasing after the feeling of being empowered by computers, the feeling that computers are the future, not merely the present.  I suspect that's part of why so many people are jumping wholeheartedly into AI fever; they've been looking for a long time for something that will make computers feel infinitely deep, again.  Like you can get lost in all the possibilities that a single computer can offer you.

In the era of the internet, it's become less important that a single computer feels like it contains infinite possibilities, because we can always gesture out the window and say, infinite possibilities are out there.  But that answer is less compelling than it seems, for numerous reasons that I don't want to get into right now.  A single computer can be an infinitely flexible tool, and we understood that from nearly the beginning.  Computers on wheels become cars; computers with cameras become security guards; computers with arms can weld or do other labor.  But many of these implementations moved away from depending on normal computers, by which I mean “the same as we use at home”, and with good reasons.  Security and stability, mostly, but also… also deeper reasons.

Computers are generally split, and again with good reason, into utility and user-facing computers.  Utility machines exist to get work done; user-facing machines cater to the person in control of them.  What is lost in this description is that any user-facing machine could do work, even if not every utility machine can be user-facing.  But that's not what we see, not what we feel when we work with user-facing machines.  Even though you might be part of a larger system with great capabilities, it doesn't feel like it.

The reason is because programmers need to reinvent the wheel, whenever they try to merge two machines into one.  It can be confusing to try to wrestle with the very concept of controlling one machine from another, not because that's difficult, but because it feels like it should be a solved problem already, and it isn't.  There are solutions, but… as my last post tries to explain, perhaps poorly, they are kind of poor solutions.  They were never meant to do what we really want them to do.

When I talk about remote procedure calls and the unified system API, what I'm talking about is controlling one program from another, one computer from another, in the same way that a programmer controls their own program.  If one of those remote computers, one of those programs, is in charge of handling hardware devices, then controlling one program from another means controlling hardware devices as though they were just your own bit of software.  If you use those remote computers to run software, to provide services, they should be as easy to interact with as if you were running software on your local machine.  The services should always be there, ready to serve you.

But most important, you should be able to write a program that just does stuff.  Not only doing stuff on your local machine, because as I just said, most user-facing machines don't actually do stuff by themselves.  There are other computers that do stuff, while your desktop, laptop, or phone just… well, provides you access, in theory.  In practice, because there are fewer standards than we'd like, figuring out how to control things that you own, even things designed to be controlled, can be an exercise in frustration.

Because the people who make those devices are all forced to reinvent the wheel.  Collecting all of the various hardware capabilities of a household's machines, or a user's personal stuff, into one gestalt system isn't a standard function built into computers, and neither is it straightforward to make good use of them once you have them collected.  If you want to connect multiple devices, you have to work with multiple vendors' proprietary and non-standard interfaces - and many, perhaps most, of these interfaces weren't meant to be controlled by another piece of software, but only by an interactive user.  That, too, is because they would need to invent that particular wheel.  They would need to secure it, and take responsibility for what happens when it leaves their warehouse and enters your home.  It's a lot of risk for little if any monetary reward.

The reason why these problems are unsolved, is that the alternative is reinventing all of computing.  That sounds extreme, but this is the argument I've laid out: software as we know it is not meant to connect multiple computers together, it is meant to be run on one single computer and nothing more.

This, ultimately, is the problem.  The big one.  It's the axle that must pivot if you want the world to change.  Run a single program that spans two or more computers - but it's not just about running it.  It's about teaching everyone who wants to program computers how to control two machines with one program.  Or three computers, or four.  Or an infinite number.

How do you simplify the process of making that program so that just anyone can do it?  So that Joe Shmoe, who has a day job and doesn't want to go to college for programming, can hack together a quick and dirty program that makes use of all the computers in his house?  What does a programming class look like, when you are teaching children to not merely twiddle with bits on a single computer, but to bring a motley assortment of machines into a room and unite them into a grand symphony, all orchestrated by a few words of text?

How do you let someone explore the infinite depths of possibility that exist within their computer?  Those depths exist, but we cannot explore them, not without reinventing wheels over and over. We shouldn't need to re-solve problems that others have good answers to, because our amateur efforts will be… amateurish.  Any skill we lack, and problem we can't tackle ourselves, derails a project.

Project MAD is all about making the capabilities of a computer available for you to tinker with.  It is about the capabilities of a single physical machine, but also, about all the capabilities of all the things you own, thrown in a pot and stirred.  It's about having a spare computer running in a closet that takes part of the workload that would be on your desktop, or your laptop, or your phone, and you don't have to set it up specifically to do those things.

It's about taking a computer that has worked fine for years and still making use of it even when you replace it with a faster and better one.  It's about dumpster diving and collecting random pieces of hardware and making a real workhorse out of them.  It's about buying an under-powered laptop and not caring, because as long as it's in a network with other machines, it will be just as powerful as any other.  It's about finding creative uses for old TVs and monitors, beyond being picture frames.

And from the corporate perspective?  It is, genuinely, about creating new products, products that people want, and it's about being able to just turn the crank and trust the process.  It's about creating an operating system image that you barely touch when you ship a product.  It's about shipping a hardware product and not meddling with software at all.  Imagine, not worrying about completely unrelated software that is only necessary for your device because you don't know how to let consumers control your hardware except with complicated software stacks running on top of full operating system images.

It's also, on some level, about going back to computer hardware being purely a commodity, able to be produced cheaply, and not needing to be shiny, chrome-plated best-in-class specs to be useful.  It means that when our need for computing power increases, we don't need to cram more into a single chip, with all the heat issues and power consumption that come as a result.  It means that the top of the line chip you buy this year will work together with the chip you bought two years ago, and the chip you will buy two years from now, instead of them all ending up in the garbage.

At the same time, it's about creating hardware that really is genuinely unique.  About creating new processors that are optimized for a specific task or problem, without needing to invent a new OS to run on them.  It's about creating hardware VPN devices that will let you connect securely to your home system while traveling, or your work system from home, just by plugging them into an otherwise unsecure system.  It's about creating software that is secure because the hardware it runs on can't be overwritten or even looked at by others in the network.

But it's also about taking your computing environment with you.  If you can expand one computer by connecting it to another, then have your phone really, genuinely be your computer.  Even if you aren't plugged in, massive processor-heavy tasks can be run on a machine that is plugged in, and all your phone has to power is the wifi.  But it's more than just power.  The system you sit in front of being the same as the system in your pocket, means that some important things aren't locked in one place or the other.  Your calendar and messages, phone calls and podcasts.  Your music and movies, your games and documents.  You shouldn't need someone else's permission to have things you bought accessible anywhere.

All of this - all of it.  Every last bit of this comes down to the idea that any program, written by just anyone, can unite multiple hardware systems, and do it as easily as we can write any other program today.  When you can do that, your system gets more powerful because other systems are available.  Your system gets more flexible when you have more options.  You can have a definitive “core" of your system that expands to fill any container you put it in.  Vendors can compete to be a thing worth adding to your system.

But most of all, once it's trivial to unite systems, we are no longer trapped within one.  In a world with the internet, it feels weird to suggest that we are trapped.  My mother bought a laptop just a few years ago that is under-powered and can't be upgraded, and all she really knew was that it was “slow”.  She hardly needs much from it - she writes, mostly - but it was still inadequate.  Someone sold a computer with so little ability that a writer feels limited using it, and there's simply no way to make it more powerful by adding to it.  You can't add memory, you can't even add disk space - not except SD cards, which doesn't help when Windows is complaining it doesn't have enough disk space for OS updates.

If she could just use that laptop as a dumb terminal for a session on a more powerful computer, she would be happy.  But she's not tech-savvy; me, I am, but I still struggle to combine my home server, main PC, assorted Raspberry Pis, and multiple laptops in any meaningful way.  I run some applications on the server as web apps (including the one I'm using to write this, before posting), and I store files on the server so that I can use them anywhere, but all those solutions are …weird.  If I tried to explain any of them to my mother, or frankly anyone else in my family, they'd dismiss the possibility.  Too much effort for a result that can't just be used.  Even I have to work around the limitations, get frustrated by the nuances and complications.  Even I chafe at making sure all my operating systems are up to date, especially when it's all redundant.  Even I, a tech enthusiast through all of my life, would like to have my entire tech world be part of a single, simple system.

It's one think to talk about the pie-in-the-sky dreams and hopes I have for Project MAD as being features.  I am reluctant to promise features because it feels like I'm trying to plate the project with chrome and gold, garnished with diamonds to give it that extra sparkle.  Clearly, from this post, I believe that the potential exists to do great things with the Project.

But it's more correct to say that I am constantly, constantly disappointed with computers.  I am disappointed with Windows.  I am disappointed with Linux.  I am disappointed with Android.  And while I haven't used Macs and iStuff in a long time, I don't think I'd feel much different about them, because it's not about the vendor.  We simply don't have solutions to some problems, currently.  We can make the best of what we have, but it's not the same as having some very real, very frustrating problems just be… solved.

Maybe Project MAD can't do that.  Maybe I can't.  Maybe all of my philosophizing and promises will turn up empty.  Who knows, maybe I'll get in a car wreck tomorrow and since there's nobody else on this project but me, it will all vanish into the aether and be forgotten.

But the problems are real.  The fact that we are working off of a flawed model is real.  MAD may not be The Answer, but it is an arrow pointing in the direction we need to go.  If it's never more than that, but if it succeeds in being that, then I'll be happy.  If I can never make the OS, if I can never make computers that snap together like legos, if my model for distributed programming is discarded and never looked at again… but if, if all of my efforts point the way to a solution later on, then that's fine.

Part of calling it Project MAD is that I'll understand if the MAD solution isn't the one everyone embraces.  But I know that there is something here.  I just… don't know how to convince anyone else.