Conceptual Walkthrough: Distributed Applications

It may be surprising, since I am here to talk about distributed computing, but I hate process forking - making copies of a running application, each running the same code but each dedicated to a different task.  I'm sure I'm not alone in this, because it's not as though people are running around calling fork() for every little thing; multithreaded and multiprocess apps are not exactly common, not unless you're doing something that really requires such a technology.

Frankly, it's a headache to try to understand.  In theory, it should be simple; there's a copy of your program, but you know which copy is which, so you just do different things with them.  If before that, you open up a pipe, both copies of the application will have access to the same file descriptor for that pipe, and they can communicate through it.  But all of those things being technically possible isn't a plan.  When that pattern is laid out in a book, it leaves you wondering: what am I supposed to do with it?

I talked in several places about programmers being obliged to reinvent the wheel, and process forking is a good example; forking wouldn't be such a terrible thing, if it was paired with a functional remote procedure call mechanism.  When two copies of the same program are connected with a pipe (or two), you can use it to pass data, but that data transfer is mostly pointless without context.  Adding context, labeling the data, makes it essentially a very simple, poorly implemented remote procedure call, where the "function" that is being "called" is simply an index specifying what to do with the data, out of a set of options fixed at compile time.  While process forking would still be a bit confusing if the process forking mechanism came with a built in RPC mechanism, at least the programmer would not have to wrestle with all the complexity of implementing their own RPC stack from scratch while also trying to build program logic around it.

Although I wasn't thinking of it when I designed the system, the ADA distributed applications model is basically exactly this, and to help you understand it, let's walk through the analogy a little more closely.

As I see it, forking has two fundamental problems with it; first, the fork is undifferentiated.  It is easy to overlook the fact that it is a feature of programming languages that we can attach variable names to data in the first place; when I say that forks are undifferentiated, what I mean is that this deliberate feature and mechanism of programming languages no longer functions.  In each half of the program, a large number of symbols exist but point to the wrong data, and it is entirely up to the programmer to recognize this and keep track of which symbols are now incorrect.

This is in fact the reason why I have specified that under the ADA, programs should be written and compiled such that each module which is to be depoloyed separately, shall have its own variables confined to a single namespace.  Namespaces in this context may be nothing more than a collection of variables given a group name, but for our purposes, it draws an explicit distinction between what is, and is not, accessible within a specific context.  You could, of course, do the same thing with a general forked application - but it still leaves us with the problem of needing to create a new remote procedure call mechanism.

Let's take the "remote" part out of the equation for now, and only talk about a forked application communicating with another copy of itself.  A procedure call mechanism in this context is a pipe that you periodically check (or block and wait), looking for a data stream in a fixed format.  That data stream will tell you what function to call, or what variable to write to, or what variable to read from, and it may also include parameters for the function (or an index to an array, etc) and then you return any out-values through the pipe to the other side with a similar mechanism.  Likewise, the calling side of the pipe will send a packet in the expected format down the pipe, and then block and wait for a return value (or merely an acknowledgement that the function is being run, or that it has completed successfully, depending on the context).

You as the programmer will most likely want to select some highly specific subset of functions or variables that you want to use with this mechanism - when building it yourself, you will most likely only accept some very few "commands" that you listen for and respond to from the other half of your application.  Because both halves are the same applicaiton, though, you could also use a lookup table that lets you unambiguously access any variable or function present on your half of the application - there are, after all, a fixed number of those functions and variables, and you know for a fact that all of them have unique identifiers, because you use those unique identifiers while programming.  And once you have selected a function and/or variable to access, any parameters that get sent along by the request can be type-checked, because you know the method signature or variable type.

Doing all of that work by hand would be an awful experience, and most if not all of that hand-done work would be wasted, as you are unlikely to call most of the functions or access most of the variables.  If however, it were a function of the programming language, compiler, and linker - that is eminently possible, because that is, essentially, the compiler's whole job.  Translating an identifier to a function, checking the number and type of the parameters, all of that is necessary to create the program in the first place.

Suppose, then, that you had a compiler that created an indexing function.  These kind of functions exist, when you use reflection tools - the indexing function would take a variable name or function and some arbitrary parameters, and call the function with those parameters, and then the indexing funciton would return the variable or return value when it's done.  Let's suppose at first that exactly one indexing function is created for your whole program.  You as the programmer might simply pass anything that comes down the fork pipe to the indexing function (unless you just called a function, in which case you would wait for the return value and parse that) - in fact, ideally, there would be a built-in function that just reads commands from the pipe in exactly this way, and a paired function that writes to the pipe in exactly the way the reader expects.

This is still relatively unsafe; there is a lot of potential for race conditions and so on, and there is nothing stopping you from requesting data from the fork that is actually stored somewhere else.  But assuming you had some need to fork an application and run it, this reflection-based indexing function would free the programmer from having to figure out how to pass messages back and forth, free them from having to write any middleware, and simply allow the programmer to use the capabilities of the other process, its data and functions.

But let's go back to the namespace thing.  You see, we are still left with the problem I started out on - that a forked process has duplicate identifiers, half of which are simply wrong, because the actual memory location where they are stored is in another process.  But, you know which process they are stored in, and you have a well-established mechanism for requesting data or calling functions in that process.  If you knew for certain at compile time that some subset of those identifiers would be in the forked process, the compiler might translate any requests for those identifiers to instead use the pipe-send function.

But wait, there's a problem: both forks of the application are running the same code, so at compile time, it's impossible to distinguish whether the request is going to be requesting data from the original or forked process.  This is where namespaces come in - you may not know whether a given namespace is in one process or the other, but you should know for certain that the contents of a specific namespace will all be in the same place.  It's only when you exit the namespace that there is any chance you will be trying to reach data that exists in another process.  That's simple, then: in the forking function, allow the programmer to specify which namespaces are being detached, resulting in two collections of namespaces, one associated with the original process, and one with the fork.  This will be a runtime check, in case there is any reason to change which namespaces get separated in different circumstances - because it is a runtime check, a checking function will be called every time any bit of code requests data or functions outside its own namespace.  (There are ways to reduce the overhead, but ignore that for now.)

Well, if you can do all this once, why not do it multiple times?  Say you have a program with five total namespaces and you want each in its own process.  There's no reason why that should be difficult, should it?  You would simply need the runtime check, the one that knows which process contains a given piece of data, to also select the appropriate pipe to use.  There is, admittedly, one complication - making sure that all forked processes receive updated routing tables - but since you've built a robust framework for sharing data already, that hardly seems like much of an imposition.

It's worth pointing out, though, that if this system works correctly, it should also work correctly if you never fork the application at all.  In that case, there is no confusion about which process owns which variable - all of them are stored in the same process.  So, your program will truly not care whether it is currently running in a single process, two processes, or five processes; when you as the programmer insist on calling a function or fetching data, the function will be called or the data fetched, no matter which process currently stores that particular piece of data.

Now, forking a process is necessarily something that happens in the same place where your application currently is, which sharply limits the usefulness of everything I've just described.  Not to say that it's useless - I think anyone who is interested in forking processes would appreciate having all of this functionality - but what I've just described is not actually the functionality that I want from the system.  A distributed application, as I've described it, is one where this forking process happens between two computers - where you can run part of the program logic on an entirely separate machine.  And conceptually, once you've done everything I've said above, that's not actually a difficult proposition... to an extent, at least.  You need some permissions, and you need to make certain that dependencies and the like are synchronized, but the technicalities of starting a process somewhere else, and binding a pipe that connects the two processes over the network, are not particularly difficult.

I've said this in other places, but to reiterate here, the value of doing this fork over the network is allowing your program to place specific bits of code where they are needed, without the program itself needing to care that those bits are not on the same machine.  Generally, these "specific bits" are in one of three categories: input, output, or computation.  The benefit I'm talking about is not accelerating your program with parallelization (you can do that if you have a whole lot of computing to do).  No, the benefit I'm talking about is a kind of program that we don't currently possess - one that replaces the idea of remote desktops and SSH tunneling with running an application in one place, and handling the input and output in other places.

This category of applications is something that should be simple.  If you want to edit files, you generally want the file editor to be where the files are, because it will involve a lot of reading from and writing to the disk; compared to that, GUI updates and keyboard and mouse inputs are much less sensitive.  I for instance have an NFS volume that holds many of my files attached to this computer, but if I want to unzip files on that machine, I tend to open a remote shell and perform the action local to where the files are, rather than having my desktop wait on network traffic in both directions for every file operation.  Likewise, some tasks that depend on accelerators, such as video editing, are best done when the data and the accelerator are in one place, but that doesn't necessarily require that the user is in that same place.  As long as input and output latency is within acceptable parameters, you can do a lot remotely - and it's only easier if the entire output stack is being run on the machine closest to the output itself, for the same reason.  A lot of system GUI updates, especially, are highly redundant; you can summarize the changes with a few bytes, while a single new image frame may be several kilobytes, and sending every single frame in an easily-reproduced GUI animation may be millions of times more expensive than doing it locally.

Instead of asking what is required to live in a world where this is possible, it is better to ask why we don't.  The legacy of computing simply assumes that everything will be in one place, and comparatively little of our development has been focused on splitting a program up into pieces distributed across a larger system.  We are encountering this need more and more, and so more and more people are getting exposed to it, but until the default way we write programs allows this kind of cross-computer access natively, accessing a program running somewhere else will continue to be a tedious affair.

And to be fair, this "conceptual walkthrough" is more difficult than it sounds, in that it requires tweaks to compilers and programming languages, on top of providing a mechanism to deploy those forked programs on another machine.  Those things happen, and there are experts in those fields, but if anyone is wondering why I haven't written my own code... rewriting program languages is simply a task that's beyond me!  Tasks that are much simpler for the compiler and linker would be an enormous headache to write by hand, but that doesn't mean that it's easy to change how compliers work.

But for the people who actually know how to do all these things... I imagine that if they just knew what they were trying to accomplish, they could change the world.  There is a lot that's possible, but which simply won't happen without the right support and tooling.

Beyond Unix: Typed Data

 So that whole rant last time was all to make this post somewhat easier.  In truth, on its own, that post is a little bit light on content, for all that it was 3500 words.  I know, I do that; sorry.  The point is that this topic is a bit complicated.  In exactly the same way I said at the top of last post, I have a lot of rewrites of this, and even after paring it down, I'm still not sure I'm satisfied.

The point of the last post was, Unix's “Everything is a file” philosophy is all about setting up a system that people can play with.  It's more “everything is the filesystem;” files as we understand them are honestly not all that important per se, which as I said last time, is good because files kind of suck.  The filesystem doesn't store any type information, which means that you have to intuit what they are based on their name, metadata, or contents - specifically the first few bytes.

And it's that bit that I want to talk about changing today - types.

We could, in theory, pack in a ton of type information about every node in the filesystem, information that must be queried and parsed every time you look up data on that node.  And it's not a terrible idea to maintain that capability, so that some files or executables can provide custom type information - but the general goal here is to standardize.  If some given data object's type is standardized, it makes more sense to simply name the relevant data type than to embed the whole thing with each object.  To be able to simply name them, that information needs to be available elsewhere - I suppose it's possible to simply name a standard but give the user and system no information about it, forcing them to rely on some external source or unnamed standards body, but that's a terrible way to build a system.  It would be better if all the information you need exists somewhere in your system (at least, for the types that already have an installed handler of some kind, the types known to the system).

Perhaps all of that type data would be organized into a directory.  A system-wide directory centered around how you understand and interact with data types.  You know, a System API Directory.  Like the MAD SAD.

That little bit of snark, however cathartic, is also completely uninformative; it leaves you asking, “Okay, but what does that actually mean?”  And part of the reason why this blog post has taken so long to get out is that it's complicated.  So complicated that I doubt this post will be the end of this topic, just as the last few weren't.  If I'm really lucky, I'll end up confident that I can at least move on to other things for a while, though I assure you, there are always still things left unsaid.

Everything has a Type

The stated goal of data types being used to describe files, pipes, and network data streams means above all else, having an official, canonical language that describes data structures in terms of fundamental data types, just as is done in low-level programming languages.  Since there are already formal descriptive languages like that, this means either canonizing one or revisiting xkcd 927.  Either way, you have fields in some given order, each described as one or more data blocks of a fixed size and meaning - primitive types plus some low-level data structures like arrays and dictionaries, presumably.  This can be slightly simplified over the type APIs used for programming, because data-in-transit and data-at-rest have fixed size and meaning; programming structures meant to expand or be modified in memory cleanly get serialized, and ambiguities get nailed down.  Data fields in a structure that have conditional uses either are present, or are not.

There are three objectives behind adding type data system-wide.  First is documentation; when something claims to have a given type, it's nice to know what that means (and from a local source), in both human-readable and machine-readable formats.  Second is confidence; type claims should (as in must, shall) be unambiguous, so when two programs, libraries, or data sources make the same claim, they are explicitly promising to be compatible with each other.  Providing that kind of confidence is valuable, or to be a bit more partisan, having a system that lacks that kind of confidence… sort of sucks?  Either way, that leads into the third - integration.  Once you have confidence in what something is supposed to look like, toolchains and workflows can be built to enrich the ecosystem, allowing administration, moderation, modification, and monitoring of the various systems involved.  When data types are no longer mysterious, when their meaning isn't buried deep in arcane documentation, then people can more readily make use of the data instead of making a new format that they can better understand.

Assigning type information to data blocks only serves as a claim that it is of that data type; going further and verifying that it actually is what it says, and validating that the data makes sense, requires either describing the verification process in formal language or having code available to do the type checking.  It's a fair to argue that going that far is unnecessary - but I'm not so sure.  Either there will be standards around verifying data, or once again, every application developer will be obliged to do the same themselves, which I've explicitly argued against several times.

Moreover, it pays to remember that we're talking about distributed systems.  On the one hand, that means that data will commonly be sent out over a network link, introducing transport errors and attack surfaces; a standardized way to verify that the data not only hasn't been modified, but is what it appears to be, would be useful.  On the other hand, in a distributed system, it's only more critical that agents agree on not only what a data type is, on disk, but what it means.  Data types can change, but more than that, small specifics in how they are interpreted can change, whether that's because the old interpretation was erroneous, or because they want to add, remove, or modify some feature, and that affects all data associated with the system.  Similarly, competing software vendors implementing some data type might disagree on the specifics, as the way one part was using the data when they wrote the file, may not match how will will be using the data when they read it.

That means that we need to move past describing the data in terms of its literal contents - though that is also important - and describe the semantic standard, as represented by the library, application, or service that will be making use of it.  In other words, it is important to describe a file type in terms of the library or application that should be used to read it.  That doesn't necessarily mean a type can only be handled by exactly one specific library or application - but the language used to specify the library or application must be an explicit promise of compatibility between adherents.  Only an explicit promise of compatibility allows the system to have a level of confidence high enough that you can build an entire ecosystem on top of it.

Once you have that level of confidence, you reach what is, ultimately, the goal: both files, and file-like data streams, become nothing more than a buffer between the data producer and data consumer, a middleware that can be effectively ignored by applications programmers.  Once you've confirmed the two are speaking the same language, the entire logic train between the function call that writes the data, and the function call that reads it, might as well be a single atomic operation - at least in cases where the time and space between the two events don't matter.  You can pretend, for those intents and purposes, that the producer and consumer are part of the same application, or that they are transferring the data object directly from one memory space to the other directly instead of using complicated middleware to make the transfer.

Which is a funny thing to say, because one topic I've thus far not talked about much on this blog is the hardware aspect of MAD - and one of my ideals, there, is that the network backbone that connects modules together literally just transfers a memory block from one machine to another.  That was true from the beginning, though it makes more sense nowadays, given the assumptions of the ADA application model.  Specifically, under ADA, most of those data transfers go from one part of a distributed application to another part of the same application, which is vastly more sensible than any alternative - but either way, on a programming level, moving data from one hardware module to another can be seen as an atomic operation, at least in an ideal implementation under ideal circumstances.  You make a single call, and either it succeeds or fails, with no third state that the application itself must manage.  Granted, that's more or less how data transfer was always supposed to work, which is why there's not been a lot of reason to talk about it on the blog.  The point, here, being that these backbone data transfers should be a system call, one that the average program gets to just assume will work, or else error out.

Adding types to files, pipes, and sockets, is simply another way to do the same thing.  If you can be sure that data-in-transit or data-at-rest states do not affect the data, then you can abstract them away.  Ideally, when you write a program, you deal with in-memory data types, and if you serialize it to disk or across the network and then deserialize it somewhere or somewhen else, you should come back with the same in-memory data object, as though all the stuff in between had never happened.

The reason why we can't just do that is that there are so many things that can go wrong, and it's generally the programmer's job to handle each and every last one of them, or at least, to stand there and be present while some library or other handles them.  When reading from a file, you need to handle the file not existing, it existing but not being of the correct type, it being of the correct type but malformed (in terms of the data structure itself), and only then can you start to parse its semantics and meaning as they apply to your program.  Those three errors, broadly speaking, can be described as one assumption made by your program: there exists a file of a given type at a given filesystem location.  The same general principle can be said of data-in-transit, with the file-doesn't-exist error being replaced by a null check or similar.

Let's suppose that under MAD, a program's data ingress and egress points must have detached precondition/postcondition blocks, written perhaps with decorators as separate functions instead of part of the ingress function.  Further, let's presume that data types are required to have a validation function, which only checks to ensure the data type isn't malformed.  When data ingress occurs from either a disk or a stream, those three fundamental type errors are checked with a single trivial command: type.validate(data).  If the file doesn't exist or the streamed value is null, it fails; if it claims to be of a different type, it fails; if it has malformed structure, it fails.  The entire rest of the function precondition (if present) is about the meaning of the data - and the main body of the function can be written presuming the precondition has been passed.

Granted, the idea of pre/post conditions is not new and yet many people don't use them, in part because there aren't good language constructs in many programming languages to handle them.  More than that, sometimes you don't know whether something is valid until you've done a decent chunk of the work that the function exists to do in the first place, or it can be difficult to pry things apart.  Unlike some people, however, I'm not insisting that people's programs work a certain way under the hood as a detached ideal - the idea that these condition blocks are separate means that the system can interface with the precondition/postcondition blocks separately from invoking the actual data ingress function - alongside type validation, it becomes part of the system workflow, verifying that incoming data is correct.

And part of the point of that is ensuring the system itself, separate from end-user applications and even from everyday services, has a workflow for validating data.  If you are building a system on what are essentially remote procedure calls, you need some way to detect not only errors (which may be due to physical programs) but also malfeasance and malice.  If the system starts seeing data packets sent over a network link that are of the correct type but routinely fail precondition checks, that sounds like it might be a malicious actor trying to find an exploitable weakness, and that might be a reason for the system to take action against the sender.

It's possible that you could do the same by insisting that data ingress points throw errors, and even with condition blocks that would help.  As with a lot of things in project MAD, I'm not necessarily trying to proscribe a single course of action, though I will make arguments in favor of one over the other.  In this case, encouraging separate condition blocks might have a lot of knock-on benefits.  Those condition blocks might be automatically parsed and made part of the documentation of a function or program, for example, which is only more valuable in a distributed system - not that debugging third-party programs is ever easy even with local access, but understanding why things are failing starts with understanding what a given error actually is.  The condition blocks (or some public parts of them) might have debug information available even when the rest of the program does not, so that you know not only that a precondition failed, but exactly which check in the process triggered a fault.

Putting the argument in its most general form, at some point the function preconditions are simply a language for describing the function itself, which should (as in, it would be nice) be a part of its public API.  And that, getting back to the topic, is part of the semantics of data types - a given data format may cover a wide gamut of possible meanings, so making a guarantee that data producers and consumers are speaking the same language may involve getting further into the details rather than just saying that the correct data type is involved.  Condition blocks that the system can make use of, are just something that the system offers to help programmers and administrators debug errors in distributed systems - and it is an offering, not a demand.  Ultimately, if you tried to demand people use condition blocks, some people would still make blank dummy blocks and then do all their value checking in the ingress function itself, and there would be no way for the system to tell this kind of scofflaw from a more nuanced situation in which handling checks with condition blocks is untenable.

But okay - the point of all that is being able to abstract away data transit and data storage.  That's lovely, but it gets away from the subject I started with, which is standards.  You can certainly standardize types, and even have nuanced validation functions, but the problem is still that people often do not agree on standards.  If you want to describe a standard, you may need a fair bit of information - which can be a problem, because the System API Directory as I've described it is literally part of the filesystem.

Which means that I'm obliged to explain a little bit about that.

Everything has an Implementation

The System API Directory can be understood as containing a couple different things using the same general mechanism.  That mechanism, is that data types are specified as a path string - the path string points into a filesystem tree, and in that tree, you will find a shared library object suitable to be dynamically linked into an application, which provides for the capabilities and handlers of that object.

That's it - that's the main mechanism.  To a certain extent, it's just a replacement for the Linux /lib directory.  However, the /lib directory is organized by source, where the SAD is organized by function.  And while there is a lot of complication in that (much of which I am explicitly and with some great frustration cutting from this blog post), the result for our purposes is that a type path is explicitly a promise that the data object (as it is being passed in from the filesystem or network) will provide certain capabilities to your program, even if that capability is just ensuring the data was passed cleanly from one end to the other.  With the libraries schema that we use today, and with some implementations under the MAD SAD, you can only ask for one specific vendor's version of one specific understanding of the data type, all others being irrelevant and unusable… and that's fine.

But the SAD is designed specifically to enable and empower standards-setting organizations, because it is hierarchical.  Although the tree may have top-level divisions that are simply categorical, once you start talking about a given type or programming interface, if you continue down that tree, you shall find child types or interfaces that claim to be fully compatible, while potentially having more features.  If for example, two GUI vendors put their heads together and agree on a common API which is, itself, sufficient for either of them to run the GUI, they can at that point also create branches of the API that have additional features, compatible with the parent standard but which are not compatible with each other.

In fact, that process comes naturally from the very idea of dynamically linked libraries (it's less clear with data formats specifically).  Library loading matches a function signature against an entry point in the library; all you need is a guarantee that certain functions exist and work a certain way.  If there are extra functions, because the library specifically uses a child standard, then you can safely ignore them just as you ignore parts of any other library that you won't be using.  The only really tricky bit is interoperability between libraries or services, in which case weird assumptions may come into play.

None of that means that application or service providers must hew to a standard at all, much less any given one, but organizing around a standard helps the ecosystem grow.  Even without standards, the SAD helps with discovering libraries that will parse certain files or data blocks - but with standards, suddenly a lot of things become possible, and all specifically because the SAD is designed to list all libraries that implement a specific type.

This discussion is, finally, about device drivers, and the role that the SAD plays in organizing a distributed system.

Towards a Device Model that Makes Sense

I don't know about you, but honestly, I don't understand how device drivers work - me being a computer and OS enthusiast with a CS Bachelor's who has used computers pretty much every day since I was five years old.  I know that vaguely, device drivers register with the core of a system that they are a device driver, and obviously the system forwards some requests to them when things having to do with that device comes up, but if you asked me for specifics, I couldn't give them.  Specifically, I can't help wondering how I would create a device driver for, eg, a cheap little USB dongle that I have programmed to do some specific thing, perhaps acting as a data source or API provider.  When I've looked into it, it seems confusing and troublesome.

Under MAD, a device driver is a service with low-level access to hardware.  That's pretty much it; end of story.  Oh, wait, we were talking about the SAD; not quite end of story, then.

You see, I said that the SAD lists data types, and I implied that data types are all about the data structures that actually make up the type as it rests on disk.  But I've also said that the SAD is full of interfaces in the programming sense - they are promises that, when you load a library and point the library at a data object of this type, the capabilities of that object will be available for you to use.  If that interface is a child of a parent interface, then you can understand that as a guarantee that the capabilities of the parent interface are all fully implemented in the child.  Logically, all of that makes some sense when you are taking about real, extant data that is already finalized and set in stone.

But you can also see how this works if the “data object” is nothing more than a reference to a device.  Loading the library is a promise that certain capabilities will be available to you, and specifying the device as a filesystem object means that you know that specific device will be the one the library interfaces with.  If some specific driver needs very special code in order to function, such that a generic implementation of the driver wouldn't work, then that device is registered as a child type of a parent - it exposes all the capabilities of the expected type, but unless another library claims to support that specific child type, no other driver code should be loaded for that specific device except the one that was designed for it.

Now again, I said that every driver is a service, and you should assume by now that every service has an Application Folder, because MAD/SAD AF is a thing I keep coming back to.  The driver service in question won't presume that some driver for their particular device exists somewhere out there in the aether - the driver service will provide its own copy of the interface library as an export.  Whenever someone else goes looking to see who implements the (parent/generic) type of that device, eg API/HW/Keyboard, that device driver, since it is providing access to the library, will be on the list.

The next question is - where is the file that represents the device?  Well… why not have that be the driver service's own application folder?  If everything has a type, that means that some things that look like and act like folders can also have types.  They can have interfaces, and they can be subject to standards.  You can validate that a folder is of a certain type by looking at its structure to see if it implements the standard.  Indeed, can't a folder have “functions” that are really programs, as long as those programs fit the expected function signature?  Of course, for that, you would need to be able to pass parameters to a program entrypoint in a type-safe way, but that was the point from the beginning, like it was planned.

Now, reverse the last two things I said.  If you can call a program entrypoint as though it were a function, given that we just added type safety measures, can't you export functions like they were executable program files?  Going further, can't some memory objects - real, programmer-type data objects - be represented as folders in the operating system, as long as the system knows how to decode that data type?  Sure, that might be a highly inefficient way to get access to your data, but you could imagine writing a program that did it, so why not have the system capable of doing it automatically, at least for data objects where that idea makes sense?

What if, for example, you have a device that reads air temperature, humidity, and pressure - when the driver is called, those sensors are queried, and then a data object is returned containing those three values.  Suppose that querying this device driver involves reading some node like a file, which will return a tuple containing those three values.  Because of the type data for that  specific file, you know that the first value is temperature, the second humidity, and the third pressure, each having a fixed size and format.  More than that, the system knows these facts about the data object - so if you were to request eg Sensor/Read/Temperature, the system could understand that request without your own program loading the library code.  You would simply request a file, and the output would look like a file - even though it was originally a smaller data chunk within a larger data set.  And you could even do it all without converting the data back and forth from a human readable format, such as strings.

It's obvious I'm getting ahead of myself, and this blog post is already getting long.  I will say that the above example is probably a good place to use another new path selection operator instead of the filesystem descent operator / - but the specifics don't matter for now.  If I go on much longer I worry I'll bake the brains of anyone still reading.

The point is - by making device drivers out to be services that respond to requests and return results in ordinary ways, we encourage people to make device drivers, allowing new, custom devices to integrate more easily with computer systems.  Why would that be an important topic when talking about a modular and distributed system?  Who knows, must be a passing fancy of mine.  But I will say that the idea of getting new, standards-compliant devices devices fills me with glee.  I'm not a neophile - I don't like everything that's new - but I am a technophile.  I want to be able to make my machines do things, and I get excited by having new capabilities.

And that, ultimately, is a goal worthy enough that I don't mind being called mad.

IP: The Worst RPC Stack

 So I was originally going to write on another topic, and you know, I ended up going off on a side rant about something I otherwise wouldn't have really thought about, and I've come to realize that it deserves to be called out on its own.  That topic is: The IP-Port-Protocol stack that we use to communicate with servers is really just about the worst form of remote procedure call that we could have, isn't it?

Now, when I say that, keep in mind I'm an author at heart - I like to make inflammatory statements to get people riled up, especially when the statement is true at heart but knowingly wrong in its particulars.  You can always imagine, or find, worse ways to call a remote procedure, than this.  But I say it this way because most people wouldn't really think of IP:Port addressing as being a remote procedure call at all.  It is how you consume a service, though, which ultimately means that you are calling one well-defined (well, maybe) function from one program in your own program, frequently over a network.

Just on the merits of success, clearly the IP stack has done a lot for us.  But we already expect more from it than it was designed for, and that's before I start talking about the many and varied requirements that I am coming to expect from an RPC mechanism, after exploring Project MAD.

The IP Suite: A Partisan Overview

So when I say “the IP-Port-Protocol stack”, it's not necessarily clear exactly what I mean.  The Internet Protocol suite, which is the basis of literally the entire internet as we know it today, can be understood a lot of ways and with various levels of depth, but I'd like to break the topic up into two pieces: Addressing, and communication.

The part that I'll summarize as Addressing contains numerous mechanisms, some of which are barely relevant to the point I'd like to make, but broadly, they let you translate a name into a series of numbers (which can be summarized as a single number, if you prefer), and then all of the hardware in between you and the destination know what to do to connect you to that number, and connect others to your number.  There is a service, called DNS, which translates friendly website names into an IP number, which is the official canonical number you use to reach that website.  Depending on how complex the service you're trying to reach is, that IP address may actually send you to a service that distributes packets and queries to a number of different handlers, to ensure that nothing gets overwhelmed or to let you use a service more local to yourself instead of a single global server.  (And I may be getting parts of this a little wrong, because it's complicated)

But it's interesting to talk about IP addresses in a local context, because for many home users, IP addresses aren't reserved for a given user or machine; they're temporary.  And even when they aren't temporary, they're mostly arbitrary, meaning that unless someone tells you what an address is supposed to be, you have no context from the number itself.  But it's inconvenient and therefore rare, for local users to set up tools like DNS for their home network, so that you are dealing with descriptive names rather than numbers.

But the inconvenience goes a bit further: all network packets in an IP switched network are targeted not only at a machine, but at a specific port on that machine.  This port is simply another number, and although there are standards suggesting that several well-known ports should be used for specific protocols and applications, there's not actually anything that's stopping you from using the port normally reserved for websites, to do almost anything else you want.  Oh, it'll be inconvenient and confusing if someone tries to access that server from a webpage, but the system itself doesn't care.  This problem cuts both ways: You can use (almost) any port for (almost) anything you want, and you can't tell what a port is actually being used for just by the number in use.  This becomes an even bigger problem when we start trying to manage complex data spaces with IP address hierarchies, but I'll get back to that in a minute.

The second half of my description for the IP suite is “communication.”  Once you've established a data channel to a remote server, you then need both computers to speak the same language across that channel, and this is known as a protocol.  The Internet Protocol itself, of course, is one such language, one that is used to get your message to the right place; but once your message is in the right place, two pieces of software need to agree on how you make requests and how results are returned.  There are several existing protocols, and the most used ones are well-known, but it can still leave you feeling ignorant of how your own machine works.  If you know the IP number and port of a service, but not the right protocol to use, you can do nothing with it - but, as I said, the IP number and port are in some sense arbitrary.  From a certain perspective, the internet is full of billions of holes, and it is your job to cup your hands and yell code phrases into exactly the right hole.  Yell the wrong code phrase or pick the wrong hole, and nothing (or nothing useful) happens.  Pick the right hole, and you'll hear a code phrased yelled back at you from the other end of a pipe.

The end result of all of this, functionally, is that if you have the right combination of IP number, port number, and protocol, you can call one function on a remote machine.  It is, in other words, the worst possible remote procedure call stack.

If that doesn't sound right to you, it's probably because many servers have come to expose a lot of functionality on a single port.  That's all down to the parameters you pass the service when you make a request; the exposed function that you're calling is itself a directory of other functions.  Perhaps, one might argue, that means that you are really exposing far more than a single function, but that argument is flawed.

In fact, generally speaking, it is the job of anything that wants to provide services to invent a new remote procedure call mechanism.  It is your task to take a data block as input and decide what function to call, with what arguments, and if data is returned, what to do with that return value.  It is one of many places in modern computing where the solution to a problem is, force programmers to find their own solutions to that problem.  One significant consequence: every programmer is forced to find their own ways to parse, validate, and verify incoming method parameters from that data block, which provides a massive and porous surface where programmer mistakes can become exploitable program vulnerabilities - and frequently, those vulnerabilities affect general purpose services that are capable of doing a lot of things.

If you want to build a system on top of remote procedure calls, the first thing you must do is not require everyone to reinvent remote procedure calls with every program they write.  It is in this context that I want to talk about how the MOS/ADA subsystems think about remote procedure calls.

RPCs Require Better Communication

The Agentic Distributed Applications model, and the Modular OS concept built on top of it, both talk about listing what remote procedure calls are available.  The list of procedure calls is used for three purposes: one, it tells remote callers what is available, two, it is used to verify that incoming requests are valid, and three, it is used to direct the final result to the function that will be handling the request.  It's worth noting that the IP suite in fact does not tell people what functions are available to be called on a server, and the IP suite itself doesn't verify or validate any incoming requests; if there is a service which can receive the request, that service handles it.

But the MOS/ADA RPC stack is expected to do more than that.  Ultimately, it is the function of the RPC mechanism and not of the individual service's back end, to convert packed data into arguments for the function to be called, and to convert the function's return value and/or error condition into a packed data type for the return trip.  This is known as not requiring programmers to reinvent remote procedure call mechanisms, or if you prefer shorthand, it is known as an RPC mechanism.

In order to do that, the system must have an awareness of data types, which is why data types are an important part of the system directory.  Not only must you be able to convert incoming arguments to the appropriate data type, but you must be able to verify that the incoming data type is an acceptable match for the type expected by the exposed function.  And by having those types be explicitly listed, the calling procedure can do certain checks before sending the request.

(I suppose you could expect every function to take an undefined, fully variable data type as parameter to every function, do no type checking ahead of time, and expect the end programmer to dedicate the first lines of every function to finding out whether or not they were passed the correct arguments, the way for example C programs do with their command line arguments… or you could compartmentalize those checks by having them done in an explicit precondition function, which may be an overridable hook, allowing exported functions to only contain that function's code.  One of those sounds like requiring programmers to reinvent remote procedure calls to me, while the other sounds like having an RPC mechanism.)

Already, we can see incredibly clear distinctions between what I am calling RPC mechanisms, and raw server sockets that get piped straight into application code.  There is a language to RPC calls, and if you want to understand that language, it can be explicitly queried, finding out exactly what the protocol expects and how to format your requests.  Rather than shouting memorized code phrases at one specifically chosen holes among billions in a wall, there is a piece of paper taped on the wall telling you what to say into which pipe to receive what reply.

Of course, someone developing their own RPC mechanism to respond to queries on a raw server socket can do all this… and they can do it separately for every single application they create, because it isn't a defined mechanism, it is a private standard that they have decided on.  It makes far more sense for this mechanism to be standardized and this directory to be machine generated, because most if not all of the information is already there when you compile your program (or run your script, if the language used doesn't get compiled).  As long as you agree on how to translate the information already required to create and present within your program, into this directory listing, it should be relatively simple to implement.

Relatively simple, of course, does not mean easy, or even simple.  There are a lot of questions about types that need definitive answers and canonical standards.  Today, before any of those questions are answered and those standards canonized, it would be relatively difficult and complex.  To put it another way, Project MAD has always, from the very beginning, been about doing the hard work ahead of time so that other people can simply walk in and do the fun parts.  This is known in some circles as creating an operating system, though perhaps not everyone would agree.

I would argue that once you have all of those standards, you can do everything that the IP-Port-Protocol stack can do, but with names instead of numbers so that you can read what you're doing, and with documentation that tells you what's available and how to use it.  A protocol, in other words, that is not only machine readable, but human readable.  One that not only allows communication at runtime, between processes that already speak the same language, but allows the programmer who created a program to communicate with users and other programmers in the future, making explicit their assumptions, needs, and intentions.

The cost in exchange, of course, is complexity and speed.  As with many aspects of Project MAD, that slowdown might be a real problem, but equally, the speed we have now comes with fragility and ignorance.  We have built a lot of very important systems on top of tools that only barely allow us to communicate; we've discovered that we don't need much structure in order to make something like the Internet work.  But as we have wanted to create more powerful and flexible tools, we've chafed at the restrictions.  Better communication is needed.

A Word or Two on Addressing

So I promised to come back to a bit about the Addressing side of the IP-Port-Protocol stack.  As I said at the beginning, this entire topic originated in a blog post about other things, and specifically, I was trying to list all of the problems that Project MAD tries to address with its solutions.  One of those is the confusing morass that is IP addressing within application containers and private networks.

For those not aware, application containers are used to explicitly contain a piece of software and its dependencies, so that if two applications each require different versions of the same library or tool in order to function, they will not accidentally select the other app's incompatible version in place of their own.  They likewise won't share some system configuration bits, which is useful if two apps each require the same embedded tool but configure them differently.  There is another conversation to be had, here, about configuration, but we'll get back to it in a bit.

Suppose for the moment you had a home server farm, with several hardware servers, each running hypervisors that split the system into multiple virtual servers, and each virtual server used IP-based application containers, with some of these containers incorporating their own private IP networks containing multiple internal containers.  It's fair to say that there would be a lot of IP addresses being used to direct traffic on this distributed system: each hardware server's network adapters have their own, each virtual server's network adapters have their own, each container and sub-container has its own.

Part of the very real problem with this setup is that all of these IP addresses are arbitrary, and due to the way their scopes work, it's entirely plausible that if you leave things to automated processes, random containers on different virtual servers may have the same internal IP address.  They shouldn't be visible to each other unless you've done something wrong, and perhaps no part of the distributed system as a whole may malfunction, but as an administrator, programmer, debugger, or power user of the application containers, it is frustrating that what is supposed to be a unique identifier is neither unique nor identifying.  An IP address alone may tell you neither what is at that address, nor can you be sure that it is used exactly once, even within your own, privately operated server room.  And it may not be quite as simple as an amateur would like to change those internal IPs so that everything is unique, especially if those unique IDs are only useful for reading logs and chasing down problems in a complex and distributed application.

Part of what I hoped to guarantee with the MOS System Directory is that across an entire, theoretically infinite distributed system, it is still plausible to uniquely identify every resource, by having explicit nested scopes.  Because systems list their internals, even privately, as long as you can distinguish peers from each other within their own context, you can create a correct, unique, descriptive identifier for every resource - meaning that at every level of context, you can uniquely identify all children of your peers (even ones you have no access to), and if you have permission to explore the peerages going up the stack and outwards as well as down, you can discover your system's parent's long-lost uncle's step-son's cousin's application container's database, and the fully qualified unique identifier for it and for you will tell you exactly what your relationship is and isn't.

At the same time, because the unique identifier is based on nested scopes, you don't need a fully qualified unique ID to describe anything, not unless you share no relationship except having the same system root.  In fact, if you know the scope you're targeting, you can convert a local directory reference into an absolute one, and vice versa.  It's still wise, of course, to use the fully qualified ID once you start going up in scope, in case there are subtleties you are missing, but it's not necessarily technically required.  Once any two resources share a scope, that scope is the highest you need to go in order for the two to identify each other and communicate across the network.  Even if there is a faster way to communicate, the two fully qualified IDs demonstrate the maximum number of scope changes required for one to reach the other.

It may be somewhat crude for me to say, that I believe an addressing protocol should allow you to give, get, and understand the address of that thing.  It may be a bit rude to suggest that the IP protocol's arbitrary identifiers make it difficult to feel like you are in charge of the layout of your system.  It is unquestionably arrogant of me to suggest that I could do better than the technology that united and changed the world.  But what I don't think is arrogant, rude, or crude, is pointing out the flaws in any technology, especially where those flaws have consequences.  Any system built on my ideals will have its own flaws, and I have to trust that some person smarter than me will someday figure out a way to address them... no pun intended.

And IP has its flaws.  All of this is without getting into the limitations of the still-ubiquitous IPv4, which hasn't had enough addresses in a very long time and has felt for a while like it should just get replaced by its successor, IPv6, to prevent some very real, and some other mostly theoretical problems.  Frankly, that's not my problem with it, not as a power user of home networks, nor as the architect of Project MAD.

My problem is that the IP stack as a whole was a very early (starting in the 1970s) way to do remote procedure calls, and all of computing has advanced and changed since then.  Even if you don't like my solutions, look at the general problems we are facing.  We want to combine systems together, tightly, in ways the designers of the internet protocol did not foresee at the time.  If you don't like my formulation of the general case problem that needs to be solved, find a better one.  But ever since I've had this general case problem that needed solving, my mind has been seeking out solutions that fit it.  You can build a distributed system on top of IP using yet another layer of abstraction, and maybe that'll be necessary.  But for addressing parts of a system, there are better ways than depending on arbitrary and not-guaranteed-unique numeric identifiers.

One Last Topic: Distributed Configuration

So I just barely touched on the topic of distributed configuration when talking about application containers, and it's an interesting side topic, one that deserves its own full blog post later, but I believe I've teased that before, and so it's better to have a brief discussion about it, in case it keeps slipping my mind in future posts.

Configuration in a distributed system is tricky, because at any given time, an application is under multiple, potentially conflicting policy and rule domains.  The user running the application may have rules and policies.  The hardware device that the application is running on may have certain specific configuration rules necessary to make it work.  The distributed system may have an administrator that is enforcing certain rules and policies.  The application itself may have multiple default policy sets, for example, one for power users and one for non-technical ones.  And an application may be embedded in another application, and the parent application may have specific rules that it needs the embedded application to follow.

But also, general rules and policies are not the only items of interest.  Anyone who has any claim over an application, such as the hardware, user, administrator, or parent application, may call out that specific application and make an explicit configuration change to it, in order to ensure proper functioning.  A user may wish to start an instance of the application with a special configuration, whether that's part of a script, or a deliberate choice made by an interactive user in a shell.

It is my intention that the Modular OS, and the Agentic Distributed Applications model specifically, has an explicit mechanism for gathering and resolving all configurations present on the system.  The resolution step may be accomplished by a callback hook, that is, an application can choose to apply configuration changes in a specific way or even may choose to misunderstand or ignore configuration changes, but there needs to be a mechanism that presents the application with a list of all configurations, and there ought to be a standard way to resolve any disputes.

Although I'm not quite happy with it yet, the working name for this mechanism is the Standard Environment Variable Evaluator, or STEVE.  STEVE is an important part of the ADA model, and arguably fundamentally necessary for ADA to work properly, just as STEVE makes little sense outside of a distributed context.

This topic, about how best to handle distributed configuration, is one that will be determined, in the end, by people who know the topic better than I.  No matter what I try to say about it now, it is a topic that will get revisited and revised once good intentions meet practical concerns, and as such, feel free to take much of what I say below with a very large grain of salt.  But naively, I believe that there is a hierarchy of default and explicit desired configuration states.  The least important configuration, for example, is the program default built into the code, which is used only if nothing else is configured.  On the other hand, an explicit decision made by the user when starting the program is the highest priority among its peers in the configuration hierarchy.

Beyond that, though, all defaults yield to all explicit configurations, and both defaults and explicit configurations are ordered by how specific the configuration is.  Configurations are less specific if, for example, they refer to an application category broadly, like Start all windows maximized.  More specific would be a configuration referencing a specific application in general, then a specific application when run by a particular user, and then a specific instance of an application when run by that user, such as when the application is passed a configuration parameter when it is launched.

But there is a major exception to this rule.  One of the things that came out of Project MAD is that, in a distributed system, every component has the ability and authority to refuse service--indeed, it's frankly the only control you have in a distributed system--and as such, everything that has dominion over an application, such as the hardware it's operating on, the user, or an administrator, can kill a misbehaving application if it does not follow policy.  Consequently, when we talk about configuration, there is a distinction between desires and rules.  A program caught not following the rules may be killed, but a program that merely disrespects the desires of others will not be.

In the hierarchy of configuration options, then, any applicable rule should take precedence over a stated desire.  If, say, the Administrator refuses to allow full-screen applications, and the user explicitly tries to start an application in full-screen mode, your options when parsing the configuration are clear: do you allow the user to do something that may attract the administrator's ire, possibly causing the application to be forcefully quit, or do you prevent the user from doing what they have explicitly said they want to do?

Of course, it's even more complicated than that, because we're talking about a distributed system, which may have multiple competing sources for various software and hardware services.  Suppose for example you have your private phone connected to a desktop machine, in order to use the desktop's monitor, CPU, GPU, and input devices to play games.  The system administrator for that desktop machine forbids full-screen applications, but your phone is not subject to that system administrator; it is completely under your control.  Your phone, however, may not be configured to not display desktop-style applications on its own screen, and for good reason - it may be too small, too low-resolution, and in the wrong orientation, and the processors may be lackluster and prone to overheating.  In that context, if you want to launch a full-screen application, do you try to launch it on the desktop screen where that isn't allowed, on the phone screen that it isn't suited to, or neither?

My thoughts on this matter are incomplete, but there is one thing that I am certain of: it should not be down to a programmer to find and read configuration options across a distributed system, nor to determine what the nominal hierarchy is.  An operating system for a distributed system should find and request all relevant configuration options and present them to the application for parsing.  This needs to be done before a distributed application selects what hardware it is going to use, because the rules and policies governing the hardware will be used to determine which hardware is selected.

A bonus that comes from the system handling these matters, is that several lists of configurations can be made explicit: the total configuration on a system with sources, the total configuration as applied to a specific application, and depending on the configuration mechanism, you may get a list of what configuration options the application itself requests, including requests from all internal libraries and embedded utilities, which tells you what configuration changes you can make to influence its behavior.  You can see a list of what configuration changes a user has made, and which of them have and have not been actually queried by applications in the last day, week, month, or year--indicating that they may be useless, malformed, or out of date.

But likewise, a system-wide configuration parser can validate a list of configuration options, providing a list of warnings and errors that indicate some options are malformed, out of date, or nonexistent.  It can simulate how configuration would be applied if you used it on a piece of hardware, on a system you're connected to.  It can tell you how Administrator-applied policies have affected you and your programs, and how your own configurations have affected you.  It can tell you when a configuration option made explicitly simply matches the default.  It can tell you when rules and configuration options have changed, in ways that affect you.

These kind of features are only possible in a system where handling configuration is a standard process.  As with the other mechanisms I've described in this blog post, the current solution is “Bring your own,” or in other words, there is no configuration mechanism.  Even centralized systems like the Windows Registry and Windows Active Directory don't give you all the information you could possibly want, and too many applications have completely internal and private configuration files that will never be explained to anyone, meaning the applications basically cannot be configured even though they were designed to be.

These are all systems that solved specific cases of the problem when they needed to, but nobody tried to solve the problem in the general case.  Perhaps if they had tried, it would have worn out its welcome wherever and whenever application developers wanted something different. I believe that it is laudable to at least examine the problem's general case and search for solutions - but more than that, in the case of a distributed system, I think it will be necessary.  To not have a standard solution to this problem will put undue burden on application developers.

Wrapping Up

I'll consider it fair if I get feedback on a lot of the points I've raised in this blog, not merely this post, as being incorrect on a technical level.  Others know more than me about a lot of the things I've discussed.  But the point of view of this blog, and the reason why it's worth writing, is that I see challenges ahead that we're not ready for.  Building a system on top of remote procedure calls requires a lot, and our current model of clients and servers is a poor excuse for that.  Arguably, the client-server model was never meant to stand in for general RPC mechanisms - but in the modern day, it's been a hammer tasked with far more than driving in and pulling out nails.

I feel like the various web services and web APIs prove this point as well as anything.  Many of these are utilizing the tools meant for web pages as a middleware to enable general remote procedure calls over the internet, and as such, they are awkward, insecure, and suffer from a lot of problems whose primary source is programmers needing to reinvent various wheels.  Tasks that should be mechanical underpinnings of a system are being hacked together.  That's before talking about using webpages as a display in non-web contexts because standards between various operating systems are massively inconsistent, or talking about webpages being the best answer we have for a remote server presenting a local interface to the user.

The tools are already stretched.  Utilizing webpages on nonstandard ports is inconvenient, even if it's the best way to get multiple separate server applications to cooperate, rather than integrating them into a single app.  Trying to understand the flow of data in a container stack is a headache, but it's still easier than trying to manage dependencies on a system not built to properly handle dependencies.  Trying to understand what is going wrong when a client and server communicate over a network can be maddening, but it's the best way we have to make use of another machine's computing resources in parallel with your own machine's.

If you want to merge machines and utilize one's capabilities from another, you need a better mechanism.  If you don't like mine, by all means, show me up.  Do better.  I'd love to hear about it.

Remote Procedure Calls under MOS/ADA

I have cast some aspersions against the existing format of remote procedure calls on this blog, and it would be unkind of me not to admit that I have never actually used remote procedure call mechanisms myself.  I know that RPCs do not meet my requirements, because my requirements are insanely high and weirdly specific, but I can admit I don't really know a whole lot about the RPC mechanism in existing operating systems.

Instead of trying to talk about existing RPC mechanisms, let's talk about calling and handling remote procedures under Project MAD.

MAD Procedures

I've already said that Agentic Distributed Applications expose a list of procedures available to be called remotely, both procedures only intended for internal use and those more publicly exported, and I've said that applications are supposed to deploy an Agent onto remote nodes in order to make use of resources.  I've heavily implied that resources are, therefore, not intended to be consumed except by Agents, I'm happy to mostly leave that there, with an asterisk saying that some very basic resources, such as sensors, might be accessible remotely.

If that sounds unrelated, it's not quite.  The directory of available API calls that the ADA server builds, must have a built-in distinction for what calls can be made remotely, and what calls are consumable only by local Agents.  After all, it makes no sense to deploy Agents to consume your own application's internal API, which is itself provided by your own Agent already; the internal API is one of possibly many examples of remote APIs.  When you need to decide where something should fall, there are a number of questions to ask, and some more obvious than others.  For example, a purely remote API call only makes sense if either:

• The call is stateless; it does not store data or set internal state that will affect future calls; generally, if multiple calls are made by different applications, there is no way for them to interfere with one another except in reasonable, predictable ways.
• Or, the procedure manages its internal state in such a way that no combination of remote calls can have unintended consequences, for example, internally managing its memory such that different callers have different memory dedicated to them.

There are interesting side topics here, such as the need for a distributed configuration mechanism, but that's not what I'd like to focus on today.

The typical example I have in my mind when thinking about remote calls, and why Agents are necessary, is the GUI.  If you want to create a window to display information to the user, you are asking the graphics subsystem to set aside a bunch of memory for your application to manage.  Ultimately, this memory needs to be co-managed by the Application and the graphics subsystem; it can't be purely under the Application's control, because the back-end needs to read that data to actually display it, and may need to take control over the memory if something happens to the Application.  But, it can't be purely under the graphics subsystem's control, because the Application needs to freely make a lot of fast modifications to the memory, possibly not using the subsystem's helper functions at all to do so.

Calls like this should never be made without an Agent, because the subsystem needs something to take responsibility for that reserved memory.  If you aren't counting on Agents, then you need some other mechanism to notify the graphics subsystem when an Application quits or crashes, or the hardware that the Application was running on gets disconnected.  Even if you had very fast mechanisms for accessing the video memory remotely, and a dedicated network bus that lets you pipe in uncompressed video data straight into the buffers without causing network congestion, the need to manage the memory itself, and assign responsibility for it, remains.

Contrast that, however, with something like an atomic sensor read, setting a hardware light to on or off, or even creating a simple popup notification in the GUI.  Even when these actions have side effects, they aren't the kind of side effects that explicitly need management.  You could draw a distinction, say, if you wanted to be the only Application in control of that hardware light, and to lock out all other users, or if you wanted a modal dialog box in your GUI that the user must interact with, and which the application needs feedback from; those would require management and therefore an Agent.  But if you simply wish to flip a light on or off, or set its color value or intensity, or if you merely wish to put a bit of text in front of the user, that does not.  It may require authorization; you may need confirmation that the application is allowed to do these things.  But you don't necessarily need an ongoing presence.

But even when talking about actual remote procedure calls, my expectations of a remote procedure call mechanism are higher than average.  Some of the reasons why are better explained later; for instance, it is my hope that the MOS system directory is also a directory of types, and that type data is used in detailing APIs, and can be used for verification of the incoming parameters.  That one-sentence summary hides a number of other thoughts and details, but it's better not to get into them right this moment.

Decentralized Procedure Calls

A more important thing to say about MOS/ADA remote procedure calls is that the distributed and decentralized nature of the system requires some explicit handling.  Let's say that you have an application that uses a machine learning chip to monitor an external camera, and when the ML chip detects that a bird is in front of the camera, it takes a picture and puts it on your computer monitor.  For our purposes, assume that all of these are on separate hardware, and therefore remote to each other.  There are numerous data streams involved in this process; most notably, from the camera to the ML chip and from the camera to the display.  However, the trigger that puts the picture on-screen doesn't come from the camera, it comes from the ML chip.

There are a couple valid ways for the ML chip to display an image from the camera on the monitor; the ML chip could store the image frame and pass it to the display if it meets the requirements, or the ML chip could ask the display Agent to go and fetch an image frame, or the ML chip could ask the camera Agent to send a frame to the display.  But I will argue that the “correct” form of this request sends two requests: one to the camera, asking it to send a frame of data to the display, and one to the display, asking it to be ready to display the frame of data it is about to receive.  This solution keeps the ML chip from needing to keep frames, but it also minimizes the overall wait, by having two operations begin in parallel.  The display process and the image send process both start at roughly the same time, and by the time the image data makes it to the display, the display is ready to handle it.

But perhaps the most interesting reason to argue for this interpretation of inter-Agent communication, is that you can imagine writing the request in a single line of highly readable code:

• video.display( camera.capture() )

This line makes intuitive sense to a programmer; you are sending the camera data to the video display.  But what is perhaps most important is what this single line of code says about how remote procedure calls under the MOS/ADA specification should work.

Specifically, in all remote procedure calls, you eventually need to send raw data as parameters to a remote function.  It is our common experience in programming that you are only able to access data that is directly under your control, but any distributed application is going to need to refer to data that exists somewhere else, when making a declarative statement of intent.  In this case, from the ML hardware node, we are calling a function on one remote hardware node, passing as parameter data from another hardware node.  All of this data is under the application's purview; there is no problem with the scope of the data involved, no reason why this request should be impossible.  But with existing libraries and programming languages, it may be a very awkward three-way handshake to try to coordinate.

Rather than saying “This is the ideal form of this data stream,” I am saying that if we work hard to enable this kind of decentralized data command as a basic programming language feature, it will empower programmers to more easily perform operations in parallel.  And programming in parallel is somewhat of a problem; computers have run multiple programs at once for, what, 35, 40 years?  More?  And yet many programmers still think, fundamentally, in single-threaded terms.  The biggest exception I can find to this rule is actually webpage programming, because the very nature of web programming depends on remote data requests.

But instead of that, and not coincidentally, the compound statement I wrote above reads more like a shell script, where two applications are both started, and input from one is piped to the other.  You could easily see how the syntax of this compound remote procedure call can be extended to run dozens of operations in parallel - by simply passing more parameters to the destination function, each sourced from a different Agent.  If the chain of logic got more complicated, for example, the camera image was passed through a filter on another module, perhaps to remove all image data except the bird (with another ML algorithm, perhaps on a different ML hardware node, or a different Agent on the same hardware node), you could see how the statement would remain a perfectly valid, single-line statement:

• video.display( ml_filter.isolate_bird( camera.capture() ) )

And if this seems a bit redundant for a single image frame, consider that if we were setting up a video stream instead, this same chain of logic becomes a workflow in which each hardware node does its own job and nothing more.  Ideally, once this workflow is set up, it can operate at the fastest possible speed, making full use of the parallel architecture to get things done without overwhelming any single piece of hardware.

Looking Under the Hood

In the meantime, we are left with a question that people may find uncomfortable: supposing that this chain of logic is acceptable, how exactly do we phrase this remote procedure call in low-level code?  Even just using the earlier example, without the filter, it can seem complicated.  The video.display function needs to know it will be waiting for a data block parameter, one not included in the RPC call that starts the function.  And the camera needs to know that the return value of the incoming RPC request will not be sent back to the Agent that made the request, but sent to another Agent specifically to be used as part of a function call that it did not initiate.

The answer I have currently is that the MOS/ADA resource directory, which I said is used to expose API calls, can also expose chunks of memory for reading and writing--and specific to this use of that mechanism, there needs to be a syntax for reserving memory for temporary variables.  There's a lot to unpack there, I admit; I was under the impression that I had already suggested in my first post that the ADA indexed memory, but I look back and don't see it (I did refer to ‘application resources’, but I wasn't explicit), so it's worth taking a moment to justify that.

Recall that the ADA's exposed API is handled primarily by the ADA server, and that all requests come with a sending application, agent, and user; as such, when I suggest that raw memory can be read or written, we are not talking about leaving memory access unprotected.  There are many processes in a distributed system that will be both data-heavy and latency-sensitive, such as parsing video, and as such, it's generally best to have some mechanism to simply transfer memory, because if there isn't a built-in mechanism, applications programmers will simply write wrappers to do the same thing, leading to code that does nothing beyond circumventing the limitations of the system.  And while those explicit wrappers are good in some circumstances, especially when they add verification and validation or similar checks, that's not a good justification for having no direct memory access mechanism.

Equally, having the ADA server provide direct access to application memory is good for debuggers, administrators, and power users.  Debugging a distributed application is a nasty business; you don't have all of your memory in the same scope all at once, meaning that if the system doesn't have some mechanism to expose any part of the application's memory on demand, you will with only the uncomfortable dance familiar to all programmers, where you insert code randomly to check, log, or output values.  While programmers will inevitably do that anyway, it behooves us to have a proper solution to the problem.

Likewise, a power user may take a relatively standard application and want to get specific data out of it.  An easy example that comes to mind are video games; while there are many data values the game developer would not want to give you access to, it would not be hard to have, for instance, your player health (and maximum health) exposed for reading (but obviously not writing), and a power user could create a third-party application to display your health bar on a separate display, or as a video overlay that can be moved around the screen.  As trivial as that may sound, players may prefer their data in a different format (bars, dials, or raw numbers) than the game designer intended.  Likewise, access to application internals can be good for accessibility, turning what would normally be video into sound, or sound into text, or text into braille, without necessarily receiving the developer's explicit permission or counting on them providing software hooks.

Coming back to our remote procedure calls, however, it makes a certain sense to be able to pass ADA URI paths as parameters to functions, or as the destination for return values.  When we are talking about setting up a data workflow between two remote targets, it makes sense to have a syntax specific to creating temporary names, unambiguously describing a location that will await a single very specific bit of incoming data, possibly only from a specific source.  In our example case, the target display sets aside memory with that ID and waits for all parameters to be received; the camera source sends its procedure return value to that ID on the target Agent.  And once the data is received, the display function goes about its merry business.

Of course, in reality, things are more complicated than that.

If you create a mechanism for setting aside memory, that can be exploited maliciously for a denial of service attack.  Presuming for now that the sender of a request can at least be verified, this denial of service can be mitigated by detecting unreasonable requests and having the requester, not the data sender, be blacklisted.  (Unless it can be guaranteed, it should not be assumed, that the two are part of the same application, but I would assume that when a malicious actor is detected, the entire application gets blacklisted, for obvious reasons.  For some multi-user systems, it may be the entire user that gets blacklisted, and an Administrator will be notified.)  But what if the request gets through the sender's entire workflow before the memory gets set aside at the receiver, due to network congestion or some other slowdown factor?  Does the sender's data packet get lost in transit?  Does the sender get treated like a malicious actor if it requests to write into a data block that doesn't exist?

The answer to that, at least for now, is that these requests are handled by the ADA server on all three sides, meaning we have the opportunity and obligation to handle these exceptions as a matter of policy.  For example, the data sender may not attempt to send data out unless the receiver acknowledges that the named data opening exists, and the data may not even be generated by the sender until the request is acknowledged at the receiver (because again, generating data with a malicious request can create a denial of service exploit).  And part of the point of having Agents being involved on all three sides, is that if there is an error due to network congestion or something similar, that error should percolate up through the chain of logic, distributed across the system, until it is handled by all relevant Agents.

Suppose for example that the requester's data packet to the receiver disappears, and therefore, when the sender attempts to send data to the receiver, there is no such named temporary variable to write to.  This will cause an error at the sender, but the sender is only an intermediary; it may be logged there, but the error condition needs to be reported to the requester, who actually set up the chain of events.  And because the sender's operation fails, that will generate its own error, and the error stack is sent back to the requester.  It is there that the error gets its full context: the receiver was not ready for the sender's data packet, despite the requester having definitively sent out both.  If this happens repeatedly, there may be a larger issue to investigate, and may require the intervention of a programmer or administrator.

This Isn't Everything

There are other interesting topics in this kind of distributed, remote API environment.  For example, callbacks and event delegates are a common mechanism in event-driven software models such as the GUI, and while these callbacks should be handled entirely by Agents (again, because it involves managing relationships, just as with memory as described above), it is worth having a discussion about event subscription in a distributed system.  And I do mean a discussion; I'm not sure I could put together another long-winded rant on the topic, but I'm sure there are pieces to the question that are more complicated than I know.  We are, after all, talking about users and applications and Agents, with possibly all of these being different between the event source and the subscribers.  There are matters of security, efficiency, and best practices that I may not be aware of.

Likewise, as I teased before, there is a question of distributed configuration, and although I'll go into that more in another post, I will say that distributed configuration is meant to be handled with an explicit mechanism in the MOS/ADA schema.  That configuration needs to account for local hardware module policy, system administrator policy, user policy, application default configuration, stored configuration changes in the user files, temporary values specific to the user login session and/or parent application session (aka, the “environment” as understood in existing operating systems), as well as values specific to the user application session (which can be understood as application-local variables, except that they represent and are best described as configuration).  Having an explicit mechanism to gather and resolve contradictions in this wide field of configuration sources only makes sense, and the general goal is to be able to simply query the configuration and receive an answer, no matter where it comes from.

There are also probably other questions I wouldn't have any immediate knowledge of or answer for.  Perhaps complications that arise where hardware APIs interface with the MOS/ADA APIs.  Perhaps complications where code libraries and embedded applications interface with the parent application, or where those embedded functions and parent application functions share (or arguably compete for) resources on remote nodes.  There are doubtless complications when it comes to truly confirming that applications are operating under the auspices of a user, or that an Agent is truly what it claims to be, running the code it is claiming to run.

The general Modular OS Agentic Distributed Applications model has a lot of nuances and complexity that I have no right to try to decide.  I can only, and have only, sketched out the broad strokes of this system.  I think I have answers, but only testing and implementation will determine how right, or wrong, I am.  All I can really say for sure, is that if you are trying to build an entire operating system on top of remote procedure calls, you have a lot of work to do in order to ensure that the system is powerful, stable, and easy for programmers to understand and manage.  And… forgive me if I may sully the name of remote procedure calls as they currently exist, but I really think it's going to take a lot more.

Most likely, the full list of what it does take won't begin to take shape until people who know these systems a lot better than me have their say.