Generic Runtime Linking

 My last post, about a Distributed Application Model, was an attempt to separate one of several good ideas buried in Project MAD in hopes of making them more accessible to others.  This post is similar; in many ways it is reframing what I said in the last time, with a specific focus on explaining the core mechanism and how it flavors everything else that is going on in the MAD OS and MAD/ADA system.

We are talking, today, about dynamic linking under Project MAD, or if you prefer, MAD/Libs.  And to begin, let's talk about how this normally works, as I understand it.

Preface: Standard Runtime Linking

Programs (let's leave aside scripts for now) are normally written in a human-readable language and then translated into machine code in a process called compiling; frequently, there will be multiple, separate bits of machine code generated when you compile a program, as programmers like to split things up into logical segments.  Because the machine code is still structurally similar to the layout of the human-readable program, you can still recognize that in the resulting machine code, for example, a particular function starts here, and a particular bit of data is there.  An index of these facts about the machine code - where things are and what each of them is - is included with this compiled object.

When one of these compiled objects wants to reference another, you must necessarily be very precise and specific, in two parts: you must know what other object file contains the thing you want, and then know how to find that specific thing among all the other things inside of that file.  During the process of compilation, it's easy to keep track of this all automatically; a mapping is made between an identifier and where exactly to find it, in both human-readable and machine code forms.  Thus if you want to combine several objects together, you simply go through the object's dependencies, and find which file to look at, and when you combine the objects together, you ensure that the function call that was previously left as a dependency is translated into an actual function call within the final object.  This process is known as linking.

There is a form of linking at compile-time which depends on code that the user themselves did not create (which is called a library); this process is known as static linking, and it is, ultimately, the same as it is when you link together objects that the user created, except that the library isn't something that you just created and so you have to know where it is.  The process of specifying and finding these libraries is standard, and the details get into operating system and filesystem standards; common libraries are usually stored in standard places, and you only need to reference them unambiguously.

If, at compile time, the library you say you need cannot be found, the compiler makes a fuss and everything stops.  Mostly, the list of these library files is stored in configuration files along with all the human-readable program source files, and the linker program just checks with the OS to see if they can find the files you want.  And of course, if someone else wants to compile your program (assuming you share the source files), they can reference this list and go looking for exactly the libraries they need to compile the program, but assuming they have or get all those libraries, they will resolve all the dependencies your program has and come away with a finished, working executable.

A dynamic or run-time linking model alters this last bit.  It allows you to create an executable that still has dependencies; the executable is considered “finished”, but when you go to run it, the process is interrupted until the system can find a copy of the libraries it depends on and link them in.  Because this process happens on computers that the programmer doesn't control, if anything goes wrong at this stage, it can be a little harder to figure out and fix, but we generally have the process figured out nowadays.  (In theory.  Mostly.  Kind of.)

Other than that, dynamic linking is only slightly different than static linking.  Because these links are designed to be made when the program is loaded rather than having the canonical on-disk file altered, they are often done with function pointers - which treat the concept of where to find the function as data which can be modified, as opposed to a static linking model where these offsets may be metaphorically set in stone.  Even then, though, the system must find and load the file containing those functions, before it's possible to update those function pointers so that they will work correctly for the program that depends on them.

There are a few added complications, but by and large, this is representative of the process that we call dynamic linking, and to summarize one last time, it requires you to keep track of identifiers (as originally written in human-readable language) and what file to look for them in, and it requires those files to provide a listing that translates the identifiers into pointers for functions and data which can be stuck into the program to allow it to run.  Ultimately, this is what it takes for a program to call a compiled function stored in another file; to put it in a word, it's just recordkeeping.

Problems With the Extant Model

The model as we use it today assumes that linking is and must indefinitely remain highly specific.  It simply does not do for someone to link any other object file in place of the requested one; it is assumed that you are referencing some complex bit of code that works on every client machine in exactly the same way as it worked on the developer's machine, and nothing else will do.  This is not without reason; it's really problematic for a program to work differently on some random computer as it works for the programmer.

The problem is, because the linking process is very specific, operating systems and programming tools lack the features that would be necessary to facilitate generic linking.  The concept does appear in scripting languages, where linking isn't a thing, and there are concepts in programming that do the same thing within a given project (using dynamic function and data tables, again), but it remains a presumption that everything will be done exactly the way the programmer originally intended - and again, this is not without reason.

There problem is, this non-generic structure is remarkably flimsy.  Too many things are assumed and never tested; there is a lack of formal structure that allows you to describe things in unambiguous, but still flexible, ways.  As a result, there are tons and tons of small, specific, things, many of which depend on other small specific things, and if any of those small specific things break, much larger things that you would not expect to be fragile may also break.

Anyone who uses Linux (and related systems) to any degree has encountered situations where they go to update their systems and receive a deluge of small updates.  If one were to examine many of these updates, it can be alarming just how many are redundant; there are libraries that do the same or similar things but have different interfaces, libraries that do the same thing in different programming languages (especially scripts, which are wholly incompatible with one another), thin front-end executables that expose library functionality to shells (usually exposing only one specific library per executable), services that just sit around exposing a library's functionality, and libraries that compete with one another and may have some incentive to be incompatible.

What makes this system fragile is that none of these copies or similar functions can be easily swapped out.  If a given library used by a given program becomes a problem for whatever reason, everything that depends on that specific library fails, because there is no easy way to transition from using that library to using an equivalent library.  You can roll back to an older version… but that's only a patchwork, a temporary fix.

Why are they all incompatible?  The easy answer is because nothing is standardized, and that's correct, but I will say it differently.  There is no standard language for describing the dependencies of a program.  Enumerating them, yes; any given executable will tell you that it depends on function X in library Y.  But that enumeration of the dependencies does not describe anything.

So long as we cannot describe our dependencies, it is impossible to offer a generic alternative.

A Standard API Is A Generic Dependency

It's probably clear to existing readers of my blog that I have been describing for a bit, the idea of standardizing APIs.  Anyone who is new will come to know this concept as the System API Directory, also known as the MAD/SAD subproject.  But what specifically am I advocating for?

Suppose that your program wants to play a sound file; suppose that sound file is provided by the user, and you do not know in advance how the sound is encoded.  When I talk about describing a dependency, I mean that your program wants to be able to say, “I want to link to a library that will play a sound, as it was stored in a file.”  There are extant libraries that provide programmers a “one-stop shop” for various sound file formats, but depending on such a library is not the same as describing the intent of your program.  You remain dependent on that one specific thing, which itself has many additional dependencies, some of which your own program will never use.

Just as your program cannot be linked with just any library, the library was not written to be used in just any arbitrary way.  The library was designed to be used some specific way, and that specific way becomes the structure of your program.  Once your program is compiled for one library, there is no going back; even if you had another library with identically named functions, it is impossible to know for certain that they are compatible.  (Technically, yes, you could try to fool the linker, and perhaps even succeed - but it is not how the system is designed.)

When I talk about a standard API, I am talking about something that exists to guide the library programmer as much or more than it guides the applications programmer that will depend on it.  It does more than simply telling the library programmer what names to call their functions, and the applications programmer what the function name to call is.  It informs the structure of both projects; it describes when memory is allocated and when it is released, what guarantees can be made, what may or may not be taken for granted, and how errors should be handled.  Only when both sides agree on these kind of terms can a specific library be replaced with a generic alternative.

The language of APIs was always meant to describe exactly these sorts of problems, but to date it has largely been left to creators to decide what's in them.  And the point I'd like to make right now, is that until standards exist, libraries cannot be tested against and held to those standards.  And so long as libraries cannot be held to standards, you cannot use them generically, only in the highly specific ways they were designed to be used.

When you look at the state of code libraries, you will see that they must, will, and have created their own ad hoc standards for almost every task.  What is perhaps most distressing, however, is how commonly these ad hoc standards are literally the first thing that came to the creator's mind.  Once the library has been written, and applications linked against it, it becomes painful to change the API and force all the applications using it to rewrite their own code.  Even before it sees any kind of widespread usage, altering the API may mean making large structural changes to how the library was originally laid out.  If there is no impetus to do so, it is most likely that the programmers never will, and whatever form the idea first had, it will continue to have indefinitely.

I choose to place the blame for the “fragility” of non-generic interfaces and linking schemes on this problem: too many have unique and bespoke interfaces, with no agreement and no impetus to agree on what those interfaces should look like.  Programs that depend on some arbitrary set of libraries may find that in different places in their code, the same task is performed by different sub-dependencies, each working slightly differently, and those differences may require different structure around each function call.

I do not intend to disparage individuality in programming methodology; having multiple ways of doing a similar task is valuable, and arguably critical for long-term success.  However, having these multiple methods be fundamentally incompatible, especially where it need not be so, is problematic.  Indeed, I have long thought that one of the benefits of this kind of generic linking is giving users (not merely programmers) the ability to swap in a more optimized version of some algorithm when it is discovered, or to swap in a more transparent version if debugging or monitoring a program.  (There is a security argument to be had here at another time.)

I am also not necessarily suggesting that there be one ultimate and irrevocable interface for some given class of problems moving forward.  Rather, I am talking about deciding upon some minimum reasonable standard which defines the very concept of utilizing a technique, algorithm, process, resource, or device.  For sounds, for example, this minimum standard may include playing the file, pausing and stopping, playing repeatedly, getting or setting the current playback position, and changing the volume.  This list by no means describes everything you may want to do with a sound file, but it is a fair minimum standard (and I would gladly listen to arguments for expanding or shrinking it).

Of course, going beyond the minimum is a matter which brings its own concerns.

Expanding the Language of APIs

It was never my intent, and it remains not my intent, to claim to know how best to organize APIs, or to lay down firm standards without discussion.  As such I hope you will take the following section as informal, tentative, and conceptual.

I have talked in the past about the MAD System API Directory as containing a hierarchical tree of types, from most generic to most specific, and to this concept I still hold.  The nature of this kind of tree makes it technically possible that all possible interfaces can be categorized into rough categories, but this alone may not prove good enough for the task of describing a large, diverse, and robust system.

Originally, I envisioned that the first few layers of this tree were broad categorizations, and once you pass a certain point, then programmers are free to create various child standards with whatever additions to the parent standard they wish to add.  This has the problem, however, of not providing granular extensions to an interface; part of the intent behind the broad categorization is to establish simple baseline APIs that allow programmers to avoid interacting with unused features, for example when reading documentation or when experimenting with the functions of a library.

If you don't have some method for granular extensions, then you are obliged to create complex child standards almost immediately.  For audio files, for example, one program might have a specific need to pitch-shift an audio file (for example, to tune a musical instrument sample to different pitches), but otherwise, you might only use basic capabilities of an audio playback library.  Another program might need to cleanly loop audio, potentially with defined entry and exit audio segments (for example background music, with an intro and coda).  If these functions are not part of the baseline, in a child-only model, it stands to reason that someone will immediately create a “everything and the kitchen sink” child API that includes both of these features and tons more.  Worse, two similar but unequal child APIs may be mostly but not completely compatible, and there is no language to describe their intersection or extremities.

Although my experience with API design is minimal, I cannot help but think that the language of API design itself needs to be extended.  The above problem, for example, seems like it is begging for a concept of “features” which describe granular additions to an API; a child API may provide, or a program may require, some set of those features, but more to the point, the baseline API standard can be read and understood absent technical discussion of advanced features.  Whether or not features in this sense are the right way to accomplish this, I believe that finding some solution to this problem is laudable.

I have other similar thoughts - enough that it would complicate this section and blog post.  Some are already discussed elsewhere on this site, others will be described at some point.

But to summarize this for right now, remember that we are talking about making libraries into something predictable and testable.  That's all that's really required for generic linking; as long as you know what will happen when your program calls a function, then you can move away from depending only and very specifically on the same file from the same vendor as you originally designed the program to use.

But for all that I've talked so far about the problems with what we have, I haven't talked much (in this post at least) about the benefits of a modular, genericized library structure.

I Did Call It MADLibs

I talked in the last post about making devices accessible using inter-process communication and remote procedure calls.  Without rehashing (or if I'm being honest, rereading) that whole argument, I'd like to point out that the idea of generic linking fits directly into the idea that devices (in the OS sense, which may include things like files) have inherent capabilities.

Generic linking suggests that anyone writing library code knows how to expose functionality in such a way that if a program depends on some specific capability (in the API sense), it can know whether your library provides that specific capability.  Devices necessarily have some kind of driver, even boring devices like files; the driver describes the capabilities of that device.  Thus it stands to reason that a library should be able to make the claim, that it provides a device specific capability, when paired with some device of that type.

At this point, we have returned to this post about assigning types to every filesystem object.  And while that kinda is the point, I'd like to phrase it a certain way right now:

Libraries which provide device-specific capabilities are verbs that act upon a noun.  Any given program may start with a specific verb and allow the users to fill in a noun, but you can also create a list of all device-specific capabilities that anyone claims to offer for a specific class of device.  In other words, a highly generic program such as a user shell (file browser, etc) can start with a noun and allow the user to select any appropriate verb, provided by any library that is available on the system.  The user, of course, has a lot of files and devices available to them; they have a lot of nouns, each with a comprehensive list of verbs.

I said in my last post that “it may surprise people who don't look at how OSes work that an average programmer probably doesn't know how to get a list of all devices present on a given machine, can't tell what a random device is capable of, and can't figure out how to control that device.”  This is the culmination of that indictment of modern computing: A Madlib-style system, mine or otherwise, that allows you to cultivate a list of verbs that work with nouns and nouns that work with verbs is absolutely possible, it just doesn't exist in systems as we have them today.  Certainly not in the generic case, where you expect the same methodology to work as well with services and hardware devices as it does with documents and media files; the “file associations” structure that modern operating systems use is primitive in comparison.

It is also important that the Madlib-style system provides a rigorous way for the system itself to know how to provide a given capability for a given device.  Remember that Project MAD is distributed; if you want to consume a capability on a remote piece of hardware, you need to know what, if anything, you need to bring with you when you send a program agent out to consume that capability, and you need to know what to expect when you get there.  If for example a remote node has a camera, and some nearby tertiary node contains a library that knows can provide a specific capability for that camera, you may need to copy the library over to the camera host and load/run it, in order to get the required capability.

It is important to consider this, because the library you're talking about may have multiple dependencies.  The camera capability you want to consume, for example, may also require an AI accelerator chip, which may not be present on the camera node, but it may exist in the larger system.  If you want to consume this AI analysis of the camera feed, a separate setup process must first begin which connects the camera to the AI accelerator, and then your program to the output of the AI accelerator.

Under MAD, this scenario makes sense.  It is reasonable to expect this stack of programs, libraries, and drivers to deploy automatically when you ask to consume what appears to be merely the capability of the camera.  It makes sense that you can simply use a verb on a noun.

If someone else finds a way to do the same… I welcome it.  Ultimately, I really just want computers to be better.  In the meantime... I hope this sparks at least some thought on the subject.

Half-MAD: A Shorter-Term Vision

 I think that one of the reasons why it has been difficult to enunciate exactly what it is I want project MAD to accomplish, is because to an alarming extent, I want to do things that we are not doing in a modern machine, but extended to encompass all of several machines at once.  Thus to my chagrin, I end up effectively making a list of demands that might be considered absurd if they were to happen in a contained, well-understood environment, and then adding “…to everything, everywhere” at the end of the list.  And to be clear, Computer Science as a discipline is well versed in problems that do not scale well.  Proposing something difficult and then demanding that it scale is, itself, extremely arrogant.

The only reason why it doesn't approach being farcical is that I am proposing each sub-project with methods already in mind, if not in hand.  Ultimately, when it comes to actually implementing the ideas, new problems will crop up that we have never experienced before.  But to better understand why those projects are worth the problems they cause, it's worth discussing some of them in a smaller, more general context.

What is a Device and What Do They Do?

I talk a great deal about how Project MAD is designed to let you access devices or resources or capabilities of multiple pieces of hardware.  This is a strange topic for an application programmer; if you haven't already delved into microcontrollers and peripheral interfaces, you may wonder why anyone would care about all the devices attached to a system anyway.  When all you are doing is building some random bit of application logic and wrapping a user interface around it, using fixed and well-defined OS concepts or third party libraries, well, if you end up needing to talk about drivers for literally any device then someone has done their job wrong.  And indeed, if I wanted the average, user-facing application developer's life to change, I would probably be making their life worse, not better.  Part of the design of MAD/ADA is that, in fact, they don't need to worry about devices.

But also, devices will absolutely be used to control how their application works and how it is distributed across the system.  That's all fine, assuming they don't have to worry about it - but of course, eventually, a professional developer will run into every bug that comes within a mile of their code, let alone the ones actually inside it.  So if they do, technically, need to worry about devices, it's worth asking: what are they, why do we care, and what do the changes I'm making actually do for us?

Obviously, at its simplest, a “device” either changes the world outside the computer, or produces data based on the world outside the computer, or else it makes some data-to-data transformation task easier.  Those are, basically, the only things a device could even possibly provide: output, input, or services.  If you want one of those three things, you are going to be touching a device, even if that's only with an OS-standard ten foot pole mounted to the nearest wall.

An application developer's life is usually pretty simple: if we need input, we take it, if we need to output, then we output, if we need some service, then we talk to it.  Usually, someone else simply handles these things, in fairly standard ways.

The trouble starts whenever the OS itself either doesn't care about a device, or makes faulty assumptions about how you want or need to use it.  Say you have a TV across the room and you want to cast video to it. Modern OSes tend to only do two things with monitors: extend the desktop onto it, or copy some other monitor's output onto it, and neither of those may serve your needs well.  That's why in 2013 Google put together a tiny little dongle with a surprisingly important feature: you could just send video to it.  It took input and produced output, entirely separate from the normal desktop (or phone) metaphor that we had become familiar with.  And while I won't say that has or hasn't changed the world, I for one am grateful for such a device.

You can understand devices like that with a simple, generic model.  They have some requirements (the Chromecast needs internet and video out), have some capabilities (it will display internet video streams on the TV), and have some way to control it (a remote, or network packets).  When I talk about devices, I pretty much mean anything that fits that, extremely broad definition of input, output, and control.

So it may surprise people who don't look at how OSes work that an average programmer probably doesn't know how to get a list of all devices present on a given machine, can't tell what a random device is capable of, and can't figure out how to control that device.

On all major operating systems today, the device model can generally be broken up into two categories: if the OS itself uses it, then the OS takes care of it and you live with the OS model for better and for worse.  Otherwise, it may be technically available but you'd better find a library or application that knows what it is and how to use it, because they ain't touchin' it.

And that's… that's kind of it.  There isn't a whole lot of nuance here.  Either the OS takes care of it, or you're on your own.  There are some cases that are better or worse, but broadly… yeah.

Why?  Well, that's not hard to understand.  Someone has to invent a way to deal with it.  If that someone works for Microsoft, then Windows can handle it; if they work for Apple, then Macs can handle it; if they enjoy open source projects, Linux can probably handle it.  If they are part of some other company, such as the manufacturer of some device, they will provide an application and/or library that works and consider their job done.

Unless the device is really, mind-bogglingly important (and/or you convince people that it is), nobody is going to reshape how the OS works in order to help your device fit into the ecosystem better.  And… it has been that way since the dawn of computing.  Generally speaking, everyone involved would rather make small changes than big ones, not least because if you make big changes, consumers may not like it and you may be out of business in anywhere from two months to five years.

Something like MAD's OS project doesn't necessarily need to reinvent the wheel so hard that it changes the way the world itself turns.  I may end up sounding like that's what I'm doing, because I want to paint a picture of what MAD is capable of, but you could just… add some of MAD's features to existing OSes, or as a third party structure on top of them.  To wit, devices.

But first: IPC.

What is IPC/RPC and Why Should I Care?

Inter-Process Communication and Remote Procedure Calls are both fancy ways of saying that two programs talk to one another.  Generally, IPC is happening among two programs on the same computer, and RPC is the same but going between machines; there are some differences there, including important ones, but let's set that aside for the moment and just talk about how computers work in general.

I said in the last segment that you can divide device support into “The OS Does It” and “You're on your own”.  Categorically, the same is true with talking to other programs.  If the OS does it (and by it, I mean, making some given program do something specific for you), then there is a function, one that probably works very similar to every other OS function that you have had to use while learning to program, which talks to that system program and then gets back to you.  If the OS doesn't care about that specific thing, at best you will find a library or programming language that makes the task of talking to some specific programs just as easy.  (There is an aside here for scripting, but that requires a whole ecosystem to set up and so, like OSes themselves, can be difficult to make pivot if a change is necessary)

However, it is at least as likely, if not more, that you will have to do a whole lot of research into how that specific program works and how developing IPC/RPC mechanisms work in general and then you do a whole lot of work to make the mechanism talk to the specific program, and oftentimes, talk to a specific version of that specific program, because that might change.  And this is why we have kind of standardized on IP and web services nowadays - everywhere you see them pop up, there are a whole lot of tools that almost everyone involved did not need to invent.  Reinvent, sometimes, but not from scratch.

IPC/RPC mechanisms, however, do still need to be made.  If you have a program, say, one that takes your company's new fancy keyboard with per-key LED lights, and exposes the light-output capabilities of that keyboard to the user (letting them make all the colors their own brand of pretty), you will probably not design a server into that program which lets other programs talk with it.  And why would you?  That sounds like a lot of work!

But look at it from another app developer's standpoint - if they want to make a key flash when a notification comes in, say, they have three general options: learn how the keyboard driver works (which may be, if I may summarize, very custom), interact with the manufacturer's application (which we just ruled out), or interact with some third party library which can do one of the other two options for them.  If nobody has or wants to make that third party library, they're also out of luck.

MAD takes a tact that may be very alarming, which is that all programs should by default have some IPC mechanisms - it's part of making them distributable across multiple pieces of hardware, after all.  But you could also… you know, just, use that same mechanism to expose some functionality.  That exposure doesn't need to be reckless; can maintain some controls, make sure the user is involved, something like that.  But if we decided that IPC/RPC should be easy, maybe we should expect you to interact with other applications more often.

And the point of deciding that it should be easy, is that having made that proclamation, there ought to be a very standard way to do RPC, such that it is just as easy to work with as when the system takes over handling that task for you, because you're talking to a system component and they know how to do that.  If we want programmers to take it at all seriously, you have to make it easier for programs to interact fairly regularly.

And if you're going to interact more often, it might be best to have standards for communication.  And if you wanted, perhaps, to make that OS/DIY dichotomy even less severe… then OS IPC should work in the same way general application IPC works, adhering to the same standards.  That way, people who are forced to learn OS methods when they start programming, don't need to learn a whole new separate thing when they start dealing with new and more interesting tasks.

Which brings us back to devices, or rather, device drivers.

Making Devices Accessible: The System API Directory

Again, this is all stuff I've talked about before, but let's dial it back.  You have devices on your machine.  How do you interact with them?

Well, generally, every device you'd like to talk to exists on the far end of some kind of connector, and what is exposed at the very bottom of the CPU/Kernel system is the near end of that connector, not the device itself.  Many of these connectors require a little bit of finesse to use properly, so there may be a relatively complex driver just to ensure that any message you send across that bus gets where it's going, meaning you will be using a slightly higher level API that accounts for the bus itself.  Only once you are in contact with that other device can you talk about, you know, talking to the device itself.  Now, modern CPUs aren't just big-banging busses (meaning the CPU itself isn't hot-looping to make sure that every bit gets sent at the correct time, or anything similar); they use support chips, microcode, and other conveniences that might be relatively specific to the processor/motherboard combo you are using, but there are standards and firmware to sort of even out the complexities of making that part of the system work.

Once you can talk across a bus to a device, you have to know what language it speaks.  If you know exactly what device you're talking to, this shouldn't be hard - the manufacturer should list all the commands, what they do, what the return data means, and so on.  A few manufacturers prefer to keep some secrets… but let's not get into that.  Some things that you will be interacting with on the other side of a bus are themselves programmable, and you may be sending code to them (which is a whole other thing that is only partly covered by MAD/ADA), but frequently, you are just issuing commands that make it do whatever it is you want to do, and listening to the bus for both expected replies and updates caused by an event.

The software which converts requests from a programmer to commands that arrive at the device, and responses from the device to normal software events and function returns, can be understood as drivers.  This is where the OS/DIY dichotomy comes into play: the OS will keep track of devices it cares about, and ensure that there are very standard ways to interact with drivers it cares about.  Devices that the OS doesn't care about are not well kept track of, and the ways to interact with them are not terribly standard - at least, not outside of the specifications of whatever bus they are on the other side of.

The MAD/SAD proposes to tackle these two problems at the same time.  It asks device drivers to expose a standard API and an implementation library; programs that utilize the API can link to that library and they will be interacting with the device itself.  Then, all device drivers are listed in a filesystem directory, indexed by a key representing the API, and linking to that library.  If more than one device implements an API, even if they use the same library to do so, the implementations are listed separately and configured differently, so when your app links to a given device's library, you will be talking to that specific device when you make API calls to it.

That on its own represents a massive difference to how libraries and devices exist in any modern operating system.  Today, libraries are just things that may or may not exist, and devices are just things that may or may not exist.  You can't just link to a library and be talking to a device - you link to a library and then it will let you ask the system if any devices exist, because the library itself isn't configured, but is instead generic.  Likewise, you can't just point to a device and get a library that interacts with it.

When I talk about “You can just add shit from MAD”, I'm talking about shit like this.

But I go a little further with MAD/SAD.  I mentioned the libraries are indexed by an API key (in the filesystem, but you can ignore that for the moment), and if you've read any of my articles, you know that the API keys are in a tree, corresponding to an inheritance model.  And since I'm trying to explain all this simply, let's touch on that for a moment.

The API itself promises that the library attached to it will have certain functions; if you call those functions, you will get certain results, and the API makes all that very clear.  That doesn't mean that those are the only functions in the library - it's only a promise that if you go looking for certain ones, you will find them.  So naturally, a library may implement more than one API, each time exposing different functions.

Likewise, an API may provide different levels of what amounts to the same API.  It may provide a dirt-simple one, and a slightly more complex one, and then a far more complex one.  Generally, the simplest one gives you the fewest ways to customize it, but it also means you need to know the least about what kind of specific device you're using; the most complex one requires you to know exactly what the device is, but may essentially give you direct access to everything the device is capable of.  Since you choose the API when you write the program, you get to decide how specific the API you're targeting is.

Thus, in the MAD/SAD API tree, the folders closest to the bottom of the tree contain very simple APIs that require you to know very little about the devices, and which apply to far more devices, but which also expose very few features - and then there are more complex APIs that require you to know what you're asking for, but give you more control, until eventually you get to a library which may provide you direct access to the device with no hand holding whatsoever.

But in an inheritance model, you don't necessarily hide the API that came before - the parent and child APIs coexist, and the child API is obliged to expose every function the parent does.

Suppose you have a device that acts as a source of character data - that may be, for example, a keyboard, but it also describes files, network streams, and some busses like serial ports.  In any of those cases, the API will provide you with a generic “read” function that provides you with a “firehose” of data sent directly from the device, and probably also a generic “write” function to shout character data back down the pipe to be heard at the other end.  Questions like “What does the data mean?” and “What should I say to this device?” are not answered at all by this level of API.  If you use that level of API, you will not know what device is on the other end, or if there even is a device at the other end, not when you are writing the program itself - the user may know, but that's not your business.  Maybe the ‘device’ is not even hardware; maybe it's software, or maybe it's a black hole that says nothing and eats all input (aka /dev/null).

That same low-level API can be more useful to a programmer that knows what the device is, but when you know what device it is, you can usually also provide better service than just “read” and “write” functions.  Not always - some things really are that dumb and simple - but usually, you can provide an extension to that character device which provides new functionality.  Under MAD/SAD, that extended API will only be presented if the device promises that the library, and the device, implement those functions.  It will also still be listed among the low-level, raw character device drivers, because the extension adds to the lower level, instead of replacing it.

Suppose the character device in question your fancy backlit keyboard, for example; it may have three or four levels of API just in the ‘character device’ tree, to say nothing of (eg) a ‘raw usb device’ tree.  First, your keyboard is a generic character device; second, it is a keyboard (the output should be understood as keys, a keymap may be involved, and you can send commands to turn on capslock and so on); third, it may implement some standard “backlit keyboard” API, if there is one, and fourth, there may be a vendor-specific “This is our backlit keyboard” API that has some juicy extras the generic API does not.  This last API extends the other three; the third API extends the first two; the Keyboard API extends the character one.  But if you really want, you can just open the keyboard as a character device and listen to every keystroke that comes in down the line, and shout random character strings down the line just to see what happens.

Of course, the OS will recognize keyboards as something special, because the OS knows that most apps will care about keyboard input in one way or another - so it will make use of that second level of API (char/keyboard) automatically.  In fact it will probably create a virtual keyboard device that collects input from all keyboards (which should be a standard itself) and uses that instead of any specific device, because then the user doesn't need to change how anything is configured when they plug a USB keyboard into their laptop.  But the real point is, we have made the generic way to reach devices good enough that the OS can just use this tree to do what the system normally does, instead of making very unique and specific ways for the system to make critically important devices accessible, ways that kind of clash with how we program other applications.

Sounds nice, right?  Maybe a little boring.  But the way I introduced this topic was about APIs and libraries - it has nothing specific about devices in it.  Any persistent application that wants to provide an API and library can do so, even if it's not a driver or even a background service - and that application's APIs will be listed in the directory, same as any other.  So… if you install a new database service, say, you will be able to find it in the list of all databases on the system, even if that database is a foreground application for some reason.

Now, here's a funny little twist (which I've already talked about in another post): when you use IP and ports to connect to running applications and services, you normally locate services by a standard port number, because that's how it was when the internet was created.  Since more than one service can't share that port, other copies of the same have to have other ports; there isn't a standard for how to do that, so the user or service has to figure something else out.  You may be able to examine the other server processes and find out what ports they have open, or examine all opened ports to see if they sound like that service, but neither of those is a smooth, well-defined operation.  Under MAD/SAD however, if you know what API the service provides, you have a list of all of them that are running, and each item in that list is preconfigured to let you talk to it.

Moreover, you don't necessarily need to publish to the SAD globally.  Remember, I said the API keys are published in the filesystem; you might publish such a configured library file privately, if you had, say, an embedded database in a larger application, and you only wanted the parent application to talk to it.  Linking to that preconfigured library would connect you to the database; it need not be exposed to the network, or configured to deal with the network.  Presumably, an administrator or user could find it if they knew where to look, but that's about all the exposure it would have

The same mechanism that helps you find and utilize devices also helps you utilize things that are not listed?  Wow, it's almost as though it's just a better way to utilize things.  But what exactly is this mechanism?  I took it for granted that you can dynamically link to a library and it will be preconfigured to talk with a specific device.  That is… also not how things work.

Perhaps more to the point: Just before this I was talking about IPC/RPC and I never got back to it.  Why did I use that segment to introduce this one?  We already have dynamically linked libraries, even if they aren't preconfigured.  Making preconfigured libraries would be a fairly straightforward thing that has nothing to do with IPC/RPC mechanisms.

Well… the point with IPCs was that MAD argues every running application should expose IPC endpoints as default under MAD, meaning that if you write a service or application, you can interact with that application via another application directly.  Of course, if you want to interact with something via some IPC/RPC mechanism… you really should have standards.  Like an API.

What's the difference between calling a function embedded in a linkable library, and calling a function via an IPC mechanism directly?  Well… the linking bit, I suppose.  And it matters exactly what application and user is “running” the code, as far as the system is concerned.  But if the library mostly exists to provide a translation layer, between the published API and IPC endpoints exposed by a service… what if there just wasn't a need for a translation layer?  Or rather, what if the “translation layer” was a bog-standard system call?

What if your client application, the one that wants to use the API, can't tell the difference between linking to a library and directly talking via IPC with a service?  What if the programmer involved need never care?  What if they simply… use the API?  What if they do so by using a system call that does whatever is needed to reach the target function?

Bringing It All Together (By Taking It Back Apart)

Now… Does this all sound like hyperbole?  Does it sound like I think this is all easy?  Because this isn't all easy.  Let's look back at a list of things described in this chain of logic that are new:

• Have all applications use detailed IPC hooks by default
• System call that translates API call into IPC to a running service
• Configuring linked libraries at runtime so that they point to a specific device or service, and using such libraries
• System call that translates API call to loading and running a configured, dynamically linked library
• The two aforementioned system calls are the same system call
• Listing APIs in a systemwide tree
• That systemwide tree is in the filesystem
• That tree is an interface-inheritance tree (or some other semantic structure)
• Allowing the API system call to be directed towards libraries that are not in the system tree
• The system uses the API system call for basic functions (where feasible)

Now, we could quibble, and maybe I've missed some small points, but if I have an idea that requires making ten changes to the way operating systems work, some of them pretty freaking fundamental, then you certainly can't take all of those changes and tack on to the end,

“Now do this for a dozen systems linked together into a gestalt system.”

Yeah, that sounds a little crazy, huh?  About that.

Anything, Anyplace, All at Once

Funny thing about those changes.  One of the key themes is that it lets you address and control devices and services as though they were files.  (In the case of applications, though I've not talked about it here, there's also a big portability thing so that you can run them from anywhere, but same basic idea - an application being accessible means it being runnable when all you have is the “file”.  See MAD/SAD AF)

The point of them being accessible as files was that it's not hard to have a filesystem that represents multiple machines at once - you basically just make a folder with multiple machines in it, and list out the contents of each machine under them.  That's literally all it takes; you can list the entire filesystem, or just what you think is important, or both in different places.  Sticking trees on top of trees… is just how trees work.

The question becomes: what's the best way to make use of that?  UNIX already has the general philosophy of “everything is a file”, so in theory, you could make a distributed system where they simply… all use a distributed filesystem listing all files everywhere all at once.  But first, modern machines don't actually stick to “everything is a file,” and second, sometimes interacting with things at the file level is very… back and forth.  Which is something you may not want to do over a network, where every back and forth has significantly longer latency than a local call.  Like… thousands of times longer, maybe.  And also, you would need some universal addressing schema for this multi-machine directory so that path strings work the same everywhere and are unique.

But anyway, so you want to do some things locally, to avoid thrashing the network.  That means running a program on the machine that hosts some specific device.  How?

We already have some technologies to do this, and the one that most people will be familiar with is the “serverless” cloud infrastructure made famous by Amazon Elastic Compute Cloud (EC2).  The point of the serverless model is that a program can be loaded on any machine that has room, and you just keep track of where it actually is loaded, so that whenever someone goes looking for it, you can direct them.  If that program is part of a larger structure, have it keep track of where other pieces of the same program are, in case they need to talk.

And that's basically the ADA (it's not, but never mind).  It's serverless infrastructure designed for an operating system and home network.  All you're really doing, is running a program somewhere else, so that when it access a file that is actually a driver or service, it is doing so locally instead of across the network.

And that's it.  With that, everything works!  Victory fanfare, parades, we all go home.  Right?  Well, sort of.  No, not really.

Clear the Air (Once Again)

You see, part of the problem with the ADA is that if you do it wrong, people will hate to touch it, let alone depend on it.  They will avoid it, frankly, because people are lazy.  And while you could, maybe probably, have the system still work when programmers avoid it, by doing a bunch of extra legwork across the network, if developers being lazy means that the experience sucks for the user, nobody will want to be a user.

Even if I'm right about how this all should work, if we do a bad job on the implementation, that's it - game over, we lose, things to back to the comfortable old ways with minimal changes.  That's why I did a lot of extra thinking about how certain things should work, most of which sounds crazy out of context, because I'm planning for a future where OSes work very differently than they do today, and then on top of that, computers are combined in some way that developers don't yet understand.  So if I were, today, to ask a developer to operate under MAD rules, or to just assume that my OS will work the way I say it does… they would be totally justified in telling me to piss off.

They'd have to be mad to believe in MAD, before it's proven.  And that's kind of the point, at least of this post.  Some of the things that make the MAD system work, might be implemented on top of a normal operating system.  The API inheritance tree doesn't have to be the only way drivers work; it doesn't even have to be the canonical system way that drivers work.  But if it's there in a form that people can use it, then other things I've said might start to slot into place.

Likewise, you can have an IPC mechanism that application use, which is separate from the normal system IPC mechanism - it would just have to use some higher-level mechanism, like an IP:Port interface, with extra code wrapped around it.  This IPC mechanism could let applications and services publish APIs into the same tree, and you could talk to them the same way you talk to drivers.

That thing about using a system call to reach the tree… well, if it's not part of the system, you'd simply have to use a standard server on each machine.  Aside from that (and it would be inconvenient), things could work about the same, with it farming API requests off to the appropriate source.

And the distributed, agent-oriented MAD/ADA programming model?  Part of its genius is that you don't write drivers and services to listen on the network - they all expect to be contacted by a process on the same machine (IPC, not RPC), which is the same way they would be used if the ADA doesn't exist in the first place (well, mostly - there's user authentication, security and so on that needs to be addressed, but forget that for now).  Meaning you could design and run such services today, if the rest of the ecosystem existed.

Is all of this still a lot?  Hell yes!  But for all of that, it's still only half-mad, because it's less than half of MAD.  Still feasible, still important, still better.  But easier to understand, less intrusive, and I presume, vastly less intimidating.

Well, we'll have to see.

 

Beyond Unix: Typed Data

 So that whole rant last time was all to make this post somewhat easier.  In truth, on its own, that post is a little bit light on content, for all that it was 3500 words.  I know, I do that; sorry.  The point is that this topic is a bit complicated.  In exactly the same way I said at the top of last post, I have a lot of rewrites of this, and even after paring it down, I'm still not sure I'm satisfied.

The point of the last post was, Unix's “Everything is a file” philosophy is all about setting up a system that people can play with.  It's more “everything is the filesystem;” files as we understand them are honestly not all that important per se, which as I said last time, is good because files kind of suck.  The filesystem doesn't store any type information, which means that you have to intuit what they are based on their name, metadata, or contents - specifically the first few bytes.

And it's that bit that I want to talk about changing today - types.

We could, in theory, pack in a ton of type information about every node in the filesystem, information that must be queried and parsed every time you look up data on that node.  And it's not a terrible idea to maintain that capability, so that some files or executables can provide custom type information - but the general goal here is to standardize.  If some given data object's type is standardized, it makes more sense to simply name the relevant data type than to embed the whole thing with each object.  To be able to simply name them, that information needs to be available elsewhere - I suppose it's possible to simply name a standard but give the user and system no information about it, forcing them to rely on some external source or unnamed standards body, but that's a terrible way to build a system.  It would be better if all the information you need exists somewhere in your system (at least, for the types that already have an installed handler of some kind, the types known to the system).

Perhaps all of that type data would be organized into a directory.  A system-wide directory centered around how you understand and interact with data types.  You know, a System API Directory.  Like the MAD SAD.

That little bit of snark, however cathartic, is also completely uninformative; it leaves you asking, “Okay, but what does that actually mean?”  And part of the reason why this blog post has taken so long to get out is that it's complicated.  So complicated that I doubt this post will be the end of this topic, just as the last few weren't.  If I'm really lucky, I'll end up confident that I can at least move on to other things for a while, though I assure you, there are always still things left unsaid.

Everything has a Type

The stated goal of data types being used to describe files, pipes, and network data streams means above all else, having an official, canonical language that describes data structures in terms of fundamental data types, just as is done in low-level programming languages.  Since there are already formal descriptive languages like that, this means either canonizing one or revisiting xkcd 927.  Either way, you have fields in some given order, each described as one or more data blocks of a fixed size and meaning - primitive types plus some low-level data structures like arrays and dictionaries, presumably.  This can be slightly simplified over the type APIs used for programming, because data-in-transit and data-at-rest have fixed size and meaning; programming structures meant to expand or be modified in memory cleanly get serialized, and ambiguities get nailed down.  Data fields in a structure that have conditional uses either are present, or are not.

There are three objectives behind adding type data system-wide.  First is documentation; when something claims to have a given type, it's nice to know what that means (and from a local source), in both human-readable and machine-readable formats.  Second is confidence; type claims should (as in must, shall) be unambiguous, so when two programs, libraries, or data sources make the same claim, they are explicitly promising to be compatible with each other.  Providing that kind of confidence is valuable, or to be a bit more partisan, having a system that lacks that kind of confidence… sort of sucks?  Either way, that leads into the third - integration.  Once you have confidence in what something is supposed to look like, toolchains and workflows can be built to enrich the ecosystem, allowing administration, moderation, modification, and monitoring of the various systems involved.  When data types are no longer mysterious, when their meaning isn't buried deep in arcane documentation, then people can more readily make use of the data instead of making a new format that they can better understand.

Assigning type information to data blocks only serves as a claim that it is of that data type; going further and verifying that it actually is what it says, and validating that the data makes sense, requires either describing the verification process in formal language or having code available to do the type checking.  It's a fair to argue that going that far is unnecessary - but I'm not so sure.  Either there will be standards around verifying data, or once again, every application developer will be obliged to do the same themselves, which I've explicitly argued against several times.

Moreover, it pays to remember that we're talking about distributed systems.  On the one hand, that means that data will commonly be sent out over a network link, introducing transport errors and attack surfaces; a standardized way to verify that the data not only hasn't been modified, but is what it appears to be, would be useful.  On the other hand, in a distributed system, it's only more critical that agents agree on not only what a data type is, on disk, but what it means.  Data types can change, but more than that, small specifics in how they are interpreted can change, whether that's because the old interpretation was erroneous, or because they want to add, remove, or modify some feature, and that affects all data associated with the system.  Similarly, competing software vendors implementing some data type might disagree on the specifics, as the way one part was using the data when they wrote the file, may not match how will will be using the data when they read it.

That means that we need to move past describing the data in terms of its literal contents - though that is also important - and describe the semantic standard, as represented by the library, application, or service that will be making use of it.  In other words, it is important to describe a file type in terms of the library or application that should be used to read it.  That doesn't necessarily mean a type can only be handled by exactly one specific library or application - but the language used to specify the library or application must be an explicit promise of compatibility between adherents.  Only an explicit promise of compatibility allows the system to have a level of confidence high enough that you can build an entire ecosystem on top of it.

Once you have that level of confidence, you reach what is, ultimately, the goal: both files, and file-like data streams, become nothing more than a buffer between the data producer and data consumer, a middleware that can be effectively ignored by applications programmers.  Once you've confirmed the two are speaking the same language, the entire logic train between the function call that writes the data, and the function call that reads it, might as well be a single atomic operation - at least in cases where the time and space between the two events don't matter.  You can pretend, for those intents and purposes, that the producer and consumer are part of the same application, or that they are transferring the data object directly from one memory space to the other directly instead of using complicated middleware to make the transfer.

Which is a funny thing to say, because one topic I've thus far not talked about much on this blog is the hardware aspect of MAD - and one of my ideals, there, is that the network backbone that connects modules together literally just transfers a memory block from one machine to another.  That was true from the beginning, though it makes more sense nowadays, given the assumptions of the ADA application model.  Specifically, under ADA, most of those data transfers go from one part of a distributed application to another part of the same application, which is vastly more sensible than any alternative - but either way, on a programming level, moving data from one hardware module to another can be seen as an atomic operation, at least in an ideal implementation under ideal circumstances.  You make a single call, and either it succeeds or fails, with no third state that the application itself must manage.  Granted, that's more or less how data transfer was always supposed to work, which is why there's not been a lot of reason to talk about it on the blog.  The point, here, being that these backbone data transfers should be a system call, one that the average program gets to just assume will work, or else error out.

Adding types to files, pipes, and sockets, is simply another way to do the same thing.  If you can be sure that data-in-transit or data-at-rest states do not affect the data, then you can abstract them away.  Ideally, when you write a program, you deal with in-memory data types, and if you serialize it to disk or across the network and then deserialize it somewhere or somewhen else, you should come back with the same in-memory data object, as though all the stuff in between had never happened.

The reason why we can't just do that is that there are so many things that can go wrong, and it's generally the programmer's job to handle each and every last one of them, or at least, to stand there and be present while some library or other handles them.  When reading from a file, you need to handle the file not existing, it existing but not being of the correct type, it being of the correct type but malformed (in terms of the data structure itself), and only then can you start to parse its semantics and meaning as they apply to your program.  Those three errors, broadly speaking, can be described as one assumption made by your program: there exists a file of a given type at a given filesystem location.  The same general principle can be said of data-in-transit, with the file-doesn't-exist error being replaced by a null check or similar.

Let's suppose that under MAD, a program's data ingress and egress points must have detached precondition/postcondition blocks, written perhaps with decorators as separate functions instead of part of the ingress function.  Further, let's presume that data types are required to have a validation function, which only checks to ensure the data type isn't malformed.  When data ingress occurs from either a disk or a stream, those three fundamental type errors are checked with a single trivial command: type.validate(data).  If the file doesn't exist or the streamed value is null, it fails; if it claims to be of a different type, it fails; if it has malformed structure, it fails.  The entire rest of the function precondition (if present) is about the meaning of the data - and the main body of the function can be written presuming the precondition has been passed.

Granted, the idea of pre/post conditions is not new and yet many people don't use them, in part because there aren't good language constructs in many programming languages to handle them.  More than that, sometimes you don't know whether something is valid until you've done a decent chunk of the work that the function exists to do in the first place, or it can be difficult to pry things apart.  Unlike some people, however, I'm not insisting that people's programs work a certain way under the hood as a detached ideal - the idea that these condition blocks are separate means that the system can interface with the precondition/postcondition blocks separately from invoking the actual data ingress function - alongside type validation, it becomes part of the system workflow, verifying that incoming data is correct.

And part of the point of that is ensuring the system itself, separate from end-user applications and even from everyday services, has a workflow for validating data.  If you are building a system on what are essentially remote procedure calls, you need some way to detect not only errors (which may be due to physical programs) but also malfeasance and malice.  If the system starts seeing data packets sent over a network link that are of the correct type but routinely fail precondition checks, that sounds like it might be a malicious actor trying to find an exploitable weakness, and that might be a reason for the system to take action against the sender.

It's possible that you could do the same by insisting that data ingress points throw errors, and even with condition blocks that would help.  As with a lot of things in project MAD, I'm not necessarily trying to proscribe a single course of action, though I will make arguments in favor of one over the other.  In this case, encouraging separate condition blocks might have a lot of knock-on benefits.  Those condition blocks might be automatically parsed and made part of the documentation of a function or program, for example, which is only more valuable in a distributed system - not that debugging third-party programs is ever easy even with local access, but understanding why things are failing starts with understanding what a given error actually is.  The condition blocks (or some public parts of them) might have debug information available even when the rest of the program does not, so that you know not only that a precondition failed, but exactly which check in the process triggered a fault.

Putting the argument in its most general form, at some point the function preconditions are simply a language for describing the function itself, which should (as in, it would be nice) be a part of its public API.  And that, getting back to the topic, is part of the semantics of data types - a given data format may cover a wide gamut of possible meanings, so making a guarantee that data producers and consumers are speaking the same language may involve getting further into the details rather than just saying that the correct data type is involved.  Condition blocks that the system can make use of, are just something that the system offers to help programmers and administrators debug errors in distributed systems - and it is an offering, not a demand.  Ultimately, if you tried to demand people use condition blocks, some people would still make blank dummy blocks and then do all their value checking in the ingress function itself, and there would be no way for the system to tell this kind of scofflaw from a more nuanced situation in which handling checks with condition blocks is untenable.

But okay - the point of all that is being able to abstract away data transit and data storage.  That's lovely, but it gets away from the subject I started with, which is standards.  You can certainly standardize types, and even have nuanced validation functions, but the problem is still that people often do not agree on standards.  If you want to describe a standard, you may need a fair bit of information - which can be a problem, because the System API Directory as I've described it is literally part of the filesystem.

Which means that I'm obliged to explain a little bit about that.

Everything has an Implementation

The System API Directory can be understood as containing a couple different things using the same general mechanism.  That mechanism, is that data types are specified as a path string - the path string points into a filesystem tree, and in that tree, you will find a shared library object suitable to be dynamically linked into an application, which provides for the capabilities and handlers of that object.

That's it - that's the main mechanism.  To a certain extent, it's just a replacement for the Linux /lib directory.  However, the /lib directory is organized by source, where the SAD is organized by function.  And while there is a lot of complication in that (much of which I am explicitly and with some great frustration cutting from this blog post), the result for our purposes is that a type path is explicitly a promise that the data object (as it is being passed in from the filesystem or network) will provide certain capabilities to your program, even if that capability is just ensuring the data was passed cleanly from one end to the other.  With the libraries schema that we use today, and with some implementations under the MAD SAD, you can only ask for one specific vendor's version of one specific understanding of the data type, all others being irrelevant and unusable… and that's fine.

But the SAD is designed specifically to enable and empower standards-setting organizations, because it is hierarchical.  Although the tree may have top-level divisions that are simply categorical, once you start talking about a given type or programming interface, if you continue down that tree, you shall find child types or interfaces that claim to be fully compatible, while potentially having more features.  If for example, two GUI vendors put their heads together and agree on a common API which is, itself, sufficient for either of them to run the GUI, they can at that point also create branches of the API that have additional features, compatible with the parent standard but which are not compatible with each other.

In fact, that process comes naturally from the very idea of dynamically linked libraries (it's less clear with data formats specifically).  Library loading matches a function signature against an entry point in the library; all you need is a guarantee that certain functions exist and work a certain way.  If there are extra functions, because the library specifically uses a child standard, then you can safely ignore them just as you ignore parts of any other library that you won't be using.  The only really tricky bit is interoperability between libraries or services, in which case weird assumptions may come into play.

None of that means that application or service providers must hew to a standard at all, much less any given one, but organizing around a standard helps the ecosystem grow.  Even without standards, the SAD helps with discovering libraries that will parse certain files or data blocks - but with standards, suddenly a lot of things become possible, and all specifically because the SAD is designed to list all libraries that implement a specific type.

This discussion is, finally, about device drivers, and the role that the SAD plays in organizing a distributed system.

Towards a Device Model that Makes Sense

I don't know about you, but honestly, I don't understand how device drivers work - me being a computer and OS enthusiast with a CS Bachelor's who has used computers pretty much every day since I was five years old.  I know that vaguely, device drivers register with the core of a system that they are a device driver, and obviously the system forwards some requests to them when things having to do with that device comes up, but if you asked me for specifics, I couldn't give them.  Specifically, I can't help wondering how I would create a device driver for, eg, a cheap little USB dongle that I have programmed to do some specific thing, perhaps acting as a data source or API provider.  When I've looked into it, it seems confusing and troublesome.

Under MAD, a device driver is a service with low-level access to hardware.  That's pretty much it; end of story.  Oh, wait, we were talking about the SAD; not quite end of story, then.

You see, I said that the SAD lists data types, and I implied that data types are all about the data structures that actually make up the type as it rests on disk.  But I've also said that the SAD is full of interfaces in the programming sense - they are promises that, when you load a library and point the library at a data object of this type, the capabilities of that object will be available for you to use.  If that interface is a child of a parent interface, then you can understand that as a guarantee that the capabilities of the parent interface are all fully implemented in the child.  Logically, all of that makes some sense when you are taking about real, extant data that is already finalized and set in stone.

But you can also see how this works if the “data object” is nothing more than a reference to a device.  Loading the library is a promise that certain capabilities will be available to you, and specifying the device as a filesystem object means that you know that specific device will be the one the library interfaces with.  If some specific driver needs very special code in order to function, such that a generic implementation of the driver wouldn't work, then that device is registered as a child type of a parent - it exposes all the capabilities of the expected type, but unless another library claims to support that specific child type, no other driver code should be loaded for that specific device except the one that was designed for it.

Now again, I said that every driver is a service, and you should assume by now that every service has an Application Folder, because MAD/SAD AF is a thing I keep coming back to.  The driver service in question won't presume that some driver for their particular device exists somewhere out there in the aether - the driver service will provide its own copy of the interface library as an export.  Whenever someone else goes looking to see who implements the (parent/generic) type of that device, eg API/HW/Keyboard, that device driver, since it is providing access to the library, will be on the list.

The next question is - where is the file that represents the device?  Well… why not have that be the driver service's own application folder?  If everything has a type, that means that some things that look like and act like folders can also have types.  They can have interfaces, and they can be subject to standards.  You can validate that a folder is of a certain type by looking at its structure to see if it implements the standard.  Indeed, can't a folder have “functions” that are really programs, as long as those programs fit the expected function signature?  Of course, for that, you would need to be able to pass parameters to a program entrypoint in a type-safe way, but that was the point from the beginning, like it was planned.

Now, reverse the last two things I said.  If you can call a program entrypoint as though it were a function, given that we just added type safety measures, can't you export functions like they were executable program files?  Going further, can't some memory objects - real, programmer-type data objects - be represented as folders in the operating system, as long as the system knows how to decode that data type?  Sure, that might be a highly inefficient way to get access to your data, but you could imagine writing a program that did it, so why not have the system capable of doing it automatically, at least for data objects where that idea makes sense?

What if, for example, you have a device that reads air temperature, humidity, and pressure - when the driver is called, those sensors are queried, and then a data object is returned containing those three values.  Suppose that querying this device driver involves reading some node like a file, which will return a tuple containing those three values.  Because of the type data for that  specific file, you know that the first value is temperature, the second humidity, and the third pressure, each having a fixed size and format.  More than that, the system knows these facts about the data object - so if you were to request eg Sensor/Read/Temperature, the system could understand that request without your own program loading the library code.  You would simply request a file, and the output would look like a file - even though it was originally a smaller data chunk within a larger data set.  And you could even do it all without converting the data back and forth from a human readable format, such as strings.

It's obvious I'm getting ahead of myself, and this blog post is already getting long.  I will say that the above example is probably a good place to use another new path selection operator instead of the filesystem descent operator / - but the specifics don't matter for now.  If I go on much longer I worry I'll bake the brains of anyone still reading.

The point is - by making device drivers out to be services that respond to requests and return results in ordinary ways, we encourage people to make device drivers, allowing new, custom devices to integrate more easily with computer systems.  Why would that be an important topic when talking about a modular and distributed system?  Who knows, must be a passing fancy of mine.  But I will say that the idea of getting new, standards-compliant devices devices fills me with glee.  I'm not a neophile - I don't like everything that's new - but I am a technophile.  I want to be able to make my machines do things, and I get excited by having new capabilities.

And that, ultimately, is a goal worthy enough that I don't mind being called mad.

History of the MOS/ADA Programming Model

 This topic is tricky to introduce.  One of the central tenets to Project MAD as it is today, is a whole new model for applications, and I've said that it was the last things that came together, and it tied everything together in a neat package.  What you won't know is that my early concepts of the MOS/DCA project also included a modification to standard programming models, and that legacy is still very obvious once you know to look for it.

It's difficult to ferret out exactly what I meant by it if you're just reading my notes from back then, but it was there: I wanted to expose the internals of executables and shared libraries so that, say, a shell command could execute an arbitrary function.  Important to making that work is having an interactive shell that can capture typed data - more like an interpreted language's REPL loop than something familiar like Bash or Windows Command shell.  And this has to be a structured tool, in some fashion; in modern applications programming, you pass nothing to an application when it starts except strings, and the application has to figure out literally everything from that.

This ties back nicely to my complains in the last couple blog posts about programmers being obliged to reinvent the wheel.

At the time, what I had wasn't anything like a plan.  At most, I had a complaint.  And that sparked a wish.  After all, especially within the GNU toolset, how many applications are actually just wrappers around library functions?  Complex wrappers, sometimes; that complexity is frequently needed so that this one tool can interact with a complex programming library function in multiple ways, or so that common data pipelines (piping a list of files through grep, as an easy example) operate smoothly and the way you would expect them to.  As with a lot in programming, things that are simple to say are frequently hard to do, and if you want to actually do them, there is a lot of nuance and complexity that must be navigated.

The other side of that, though, is that the nuance and complexity spawns more nuance and complexity.  Your complex tool needs to be interacted with in complex ways.  Add a third layer on top, that assumes access to tools that use libraries; then add a fourth layer and more, tools on top of tools.  As long as nothing changes, eventually you get a level of tooling where you can issue a one-word command to do exactly what you mean to.

But every level in between the core library and the final level of tooling can shift.  Needs to shift.  Every layer must be updated whenever any of its dependencies update, for security reasons if nothing else, and sometimes bigger shifts are called for.  To a certain extent, there's no getting away from that - but if it were possible to have the library function that you actually care about, be executable in its own right without an application wrapping it, there can suddenly be whole levels of redundant tooling that are at least simplified, if not eliminated.

And of course there will probably be several other layers of tooling that crop up around this new concept; I won't lie.  Ultimately, we create tools because we have specific needs, and as long as the tools fulfill specific rather than general needs, they will diversify.  But many of those pain points that we are removing are redundant ones.  Every application needs to transform text input into data streams.  Applications that read from and write to data files have to pick or create libraries to that work with the specific protocols and data streams.  And if you're creating a workflow that involves data files being processed, you need individual applications whose only role is to be a part of that workflow, applications that parse files and put them in a form other applications can take as input.

Many of these niche applications, that only exist to take one data format and make it available, or take text input and render it back into a data file, are at once important to someone's workflow and possibly security, but also are tools nobody wants to maintain.  Nobody wants to need to maintain them.  It's important that someone creates a tool that parlays between data files and workflows, but it's a thankless job, and a product nobody wants to pay for.

In short, a shell that understands data objects and types, and can use that to call a function built into an application or library directly, instead of needing to build one application after another to handle the workflow… that would solve a lot of problems.

The steely-eyed veterans in the audience are saying, “Well, we have exactly that with interpreted languages like Python.  In Python, you can open a library and call a specific function out of it.  If you have a python file that is meant to be an executable, you can also load that as though it were a library, and call functions out of it, assuming it's designed right.”  And this is, of course correct.  (For the record, I had these thoughts before I was introduced to Python.  Probably.  Around that time, at the latest.  Again, my notes are not great.)

The difference is having that be an assumption built into the entire operating system, into the fundamental programming method, so that the entire system is built on one foundation instead of a shifting mix of dependencies from various legacies.  That is, of course, making a virtue of a vice: it means starting over and doing everything from scratch, which is... painful.  And these initial thoughts of mine were missing critical parts of the picture, which is why for many years I set them aside as irrelevant and a waste of time.  At the time, all I was thinking was that, doggone it, we shouldn't have to reinvent the wheel over and over.

And in truth… there are still today whole categories of errors that only exist because library functions are sealed away.  For example, languages like Python, PHP, Go, and many others can't easily use functions buried in libraries for other languages like C; people have to write new versions of old libraries, and each needs to be maintained separately, by different people, using different language constructs and different dependencies.  Ultimately, it becomes such a different problem to maintain each of them that even if one person was willing to maintain every copy of the same algorithm, it would become untenable for some fraction of them.

All this despite the fact that ultimately, all you want is to pass parameters in and get results out.  If there were no worry about dependencies or languages, if all you had to do was call the function and capture the return, it would be simple to do from any language.  Of course, from a certain point of view… that's always been all you have to do, if your language is willing to parse library headers or code from other languages.  But the library formats in use, Linux's .so and Windows' .dll, neither comes with a human- or machine-readable directory of functions meant to inform programmers of how to use them.  If you want to know how to use them, you need specific files intended for developers to use.  That means if you have an entirely different programming language and they want to use existing library functions, they need to read those files, …despite them being in another language.

In short, there's a reason why we are where we are.

When I talk about Project MAD as a change in the entirety of computing, it can sound like I'm vastly overestimating my own cleverness - and well, maybe.  But these are real problems.  My first concept almost sounds promising on its own: just call library functions.  But that concept does nothing about compatibility and discovery problems.  It hardly matters whether or not the code exists on your system if you don't know that it's there, where it is, what it does, how to call it, what side effects to expect, and what the return values and errors mean - and that assumes that errors get propagated correctly.  And the details need to be exact; one bit out of place changes everything.

Just like there's a reason why we got here, there's also a litany of reasons why we haven't leaped forward.  Even if we all agreed on one bright idea that we would work together on, it would be a massive effort with a lot of problems that need to be solved.  And if the bright idea that we all agreed on happens to have massive holes inherent to it, because it's not as bright an idea as we thought… that's a lot of work for nothing.

That's to say, I do understand, have always understood, and will always understand, that none of this is simple, that I can be wrong, and that ultimately, even if my idea was core to anything, it would be the people who actually achieve it that deserve the credit, because it will be very difficult.

But also…

But also, various subsystems that are a part of the MAD concept (here, specifically, the Modular OS and Agentic Distributed Application model) are there specifically to solve these problems.  I've described the system directory in this post on the Unified System API, but not all parts of the solution came around at the same time.  Part of it, as I've just described in this post, was there from near the beginning.  Many years later (five to ten, maybe) I started thinking about the directory, and specifically, having types understood system-wide and having them be a central part of the system directory.  Before this, all I really had was “Call a function inside a library, so we don't have to reinvent wheels.”  But it didn't make sense, quite.

I said in my first post that the ADA came about in the process of writing blog posts about the MOS/DCA concept.  I was trying to explain, and things still felt incomplete.  I had deduced that functions and hardware drivers needed to be sorted by function, which fed into the type hierarchy in the system directory, and I was also thinking about embedded libraries and applications, and how that made application deployment and dependency management neater.  I was writing about wrapping functions with scripts so that some logic was happening on remote nodes, but that wasn't working for me.

The reason why I like the ADA goes beyond the model itself.  The model slotted well into the weirdly shaped hole that the whole “software model” side of the project had.  Yes, you could call functions directly instead of only being able to run applications - but why?  What made that so compelling, and not merely one person ranting at the sky?  Sure, it would be nice to have a directory of all the functions and types within a system, but what made that so critical to the operation of an OS that it has to be a fundamental component?

But within the ADA, you may be calling functions within your own application agents and within others'.  You will be deploying agents to hardware modules specifically because there is software on that module that is required by that Agent.  And it goes beyond functions and code (or, well, it doesn't, but it helps to think of this way): the ADA is also about exposing data so that applications can be monitored and debugged, but that requires types to be in some sense objective, even if that only means a type connected to the application and not a system-wide one.

Equally, within ADA, an Agent exists to handle the tricky problems of accountability.  There is an entirely reasonable question when you try to, for example, perform GUI API calls remotely.  The memory being used by your GUI application can't leave the display controller, or at the very least, there will be a copy of the output in the display controller, which must be managed by the GUI stack itself--while being owned by an external party.  Without an agent such as the ADA's, there are massive, massive questions to be answered about how this is not an absolutely terrible idea.  But if you can't do things like call GUI functions remotely, then... what even is the plan?

But perhaps most importantly, in the ADA, deploying Agents and routing messages is a system function, one attached to the ADA servers and not one implemented by applications themselves.  That means that knowledge of functions, data, and types needs to be standardized at a system level.  If the ADA model were implemented separately by each application, then each application would need to separately track and manage type and API version compatibility.  Implementing this as a system function makes things not only possible, but easy for programmers to handle, by doing the hard work ahead of time, in the operating system.

I like to, perhaps arrogantly, think that I am touching on fundamental truths with some of these analyses.  It's why I keep hammering, for example, on don't force every programmer to reinvent the wheel.  It is hubris to say I alone know best (I've demurred enough that you know I'm not that hubristic), but it's still satisfying to line up a whole bunch of problems in a row, pointing at them, and say, Example one, example two, example three.  Tying things up with a bow on top is satisfying, and the point of feeling genuinely excited about the ADA and Project MAD in general is that a stack of problems are placed in a box and wrapped neatly up.

It also feels good because this isn't all the work of a day, month, or year.  While this talk of modifying programming models may not have happened in the first year of the project (the first part was mostly the DCA, honestly), it's at least 15 years in the making.  Tricky problems that have tickled the back of my mind for years are coming together.  These are problems that weren't just tricky in practice.  They were tricky conceptually; it wasn't clear that there even was an answer.  It wasn't hard to imagine that I was wasting my time, because there was no clear vision of the end.

There's still a lot rough, and there's still lots of room for me to be wrong.  But it's astonishing how the last few pieces have made the puzzle come together.  At least from where I stand, it looks promising, like there's really an answer.  Not an easy one, but one that leaves us all much better off.

And even if I'm wrong, perhaps getting more eyes on the collection of problems I've been working through will help.

MOS: The Unified System API

 So I've said for a bit now that under the ADA distributed applications model and the Modular OS paradigm, applications post a bunch of APIs to a central directory managed by ADA servers and by the OS.  Naturally it's important to talk about what that means.  And while I will try to sound like this is all well-thought-out by me, this remains one of the many aspects of the project where I will cheerfully bow to people who know more about API design - and the many other related, difficult tasks involved - than I do.

To be clear: This discussion is categorized under the Modular OS portion of the project, and to a certain extent, is independent of the ADA model.  That means that the ADA can exist without this, and this can exist without the ADA.  But if you want to understand the whole project as it exists in my brain, it may be important to talk about these frustrating implementation details.

Types and MOS Standard Interfaces

Before I talk too much about the directory, I'd like to get a little bit of groundwork out of the way, and that is to ask: assuming you have a massive distributed system whose capabilities are listed in a directory, what sort of things does the directory itself contain?  Perhaps the easiest thing to imagine is that the directory lists nothing more than addresses where you can find system functions--it is a massive list of strings matched to addresses.  If you want to invoke a particular method, in other words, you only need to send that request to a certain address, along with all necessary parameters, all formatted appropriately.

This gets complicated when you want to do more than simple projects, for a number of reasons.

One very important topic, for example, is type safety: sooner or later, you are going to be returned (or sent, as a parameter) a data object from a remote procedure call, and that mere fact is more of a problem than it may seem.  Attaching methods to types, for instance, is something that is done when you create your program, not when you run it; object methods are a part of your program's code, not a part of the object per se.  Nor would you necessarily want every object that is passed back and forth between application Agents to carry its own independent copy of the object's methods around, even if that would be the most portable option; that would be data-intensive and prone to security problems, as each data object at rest or in transit would be another chance for malicious actors to alter your code and inject some nefarious payload.

But at the same time, the idea that object methods are compiled into your programs creates significant issues in a distributed system.  It becomes certain that eventually, due to a version difference, old and new code will view the same data stream differently, and running old methods on new objects will create an incorrect result.  Sure, this won't happen within a single ADA application, assuming all affected Agents are recompiled every time the master program's object definitions are changed.  But whenever and wherever you are depending on third party applications and libraries, there is the possibility that a breaking change in the definition of an object means that new and old code won't interpret the same data object the same way.

In order to address this issue, it has been my belief for some years that the system directory will also contain a list of canonical types, and if an Application fetches a canonical type from this list, they fetch the up-to-date code for that type's methods, to be used in place of compiled-in object methods.  That means that if two applications developed years apart both reference the same canonical type, they will use the same method code to interface with the data object per se.

Of course, that's not a panacea.  There are frequently changes to APIs that change how methods work, their assumptions and guarantees; in that kind of case, the old code would expect new methods to provide old-style results instead of what they actually will produce.  Left alone, this continues to be a case where old and new code collide and produce erroneous results.  And that, of course, is why I never planned to leave things there.

In fact, there are a few different problems with having exactly one list of canonical types.  One is the versioning problem above, but there is also a vendor problem.  Suppose that you have two competing vendors each providing a backend that, for instance, implements a GUI.  In order for the GUI to work in general, both of these vendors must implement a shared, standard API, but it's also reasonable to expect that each of these vendors will have advanced features that require another, more expressive API.  When you receive a data object representing these vendor-specific APIs, they may contain more information than the basic standard, and as such, you will need to use the methods from that vendor to ensure that the extra data is properly handled, even if you are only using methods and functions from the standard-compliant API.  And, naturally, these competing vendors' API extensions are unlikely to be compatible with each other.

This kind of problem is already addressed in some advanced programming languages, such as Python or C#.  Because certain type information is embedded in the objects, if one type inherits an interface, you can treat the object in code as an “object implementing this interface”, no matter what its actual type is, and that solves our problem above.  However, to my (amateur) knowledge, these languages do not attempt to provide a directory sorted by type, and therefore, there is a problem here that they largely leave unsolved, one which the MOS project must address to accomplish my goals.

Specifically, the modular OS requires that Applications are able to specify an interface API and receive from the OS or ADA server, a list of providers of that API.  Suppose for example those two competing GUI vendors exist on the same system, and suppose that the application being run isn't targeting either vendor's specific API, but rather, targets the core OS standard.  When the application is first run on this system, the application (and by extension the user running it) needs to be faced with a question: which API provider do you wish to use to implement your GUI?  And in order to be faced with this question, the listing of providers for the GUI should include both vendors, even if they are each providing vendor-specific implementations of the underlying API interface.  When you talk about listing canonical object types, then, these two competing vendors both implement the canonical GUI API interface, in addition to providing a specific vendor-specific version of that interface, and so both must be listed when you ask for a basic GUI API object.

If this sounds trite, it's probably because you haven't yet understood that I am an idealist, and arguably a perfectionist.  It may sound trivial to list various vendors, and various versions, of API interface providers, as child nodes under the core standard, within the directory.  This would let you not only use competing vendors to implement the same interface, but would let old programs continue to use old interface versions (in case something has changed) while newer programs can use the modern interface.

But wait, that bit about versions contradicts what I said earlier, doesn't it?  You want new and old programs alike using the same codebase, not different ones, don't you?

There is a distinction here: I said that old interface versions, not old code, would be listed in the directory.  That means, for instance, that when a library is forced to change how a function works, in order to fix a security vulnerability, it is expected that they will provide shims that translate calls aimed at the older API to calls compatible with how their code now works.  These shims use the assumptions and requirements of the older API version, but may error out if the program does something unsafe.  This is still not a true cure-all, because sometimes the unsafe behavior is actually part of program logic and can't be removed, but it significantly closes the gap, and allows API interfaces to evolve in form and function without necessarily breaking old code.

When I say that I am an idealist, however, that extends beyond ensuring that this kind of capability exists in the system directory, into describing its form.

Directory Nodes and the Virtue of Canonical Shorthand

All of this brings us back to the question: what is actually stored in the system directory?  What do you actually get when you query the directory and get information back about a particular node?  At the very least, there needs to be some kind of type data there.  If you were to fetch a GUI API interface, for example, you would need to know what methods to expect on that object, what types are expected for parameters and return values of those methods, and what data may be contained within the object itself, along with any type data for that data if it is accessible.

But also, the node in the system directory has other specific data.  If the node represents an API type, for example, you may want a listing of versions and vendors provided for that type.  If the node represents a file on disk, you may want metadata about that file rather than its contents.  In other words, the directory node is its own object with its own data, and is simply another standard object type.  And because the directory itself is a hierarchical tree, one of the functions of this directory node object will be confirming that what comes after it in a request string, is valid.  If you take an object that has no children and attempt to get a child object from it, that is invalid, and equally, if you take a node that has no alternate versions and attempt to get a different version of it, it is invalid.

This is required to be a function of the node, and not a centralized system function, because the system itself is distributed.  You may not know until the first time you query the directory, whether your specific combination of Application ID and running user ID is permitted to perform a specific operation.  If you query an application and determine that are allowed to, for instance, list all the files stored by within, you may be directed to another piece of physical hardware in the system that actually contains those files, and the provider there may not allow you to actually read any of those files.  Likewise, you will need and may be permitted to read a listing of methods available on that node object, but you may not be allowed to call any such method, or you may be allowed a list of child nodes under that directory node, but not be permitted to use any of them.

This task of handling directory permissions on a cascading basis is a distressingly un-optimized solution, and there may be no optimized solution, but you can see how the distributed nature all but requires it, because each independent Application server may have different access controls.

But one of the sub-topics here that I find interesting is the use of shorthand to minimize string pattern matching, especially when dealing with standard queries.  For example, if the list of API versions is always stored under the sub-directory “VERSIONS” within the directory, every time you need to request that list, you must check that an eight-character string exactly matches (or worse, loosely matches) the canonical name.  It only gets worse if you try to make that mandatory identifier string something that no Application programmer would never use themselves; as an example, Python likes to have two underscores at the beginning and end of certain reserved words, which would take the eight character string VERSIONS and produce a twelve-character string, __VERSIONS__, which must be exactly matched on every check.

Given that something like the Versions subdirectory is actually a mandatory, system-designated resource, and given that latency is a critical problem in distributed systems, it seems reasonable to use shorthands to minimize characters in a directory path string, and this is a topic that I actively enjoy theorizing about.

For instance, I've mandated in this post already that type information be provided very frequently.  It is attached to data, to parameters, as part of directory nodes, and it is also listed in the directory itself, meaning that a canonical type string is an ADA-compatible URI string.  Assuming that the system directory starts from a common base, you would generally expect that all type strings begin with “/API/” or similar.  And while five characters is not an offensive quantity, it's a lot to ignore repeatedly when reading a list of types.  But this goes beyond API calls; for instance, it will be very common for Applications to reference a URI that is relative to their own root within the directory, or relative to their user's directory root.  If all these calls needed, for example, “/Applications/Example/” or “/Users/GenghisKhan/” prepended to the URI, this would need to be parsed and re-parsed constantly, each time ensuring that the strings exactly matched.

Instead of starting every ADA URI from the root of the system directory, it is my expectation that the first character in the URI determines which of several possible directory roots the request starts from.  I'll use placeholders here, but for example, “%GUI” may be short for “/API/GUI”, or “@Documents” may start with the currently running user's root, accessing  the subfolder Documents.  “$Cache” might be the cache directory for the currently running application, while of course, “/” remains the symbol indicating that you start at the full system root.

Likewise, once you have a directory node and are trying to perform an operation on it, it only makes sense to use shorthand for some functions, not least because it could prevent namespace collisions.  For example, you may want to use an internal method name that is in some contexts a reserved word, like Versions or API.  If you wanted to call the method “API” on your application, that string may look like “$.API()”, where if instead you wanted to get a listing of the APIs exported by your application, that request string may look like “$API/” or “$%”.  Because the “.” operator would resolve to a method name (whose canonical name might be, for instance, “$Methods/API” or similar), there is no ambiguity between the strings.  And if you are trying to find specific version of an API, perhaps that will be found under "%GUI:4".

There may, of course, be a problem with readability depending on how you implement this, and I acknowledge this is a limitation, especially when dealing with the basic ascii characters present on your average keyboard.  Too many basic symbols look like each other or like letters; $ and S, the various bars like \|/, the various dots like .:;, and so on.  Presumably if it became common to use specific unicode or advanced characters for path operators, you would start finding them on keyboards, but one way or another, I'm sure that it's a solvable problem, if not one for which I have a genius answer prepared.

The point of these symbols is truly only to act as a bridge between three requirements: these path strings should be short, human-readable, and unambiguous.  The various operating systems that exist today have largely eschewed the idea of advancing the language of path strings, and I think that's a mistake.  Using long names for concepts that are unambiguous in a given context is not ideal either for readability nor for fast parsing.  If you can represent that by using a single reserved character, that's preferable to matching long strings.  Granted, it's only really critical here because of how often we'll be performing these parsing operations one after the other, but I still think it's important.

Honestly, if the perfectionist in me had my way, I'd also compile some path requests directly into domain-specific function calls, so that this parsing was done at compile time and not runtime, for instance, having versions of the ADA path search or call that are specific to various types as implied by the operators used on them, but I'm not going to insist on that.  Goodness knows, this concept is complex enough as it is!

Shims, Type Conversions, and CODECs

One of the interesting consequences to everything I've said above is that the filesystem coexists with type information and objects containing methods.  It is therefore plausible that when you can be assured a file on disk matches a system type, that file on disk can have methods as though it were a data block retrieved from a program.  Which is convenient, because in order to retrieve the data from that file, you need a program which will return it as a data block.

In other words, it is to be assumed here that the system directory will allow you to make the border between data-at-rest and application data very fuzzy.  If you wish to store data objects at rest on a disk, or you wish to translate a stored data object into a live version, you almost don't need to worry about it at all.  You could, for example, have an image file with the same kind of methods that an uncompressed image in memory would have.  (It might be difficult to make a clear distinction with syntax alone, between changing the at-rest version of that data, and changing a copy of the data, but I have faith it can be done.)

All of this comes from the public type directory, and specifically, the fact that these public types include the type's method code for distribution to Applications.  The way programs currently work under modern operating systems presumes that each application will figure out its own way to translate data formats into live data, which means that for certain standard formats, you can have many applications each using different third-party libraries of various quality all to accomplish the same task.  And while you can have multiple providers of the same API under MOS, this is no different from dealing with live objects; the fact that these types are public means that you can have old programs use new code, or keep multiple programs using the same code between them, as long as the providers that you are using provide API shims that the old programs expect.

But there are more uses for API shims than this.  I've said that you can use API shims to keep compatibility between new and old versions of the same software, and perhaps you read between the lines and might have realized that competing API vendors can create shims that let them work on each other's data objects, extracting and encoding information that is stored differently by the competitor.  If that seems odd, it is; I am talking about Vendor A providing a shim compatible with Vendor B, such that if you are using Vendor A as your back-end and someone requests advanced functions using Vendor B's API, Vendor A can still provide them, while presenting the requesting program an object that claims to be from Vendor B.  This only makes sense if these shims are completely compatible - but when the API, its requirements, its assumptions, and its guarantees are all public knowledge, that isn't an impossible task.  Difficult and perhaps unlikely, but not impossible.

But perhaps more interesting than that: API shims can be used to convert various objects into various other objects, by allowing one data type to masquerade as another.  This happen a fair bit when we are talking about serializing data objects to and from text, for instance, but in our case it's equally reasonable to, for instance, convert a live data format into an at-rest data format, and vice versa.  In that case, for example, a provider of an image manipulation API may provide a shim that allows you to treat JPEG images stored on disk as though they were an instance of that live data object.  Or, the type for at-rest JPEG objects might include a shim that allows you to encode any live image of the standard type into that form.  Or… it's possible that I have those two examples backwards.  That sort of thing depends on implementation and finalized design, and to be frank, there's no reason for me to care which way it goes at this stage.

One interesting side topic in API shims I already touched on in my first post, and that is that ADA applications can target an API provided by an embedded library or application (For example, $/libraryX%XLib, that is, embedded Library X within the current application providing an API called XLib), and that embedded library or application might in fact do nothing else except serve as a translator for the API calls to another format.  This kind of case would be very useful if you wanted to ensure forwards compatibility; you are guaranteeing that your application has a specific place where you can place your own shims, or equivalently, where hackers and power users in the future can place their own shims in order to provide your application compatibility with libraries and applications that have changed since your last update to the application.  This would be most useful when applications are being abandoned; it allows third parties who do not have access to the application code to change how your program interfaces with the system years after maintenance has ceased.  If GUI standards have changed, if required libraries have security flaws, or if your preferred library vendor has also quit and now your application needs to target another, it would be trivial in all these cases to add shims.

Obviously, for some more-secure applications and for things like multiplayer games, some things need to specifically not be shimmed or exposed externally, but that can be monitored by checking that the code remains in a known state.  And because this choice to add internal shims to your application is done by the application developer themselves, it can provide as much or as little flexibility as the developer wishes.

Reaping the Benefits: Shell Scripts as Programs

So after having talked about what I want and why I think it's necessary, there's also an obvious question to be asked: is there really a significant benefit for all of this extra complexity?  And I'd like to think that the question can already be answered “yes” from the above, but there's more to say about the benefits of this advanced system directory.

One thing that I have always assumed, but perhaps not made explicit yet, is that the system directory can be navigated by an interactive text-mode command shell, and the addition of all the type system information to the system directory makes it possible for shell scripts to become nearly as powerful as programs.  I suppose I'm talking about something more akin to an interpreted language like Python's REPL loop, but with the added advantage of having data from the filesystem and from external programs just as accessible as data from your own program; in either case, the result is the same.

Most importantly, this capability exists without needing to create specific code libraries and application versions dedicated to command-line tasks.  For example, the Linux shell has had multiple people try to make shell- and script-compatible GUI frameworks, but they are all somewhat more awkward than doing the same tasks in a first-party library.  Under MOS, in theory, you could use exactly the same GUI functions while scripting as you would while running a compiled application - though, you would need to set up, for instance, Agents to handle your GUI memory.

Now, it's fair to frown and even balk at the idea of interactive shells having Agents, because Agents are explicitly and literally on another hardware machine, and multiple Agents will be running at the same time.  But at the same time, most Agents are not actually the core of the application, and are not producing events apropos of nothing; they are awaiting external events and providing functions to the main application.  So it makes a certain sense for the user of the Shell to simply be able to context-switch their command interpreter to be currently working from and executing commands on a different hardware node, with all of the other Agents, including what would otherwise be the application core, simply responding to events from wherever your command interpreter currently is.  It only really gets complicated if more than one Agent is the source of new events at any given time.

In this way, you could build a complicated and interwoven application line-by-line, by switching to various Agents, defining code on them, then switching away and utilizing the functions you left behind.  It would… perhaps be very slow.  But it would allow you to explore the capabilities of your system, piece by piece.

Speaking of exploration, one of the nice things about having a directory of types being a feature of your operating system is allowing programmers, power users, and shell scripters (some of which are not properly programmers) to explore the system as it is in fact, not merely trusting to theory.  As such, I have always assumed that this interactive system mode comes with ways to translate APIs and type information into interactive text, a feature that is so obvious it barely needs to be said, but is also so exciting for power users and new programmers that it cannot be underestimated.

This “shell plus system API directory” allows you to simply interact with systems and data on your system, and to simply understand how your program's data works and what data you have been returned by various functions.  Yes, you can write programs, but there are some tasks and some requests that should be commands, not programs.  If you want to, say, read every temperature sensor on your system and output them all as text, that shouldn't require someone to make a program; the list of temperature API providers is a native function of the directory, and iterating through them, reading them and translating the result to text, are all standard shell operations.  If you want to write a shell script that, for example, raises an alert every time a temperature sensor goes above a certain level, that's likely just as straightforward.  You could write that as a program and compile it, but you could also just write a single command that performs the action for you, at least during your current session.

That is, ultimately, exactly what the premise of the Modular OS project always was.  If the system has capabilities, you should simply be able to make use of them.  If you need to read a sensor, the sensor should be providing an API that lets any user, and command interpreter, extract that data effortlessly.  If you want to create a GUI window, you should be able to issue a command that creates a GUI window using the native system capabilities, even if making good use of it is somewhat more complicated than that.  And the same with other things; if you wish to record sound, or play back sound, or convert text to sound, or convert sound to text, as long as those capabilities exist within the system, the existence of the capability should necessarily mean that you can issue a command to do that thing.

In fact my ancient notes from the first years of the MOS/DCA project have random, difficult-to-parse thoughts on exactly this topic.  I was thinking about narrowing the distinction between library functions and shell scripts; I thought, surely, if you just expanded the capabilities of your command parser, you could capture the data block being returned by a library function in a variable, and pass it as a parameter to another function.  Those thoughts were naive; I was really thinking about reducing code duplication.  The idea only began to make sense once the type directory was added to the Project, but my point is, the thought was there from nearly the beginning.  (Actually, the papers aren't dated, so it may have been from some years afterward; I no longer remember, except that it is not recent)

The Modular OS Unified System API

In this context, what exactly do I mean by the Unified API?  Well, I mean that everything, from compiled applications, to shell scripts, to remote procedure calls, everything can be on the same page, in ways that current operating systems can't be.  If you want a shell script under Linux to be able to read a JPG file and display it on screen, that's… complicated, unless you have the exact and specific tools already present on your system.  It feels like it should be possible to just do, because other programs on your computer can do all of those things natively.

But the APIs between programs and the shell are not unified under Linux, nor under any other OS.  The shell simply isn't meant to do anything and everything; it was meant to, first and foremost, load and launch other programs.  Making a shell capable of doing ever more, making it more and more flexible, that makes less sense than making individual programs to do the specific things that you want.

Except that eventually, a shell that is considered feature-complete becomes the place where you make the system do what you want, and if you can't do what you want with the shell, it suddenly stops feeling complete.  If the system can do something, but you can't simply command the system to do that thing, it feels like you should be able to, like the function should already exist.  So administrators and power users sometimes feel cheated by the lack of a program doing the very specific thing they want, and they sometimes treat the people who create and maintain the programs that they rely on poorly, taking them for granted.  While using a shell, it feels like you should simply be able to do, what the system can do.

Perhaps nowhere is this more obvious than hardware.  I am something of a tinkerer; I have many Arduino-compatable devices around, and have an Adafruit micropython-based macro keypad constantly to the left of my keyboard.  I have for many years followed with excitement peripherals such as keyboards with blinky lights, dials, and text displays, or side USB monitors.  It feels like making use of these peripherals should be easy, but the systems were never meant to be played with on the kind of API level that I'm talking about.  Peripherals on most operating systems still mostly depend on low-level busses, kernel drivers, and programs designed very specifically to interface with one piece or brand of hardware.  They do not simply have their capabilities exposed.

In contrast, for example, suppose that a standard 16x2 text display is connected to your MOS system.  This is a very cheap and very standard kind of external peripheral, and it usually connects to your system via one of a few low-level busses, such as SPI, I2C, or serial.  On modern operating systems, if you want to make use of this display, you need a program that controls the low-level bus and sends the appropriate commands, and whenever that program or one like it isn't running, the display is useless.

Under MOS, I will assume that text-only displays in general have their own unique standard API, and potentially that small displays are a subset of that, and that the text display attached to the system can be recognized as being what it is.  A small driver program would be required to translate the bus commands to the device, but once that is done, you can simply treat the text display as being a text display, and any application in your entire system that expects a text display can find it and output text to it.  Obviously if they expect a lot of free space, that will be a bad idea, but if you want to, for example, use the display to output temperature data, that may be the kind of thing that doesn't need an explicit program, and can simply be done by issuing commands.

The idea extends to other displays.  Under current operating systems, there are only two metaphors for displays: either they are a part of the Desktop, or they are some specific device driver that has to be targeted specifically.  But under MOS, if you attach a small display that isn't part of the desktop, it may still be treated as a display--though perhaps not a hardware-accelerated one--and all normal display functions should apply.  If that display also has a touchscreen, then your normal display that is not on the desktop becomes a customizable button, or several.  If you want to create that set of customizable buttons with commands instead of an explicit program, that can work.

In short, you can just play with the capabilities of the system.  That is the goal, and it has always been.  That is what gets me excited about something like the Modular OS project.  It is why I have continued to return to it year after year.  Because operating systems today simply do not let you play with the system's full capabilities.  You can't interface that simply with hardware, or make use of system functions natively from the shell.  You can't just treat your system like all of its abilities are at your fingertips.

But you should be able to.  It's your system.  You're in control.  That's the ideal behind the MOS Project, and while that's difficult, and probably quite naive, I still think it's worth chasing after.