So that whole rant last time was all to make this post somewhat easier. In truth, on its own, that post is a little bit light on content, for all that it was 3500 words. I know, I do that; sorry. The point is that this topic is a bit complicated. In exactly the same way I said at the top of last post, I have a lot of rewrites of this, and even after paring it down, I'm still not sure I'm satisfied.
The point of the last post was, Unix's “Everything is a file” philosophy is all about setting up a system that people can play with. It's more “everything is the filesystem;” files as we understand them are honestly not all that important per se, which as I said last time, is good because files kind of suck. The filesystem doesn't store any type information, which means that you have to intuit what they are based on their name, metadata, or contents - specifically the first few bytes.
And it's that bit that I want to talk about changing today - types.
We could, in theory, pack in a ton of type information about every node in the filesystem, information that must be queried and parsed every time you look up data on that node. And it's not a terrible idea to maintain that capability, so that some files or executables can provide custom type information - but the general goal here is to standardize. If some given data object's type is standardized, it makes more sense to simply name the relevant data type than to embed the whole thing with each object. To be able to simply name them, that information needs to be available elsewhere - I suppose it's possible to simply name a standard but give the user and system no information about it, forcing them to rely on some external source or unnamed standards body, but that's a terrible way to build a system. It would be better if all the information you need exists somewhere in your system (at least, for the types that already have an installed handler of some kind, the types known to the system).
Perhaps all of that type data would be organized into a directory. A system-wide directory centered around how you understand and interact with data types. You know, a System API Directory. Like the MAD SAD.
That little bit of snark, however cathartic, is also completely uninformative; it leaves you asking, “Okay, but what does that actually mean?” And part of the reason why this blog post has taken so long to get out is that it's complicated. So complicated that I doubt this post will be the end of this topic, just as the last few weren't. If I'm really lucky, I'll end up confident that I can at least move on to other things for a while, though I assure you, there are always still things left unsaid.
Everything has a Type
The stated goal of data types being used to describe files, pipes, and network data streams means above all else, having an official, canonical language that describes data structures in terms of fundamental data types, just as is done in low-level programming languages. Since there are already formal descriptive languages like that, this means either canonizing one or revisiting xkcd 927. Either way, you have fields in some given order, each described as one or more data blocks of a fixed size and meaning - primitive types plus some low-level data structures like arrays and dictionaries, presumably. This can be slightly simplified over the type APIs used for programming, because data-in-transit and data-at-rest have fixed size and meaning; programming structures meant to expand or be modified in memory cleanly get serialized, and ambiguities get nailed down. Data fields in a structure that have conditional uses either are present, or are not.
There are three objectives behind adding type data system-wide. First is documentation; when something claims to have a given type, it's nice to know what that means (and from a local source), in both human-readable and machine-readable formats. Second is confidence; type claims should (as in must, shall) be unambiguous, so when two programs, libraries, or data sources make the same claim, they are explicitly promising to be compatible with each other. Providing that kind of confidence is valuable, or to be a bit more partisan, having a system that lacks that kind of confidence… sort of sucks? Either way, that leads into the third - integration. Once you have confidence in what something is supposed to look like, toolchains and workflows can be built to enrich the ecosystem, allowing administration, moderation, modification, and monitoring of the various systems involved. When data types are no longer mysterious, when their meaning isn't buried deep in arcane documentation, then people can more readily make use of the data instead of making a new format that they can better understand.
Assigning type information to data blocks only serves as a claim that it is of that data type; going further and verifying that it actually is what it says, and validating that the data makes sense, requires either describing the verification process in formal language or having code available to do the type checking. It's a fair to argue that going that far is unnecessary - but I'm not so sure. Either there will be standards around verifying data, or once again, every application developer will be obliged to do the same themselves, which I've explicitly argued against several times.
Moreover, it pays to remember that we're talking about distributed systems. On the one hand, that means that data will commonly be sent out over a network link, introducing transport errors and attack surfaces; a standardized way to verify that the data not only hasn't been modified, but is what it appears to be, would be useful. On the other hand, in a distributed system, it's only more critical that agents agree on not only what a data type is, on disk, but what it means. Data types can change, but more than that, small specifics in how they are interpreted can change, whether that's because the old interpretation was erroneous, or because they want to add, remove, or modify some feature, and that affects all data associated with the system. Similarly, competing software vendors implementing some data type might disagree on the specifics, as the way one part was using the data when they wrote the file, may not match how will will be using the data when they read it.
That means that we need to move past describing the data in terms of its literal contents - though that is also important - and describe the semantic standard, as represented by the library, application, or service that will be making use of it. In other words, it is important to describe a file type in terms of the library or application that should be used to read it. That doesn't necessarily mean a type can only be handled by exactly one specific library or application - but the language used to specify the library or application must be an explicit promise of compatibility between adherents. Only an explicit promise of compatibility allows the system to have a level of confidence high enough that you can build an entire ecosystem on top of it.
Once you have that level of confidence, you reach what is, ultimately, the goal: both files, and file-like data streams, become nothing more than a buffer between the data producer and data consumer, a middleware that can be effectively ignored by applications programmers. Once you've confirmed the two are speaking the same language, the entire logic train between the function call that writes the data, and the function call that reads it, might as well be a single atomic operation - at least in cases where the time and space between the two events don't matter. You can pretend, for those intents and purposes, that the producer and consumer are part of the same application, or that they are transferring the data object directly from one memory space to the other directly instead of using complicated middleware to make the transfer.
Which is a funny thing to say, because one topic I've thus far not talked about much on this blog is the hardware aspect of MAD - and one of my ideals, there, is that the network backbone that connects modules together literally just transfers a memory block from one machine to another. That was true from the beginning, though it makes more sense nowadays, given the assumptions of the ADA application model. Specifically, under ADA, most of those data transfers go from one part of a distributed application to another part of the same application, which is vastly more sensible than any alternative - but either way, on a programming level, moving data from one hardware module to another can be seen as an atomic operation, at least in an ideal implementation under ideal circumstances. You make a single call, and either it succeeds or fails, with no third state that the application itself must manage. Granted, that's more or less how data transfer was always supposed to work, which is why there's not been a lot of reason to talk about it on the blog. The point, here, being that these backbone data transfers should be a system call, one that the average program gets to just assume will work, or else error out.
Adding types to files, pipes, and sockets, is simply another way to do the same thing. If you can be sure that data-in-transit or data-at-rest states do not affect the data, then you can abstract them away. Ideally, when you write a program, you deal with in-memory data types, and if you serialize it to disk or across the network and then deserialize it somewhere or somewhen else, you should come back with the same in-memory data object, as though all the stuff in between had never happened.
The reason why we can't just do that is that there are so many things that can go wrong, and it's generally the programmer's job to handle each and every last one of them, or at least, to stand there and be present while some library or other handles them. When reading from a file, you need to handle the file not existing, it existing but not being of the correct type, it being of the correct type but malformed (in terms of the data structure itself), and only then can you start to parse its semantics and meaning as they apply to your program. Those three errors, broadly speaking, can be described as one assumption made by your program: there exists a file of a given type at a given filesystem location. The same general principle can be said of data-in-transit, with the file-doesn't-exist error being replaced by a null check or similar.
Let's suppose that under MAD, a program's data ingress and egress points must have detached precondition/postcondition blocks, written perhaps with decorators as separate functions instead of part of the ingress function. Further, let's presume that data types are required to have a validation function, which only checks to ensure the data type isn't malformed. When data ingress occurs from either a disk or a stream, those three fundamental type errors are checked with a single trivial command: type.validate(data). If the file doesn't exist or the streamed value is null, it fails; if it claims to be of a different type, it fails; if it has malformed structure, it fails. The entire rest of the function precondition (if present) is about the meaning of the data - and the main body of the function can be written presuming the precondition has been passed.
Granted, the idea of pre/post conditions is not new and yet many people don't use them, in part because there aren't good language constructs in many programming languages to handle them. More than that, sometimes you don't know whether something is valid until you've done a decent chunk of the work that the function exists to do in the first place, or it can be difficult to pry things apart. Unlike some people, however, I'm not insisting that people's programs work a certain way under the hood as a detached ideal - the idea that these condition blocks are separate means that the system can interface with the precondition/postcondition blocks separately from invoking the actual data ingress function - alongside type validation, it becomes part of the system workflow, verifying that incoming data is correct.
And part of the point of that is ensuring the system itself, separate from end-user applications and even from everyday services, has a workflow for validating data. If you are building a system on what are essentially remote procedure calls, you need some way to detect not only errors (which may be due to physical programs) but also malfeasance and malice. If the system starts seeing data packets sent over a network link that are of the correct type but routinely fail precondition checks, that sounds like it might be a malicious actor trying to find an exploitable weakness, and that might be a reason for the system to take action against the sender.
It's possible that you could do the same by insisting that data ingress points throw errors, and even with condition blocks that would help. As with a lot of things in project MAD, I'm not necessarily trying to proscribe a single course of action, though I will make arguments in favor of one over the other. In this case, encouraging separate condition blocks might have a lot of knock-on benefits. Those condition blocks might be automatically parsed and made part of the documentation of a function or program, for example, which is only more valuable in a distributed system - not that debugging third-party programs is ever easy even with local access, but understanding why things are failing starts with understanding what a given error actually is. The condition blocks (or some public parts of them) might have debug information available even when the rest of the program does not, so that you know not only that a precondition failed, but exactly which check in the process triggered a fault.
Putting the argument in its most general form, at some point the function preconditions are simply a language for describing the function itself, which should (as in, it would be nice) be a part of its public API. And that, getting back to the topic, is part of the semantics of data types - a given data format may cover a wide gamut of possible meanings, so making a guarantee that data producers and consumers are speaking the same language may involve getting further into the details rather than just saying that the correct data type is involved. Condition blocks that the system can make use of, are just something that the system offers to help programmers and administrators debug errors in distributed systems - and it is an offering, not a demand. Ultimately, if you tried to demand people use condition blocks, some people would still make blank dummy blocks and then do all their value checking in the ingress function itself, and there would be no way for the system to tell this kind of scofflaw from a more nuanced situation in which handling checks with condition blocks is untenable.
But okay - the point of all that is being able to abstract away data transit and data storage. That's lovely, but it gets away from the subject I started with, which is standards. You can certainly standardize types, and even have nuanced validation functions, but the problem is still that people often do not agree on standards. If you want to describe a standard, you may need a fair bit of information - which can be a problem, because the System API Directory as I've described it is literally part of the filesystem.
Which means that I'm obliged to explain a little bit about that.
Everything has an Implementation
The System API Directory can be understood as containing a couple different things using the same general mechanism. That mechanism, is that data types are specified as a path string - the path string points into a filesystem tree, and in that tree, you will find a shared library object suitable to be dynamically linked into an application, which provides for the capabilities and handlers of that object.
That's it - that's the main mechanism. To a certain extent, it's just a replacement for the Linux /lib directory. However, the /lib directory is organized by source, where the SAD is organized by function. And while there is a lot of complication in that (much of which I am explicitly and with some great frustration cutting from this blog post), the result for our purposes is that a type path is explicitly a promise that the data object (as it is being passed in from the filesystem or network) will provide certain capabilities to your program, even if that capability is just ensuring the data was passed cleanly from one end to the other. With the libraries schema that we use today, and with some implementations under the MAD SAD, you can only ask for one specific vendor's version of one specific understanding of the data type, all others being irrelevant and unusable… and that's fine.
But the SAD is designed specifically to enable and empower standards-setting organizations, because it is hierarchical. Although the tree may have top-level divisions that are simply categorical, once you start talking about a given type or programming interface, if you continue down that tree, you shall find child types or interfaces that claim to be fully compatible, while potentially having more features. If for example, two GUI vendors put their heads together and agree on a common API which is, itself, sufficient for either of them to run the GUI, they can at that point also create branches of the API that have additional features, compatible with the parent standard but which are not compatible with each other.
In fact, that process comes naturally from the very idea of dynamically linked libraries (it's less clear with data formats specifically). Library loading matches a function signature against an entry point in the library; all you need is a guarantee that certain functions exist and work a certain way. If there are extra functions, because the library specifically uses a child standard, then you can safely ignore them just as you ignore parts of any other library that you won't be using. The only really tricky bit is interoperability between libraries or services, in which case weird assumptions may come into play.
None of that means that application or service providers must hew to a standard at all, much less any given one, but organizing around a standard helps the ecosystem grow. Even without standards, the SAD helps with discovering libraries that will parse certain files or data blocks - but with standards, suddenly a lot of things become possible, and all specifically because the SAD is designed to list all libraries that implement a specific type.
This discussion is, finally, about device drivers, and the role that the SAD plays in organizing a distributed system.
Towards a Device Model that Makes Sense
I don't know about you, but honestly, I don't understand how device drivers work - me being a computer and OS enthusiast with a CS Bachelor's who has used computers pretty much every day since I was five years old. I know that vaguely, device drivers register with the core of a system that they are a device driver, and obviously the system forwards some requests to them when things having to do with that device comes up, but if you asked me for specifics, I couldn't give them. Specifically, I can't help wondering how I would create a device driver for, eg, a cheap little USB dongle that I have programmed to do some specific thing, perhaps acting as a data source or API provider. When I've looked into it, it seems confusing and troublesome.
Under MAD, a device driver is a service with low-level access to hardware. That's pretty much it; end of story. Oh, wait, we were talking about the SAD; not quite end of story, then.
You see, I said that the SAD lists data types, and I implied that data types are all about the data structures that actually make up the type as it rests on disk. But I've also said that the SAD is full of interfaces in the programming sense - they are promises that, when you load a library and point the library at a data object of this type, the capabilities of that object will be available for you to use. If that interface is a child of a parent interface, then you can understand that as a guarantee that the capabilities of the parent interface are all fully implemented in the child. Logically, all of that makes some sense when you are taking about real, extant data that is already finalized and set in stone.
But you can also see how this works if the “data object” is nothing more than a reference to a device. Loading the library is a promise that certain capabilities will be available to you, and specifying the device as a filesystem object means that you know that specific device will be the one the library interfaces with. If some specific driver needs very special code in order to function, such that a generic implementation of the driver wouldn't work, then that device is registered as a child type of a parent - it exposes all the capabilities of the expected type, but unless another library claims to support that specific child type, no other driver code should be loaded for that specific device except the one that was designed for it.
Now again, I said that every driver is a service, and you should assume by now that every service has an Application Folder, because MAD/SAD AF is a thing I keep coming back to. The driver service in question won't presume that some driver for their particular device exists somewhere out there in the aether - the driver service will provide its own copy of the interface library as an export. Whenever someone else goes looking to see who implements the (parent/generic) type of that device, eg API/HW/Keyboard, that device driver, since it is providing access to the library, will be on the list.
The next question is - where is the file that represents the device? Well… why not have that be the driver service's own application folder? If everything has a type, that means that some things that look like and act like folders can also have types. They can have interfaces, and they can be subject to standards. You can validate that a folder is of a certain type by looking at its structure to see if it implements the standard. Indeed, can't a folder have “functions” that are really programs, as long as those programs fit the expected function signature? Of course, for that, you would need to be able to pass parameters to a program entrypoint in a type-safe way, but that was the point from the beginning, like it was planned.
Now, reverse the last two things I said. If you can call a program entrypoint as though it were a function, given that we just added type safety measures, can't you export functions like they were executable program files? Going further, can't some memory objects - real, programmer-type data objects - be represented as folders in the operating system, as long as the system knows how to decode that data type? Sure, that might be a highly inefficient way to get access to your data, but you could imagine writing a program that did it, so why not have the system capable of doing it automatically, at least for data objects where that idea makes sense?
What if, for example, you have a device that reads air temperature, humidity, and pressure - when the driver is called, those sensors are queried, and then a data object is returned containing those three values. Suppose that querying this device driver involves reading some node like a file, which will return a tuple containing those three values. Because of the type data for that specific file, you know that the first value is temperature, the second humidity, and the third pressure, each having a fixed size and format. More than that, the system knows these facts about the data object - so if you were to request eg Sensor/Read/Temperature, the system could understand that request without your own program loading the library code. You would simply request a file, and the output would look like a file - even though it was originally a smaller data chunk within a larger data set. And you could even do it all without converting the data back and forth from a human readable format, such as strings.
It's obvious I'm getting ahead of myself, and this blog post is already getting long. I will say that the above example is probably a good place to use another new path selection operator instead of the filesystem descent operator / - but the specifics don't matter for now. If I go on much longer I worry I'll bake the brains of anyone still reading.
The point is - by making device drivers out to be services that respond to requests and return results in ordinary ways, we encourage people to make device drivers, allowing new, custom devices to integrate more easily with computer systems. Why would that be an important topic when talking about a modular and distributed system? Who knows, must be a passing fancy of mine. But I will say that the idea of getting new, standards-compliant devices devices fills me with glee. I'm not a neophile - I don't like everything that's new - but I am a technophile. I want to be able to make my machines do things, and I get excited by having new capabilities.
And that, ultimately, is a goal worthy enough that I don't mind being called mad.