Clouds of Spaghetti and Meatballs - The Wantonly Exposed Implementation Details of the Modern Cloud API


The Assembly Language of Cloud Native Applications

Let's focus on AWS, the preeminent Cloud API.

To its credit AWS is reasonably performant, secure and amazingly robust and bug-free [Yep. Just compare it to other popular platforms like Facebook (capriciously buggy). Or Wordpress (slow and insecure and backwards incompatible). Twitter (from we-love-you developers firehose, albeit with fail whales, to we-despise-you developers walled cesspool with unpredictable behavior - e.g. the permanent suspension of this tool's Twitter account and its single lonely machine learning product announcement tweet)].

What we have here, with respect to the cloud and AWS and it relative success, however, is a failure of abstraction. A devolution of what it means to be a software engineer building software on top of a framework or API.

The state-of-the-art now has us working in what is isomorphic to an extremely verbose low-level assembly language-like hardware/software interface.

Here is where it starts getting really painful: in addition to the low-level nature of the cloud API, we got the stale biscuits to accompany that late-night heartburn-inducing cola:

  • there are now tens of thousands of instructions (opposite of the elegant new MIPS architecture which has 16).
  • the instructions often have some intricate relationship or prenuptial dependency requirement with respect to previous and/or related instructions (just to be clear, this is all in addition to the natural in-app collision management expected when an application's services share hardware platforms and data stores).
  • and most of these instructions have several if not dozens of possible parameters.

To continue to stretch the analogy, this is CISC (vs. RISC) chip-design on Acid. This is MSFT Windows with no hardware constraints and no target office-worker demographic to keep its API [just] within the bounds of human accessibility and usability.

One has to wonder if this is unreasonably, frustratingly wasteful of the developer's time and imagination.

Just how bad is it? You may not have had time to wonder about these issues as you try and navigate the documentation for the four (4) different interfaces (Console, CLI, API, CloudFormation) to figure out what call to make next among the [yes, rich] proliferation of often extremely subtle variations and alternatives available to you at any given step of the development process.

A person does just get used to accepting the pain of dealing with the extreme complexity required to do the simplest things.

Just suck it up. Everybody has to wade through this stuff, right?

Is this in exchange for the ability to also do the hard things? Nope. Is this because of the Microsofty put-em-all-in-the-menu-let-the-user-choose design philosophy run amok? Us developers may prefer this to the [Apple-like] lowest-common-denominator do-it-just-this-way-or-die approach - but aren't we just choosing between a fiery hell and a frozen hell?

It doesn't have to be this way.

Before we look at Un*x as a model of what could have been and may well be, hopefully, someday, let's take MSDOS as a model.

Would love to see the look on your faces! Ha!

It is popular these days to ridicule MSDOS. But a person could learn its entire functional surface area in just a few days - and a reasonably mature developer could have some reasonable expectation that they could, if need be, implement a reasonable clone by themselves. The API was accessible and its underlying model of operations minimalistic [in a good way].

MSDOS led to the greatest explosion of software applications and hardware extensions of all time [Let me know if I missed something.]. No other platform has shown this almost vulgar friendliness to software/hardware innovation.

In comparison, Cloud APIs purpose has been primarily to allow boring applications to be ported from traditional architectures e.g client-server to the Cloud.


There is just not a heckuva lot of innovation going on here [besides the Cloud platform itself, ironically. Keep thinking this has some deeper meaning... Seriously. But what?].

The only thing close to MSDOS as a rich platform for innovation, and promiscuous with respect to hardware, is embedded Linux.

And only thing close to MSDOS as a rich platform for innovation, and promiscuous with respect to software, is HTML/DOM and javascript.

The point here is that the complexity of the Cloud severely limits innovation. It also limits access to the cloud to larger organizations that have oodles of people to throw at dev ops.

[Ever notice how AWS-supplied examples are usually of edge cases of use to hardly anybody but that one corporation that wanted that particular work-around? Great for that one corp, of course. Awesome, in fact. But....]

[Yup, of course you can spin up an EC2 (or DigitalOcean etc.) server using some declarative orchestration like Terraform or CloudFormation et. al. and just move on but this is essentially ignoring the Cloud - treating it only as a bunch of meatballs.].

[Here again - one might expect that by this time - 10! years on or so? - both Terraform and CloudFormation would have 1000s if not 10s of thousands of turnkey examples, navigable by filters and by custom and popular search engines, so we could just choose one to setup our application network and move on. But nooooo... they want^H^H force you to learn their intricate and (ever-changing - looking at you Terraform version 0.12) language and massively extensive parameter space. WTF indeed.]

[And still no cloud-native ability to just reroute to without resorting to using cloud-external registrars or your low-level webservers - something that every website has had to do since day one.]

The way forward - a Cloud Operating System

What is the Cloud, anyway?

Let's assume it is just abstract CRUD in the sky, with not just data but human, service, hardware and network objects, integrated with a security complex

Let's mentally experiment a little. Let's try and make the Cloud 'un*xy'

CRUD - Create Read Update and Delete

One can argue that CRUD APIs are a solved problem.

In this corner we have data and S3, DynamoDB, RDS, and Cloudwatch and Cloudfront, etc.

  • Cloudfront is "ln -s cloudfront "
  • S3 is mv, cp, rm, rmdir and touch (with extra parms to handle mime-type, permissions etc)
  • cloudwatch is > cloudwatch://. Alarms are cloudwatch | grep "error" | filter -uniq | email &

Hardware provisioning is thankfully finally moving from malloc-like manual alloc and free to modern automatic allocation with garbage-collection with automatically containerized serverless approaches.

Programming language modules can be tagged with hints about memory and performance requirements and limitations with respect to costs. Hints can be expressed declaratively or algorithmically or 'learned'

  • exec -name XYZ "{ budget: { typ: 30.0, max: 50.0, duration: 'month' }, performance: { latency: { typ: '500ms', max: '4000ms' } } }" &

It is the job of the Cloud to figure out how to achieve the budget and performance goals. Budget priority > performance priority


AWS has chosen to focus on larger business by incorporating advanced low-level role-based security into the operating system itself [Like un*x and arguably a great idea]. Here they exposed not just the implementation API but the under-lying modules as well.

We got Cognito to manage users. With IAM for Groups and Roles both to reify a user class which is a collection of detailed declarative Policy DSLs. But no first class user object for application users vs. account users?

No algorithmically determined permissions [We want this user to be able to write to this S3 folder a maximum of this many bytes]?

No basic "please tell me why the heck this user cannot do this operation and what steps we can take rectify this?

No sudo "I want to execute this based on the permissions I have without having detailed knowledge of the names of the implementation's underlying data structures and operations?

This is like having to understand how mechanical locks are constructed in order to withdraw ten bucks from a ATM or open your front door. Well, isn't it?

  • touch s3:// "{ : [, : ], mime: }"
  • touch app:// "{ : [, : ], budget: , performance: }"
  • touch group:// "{ S3: { perm: , limit: "10G/day", alg: , inherit: } }"

"t" is for touchable which we are expanding to include updates of any kind of property outside the actual content of the object.

  • whynot touch s3:// "{ : [, : ], mime: }"
  • touch db "{ permissions: fn://mypermits, groups: }"
  • fn mypermit1 mypermits "(user, data, resource) => { in resource.groups && sizeof(data) < 1000 && updates_per_day() < 100 }"
  • update db://

    Any attribute can be replaced with 'fn://' to allow algorithmic control of behavior for more subtle or complex situations.

    fn's can subscribe to events on everything, by name or by class and run in background like un*x daemons but callable and updatable by name. AKA they are CRUDable.


    • ls apps://
    • rm app://
    • ping app://
    • pause app://
    • cp app:// dns:
    • cat app:// > app:// &


    Un*xy OSes are traditionally file-based. They try and treat everything like a file. This approach here may follow a more function-based model. Everything (data, apps, resources, users, permission groups) is ephemeral, always 'running' and can generate events at anytime.

    This approach relies on the modern automatic provisioning of Cloud services like databases (e.g. dynamodb [Oh we love you. Oh we hate you!]) as declared inside the app [IRL the DB and other resources should be parameters or ENV-like variables].

    AWS et. al. have provided us the underlying instruction set for a real Cloud Operating System. It is time for us to get our heads out of the primordial swamp of Cloud APIs and their infinite depths of patchwork documentation and into the rich fields of innovative software engineering.

    The optimal future seems to be the one that involves this kind of more simple more powerful operating system on top of the wild proliferation of API endpoints that has become today's Cloud. This would allow the vast majority of developers to more quickly innovate in areas that are more useful to their customers and their businesses [and more freaking fun! dammit, life is short.], spending far less time on the spaghetti works.

    -the rest of us