Could LLM's replace abstractions?
Software engineering to a large extent is a case of turtles all the way down. Abstraction over abstraction over abstraction. Could LLM based services like ChatGPT or Copilot replace some of layers?
Technology has always been driven by abstraction. Machine code is abstracted to assembly code, abstracted to high level languages, and those high level languages often get abstracted themselves to transpiled languages. This is the way, and always has been. Could LLM’s be about to change all of that?
Instead of asking an LLM to generate code in a high level language could we ask it to generate code a layer or two below and work with that?
Stop to think if we should
Why would we want to do this? The abstractions provide value. They make opinionated decisions for us that save us from our own folly. They make code easier to write and more maintainable, by removing the need for us to know arcane lower level languages. We could lose a lot by doing this. There is definitely a risk of creating something with critical design flaws and is relatively unmaintainable.
The flip side to this, is that we could create something more performant, and remove a set of problems and errors introduced by abstractions.
Instead of writing Java do we get an LLM to generate Assembly for us? Would the benefits of a performance improvement be worth the loss of maintainability. Would it even be feasible if there wasn’t someone on your development team with Assembly knowledge to debug the code generated by the LLM?
At this point I don’t think this example of skipping Java to write Assembly is feasible. We know LLM’s don’t write perfect code, and we know a person will be needed to tweak what’s generated. This alone will negate any benefits. Maybe in a world where LLM’s generate perfect code we can trust, potentially. For now, while a human is needed to finish the final product, it’s a non-starter.
Take me Higher
To quote the band Creed’s hit song Higher “Can you take me higher? To a place where blind men see. Can you take me higher? To a place with golden streets”, for now we should look higher up the stack for opportunities to reduce abstractions.
Let’s look at AWS’s CloudFormation as a potential target. CloudFormation is a declarative templating language used to define your AWS infrastructure. It works with the AWS CloudFormation service to deploy infrastructure defined in a template safely. The service provides safe deployments in a reliable idempotent way. The language is how the service is instructed.
CloudFormation templates can be defined in either JSON or YAML, and has support for transforms, macros, parameters and intrinsic functions. Due to the sheer breadth of AWS services, and the interdependencies between them, CloudFormation itself can become daunting. AWS has over 200 products, the majority of them have multiple associated CloudFormation resource definitions. For example the Amazon API Gateway service has 19 resource types under the AWS::ApiGateway namespace and another 13 resource types under the AWS::ApiGatewayV2 namespace. You need to understand these underlying resources and their dependencies to understand which resources to define in your CloudFormation template.
The CloudFormation syntax itself is relatively simple. The complexity is all in the resources and dependencies. To make this easier, a number of frameworks have been developed over the years to abstract away the writing of CloudFormation. These frameworks make opinionated choices on how to configure infrastructure, and have an understanding of required dependencies.
For example if you want to deploy a Lambda function that is invoked by API Gateway and reads data from DynamoDB, beyond the core resources to be configured you need to create IAM roles to handle the permissions to allow this. You also need to include the DynamoDB table name in the IAM role used by the Lambda function, and you need to write that same table name to an environment variable in the Lambda function.
Abstract! Abstract! Abstract!
With the Serverless Framework, Architect framework or Serverless Application Model (SAM) it can be a trivial number of lines to generate the CloudFormation for the example above as these frameworks understand the dependencies. These frameworks will generate the required CloudFormation and submit it to the CloudFormation service to deploy the infrastructure. These frameworks implement a declarative domain specific language (DSL) on top of CloudFormation. They work brilliantly for the architectures and resources they understand.
If you need to work outside of those well understood architectures and resources these specific tools optimise for, then there is the Cloud Development Kit (CDK). CDK supports general purpose high level programming languages like JavaScript, TypeScript, Python, Java, C# and Go. It allows you to define your entire infrastructure in the language of your preference, and will generate the relevant CloudFormation template for you. The advantage of using a general purpose high level language is you can implement custom logic, and create re-usable components, called constructs in CDK terminology, that can be shared.
CDK constructs are often layered, with Layer 1 constructs just a JavaScript class over a CloudFormation Resource (eg. the CfnFunction construct, and Layer 2 constructs providing components that understand resource dependencies (eg. the Function construct), and Layer 3 constructs which implement more opinionated configuration with some build steps and additional functionality in one module (eg. the NodejsFunction construct)
CDK itself is written in TypeScript, which is transpiled to JavaScript. It uses a framework call JSII to expose JavaScript classes to other languages through generated bindings, which enables the multi-language support. It’s turtles on top of turtles.
More turtles get added to the stack with projects like SST which uses a lightweight fork of CDK to enable them to call CDK method programatically.
Both approaches, declarative and opinionated, versus imperative, flexible and layered have drawbacks. The declarative and opinionated approach falls down when you need to work outside of the scope of what the framework supports. The imperative approach with CDK, while flexible, could add up to 5 layers of abstraction (CDK core, JSII, and Layer 1 to 3 constructs) to debug if there is a problem. These layers of abstraction can also make the overall process of synthesising code to CloudFormation slow.
Luscious Levitating Magic
These classes of abstractions that generate CloudFormation could be a good target to replace with LLM’s generating the CloudFormation. Replacing the abstractions with prompts that need to be validated and tweaked could be preferable, with the potential to improve performance and maintainability. This replaces the ‘magic’ of the abstractions within the CI/CD flow with the ‘magic’ of LLM’s in the development flow.
The hypothesis that I’m proposing is using an LLM to generate your CloudFormation that is then edited and validated by a mix of humans and tooling (ie the CloudFormation linter), is more efficient than using an abstraction.
It should:
Improve developer productivity as only need to know basic CloudFormation to maintain, though you still need to understand the services being used.
No need to learn another framework on top of CloudFormation
Reduce CI/CD errors as the abstraction is not required during CI/CD.
Improve deployment speeds as there is no synthesis/transpilation/compilation
Remove dependencies that need to be continually updated (runtime and npm packages)
There are some caveats to this. These frameworks that I’m proposing get replaced by an LLM do more than just create CloudFormation and interface with the CloudFormation service. They all help improve the development experience beyond abstracting CloudFormation away, whether it’s providing local function runtimes, watch modes that continuously upload code or type bindings used by IDE’s.
Additionally there is the privacy/secrecy aspect. For commercial use there needs to be guarantees around the privacy of the LLM generated CloudFormation templates to ensure they’re not training public models. Microsoft’s OpenAI service could be one route, or a private hosted version of a public model another route.
If CloudFormation generation is more efficient through LLM’s, something else would be required to replace the other features that these frameworks provide, to completely replace them.
Before heading down that route, the initial hypothesis that LLM’s are more efficient at generating CloudFormation than abstraction frameworks need to be tested.
I’ll explore that in detail in follow up posts. For now, here is a short video I did of me using ChatGPT to create a CloudFormation template.
The one aspect I noted is that you could ask ChatGPT to evaluate the generated template for errors, and it not only finds what’s been missed but it can correct them.
In the end, this short experiment I did gave me a template that was close to usable. There is definitely potential here, that I will be exploring in subsequent posts looking at ChatGPT, Copilot and Amazon CodeWhisperer among others to see if this approach is viable.