Posts | Chris K Wensel

Determinism Injection

Sat, 29 Aug 2026 00:00:00 +0000

I’ve been refining a specification-driven development model for LLM software engineering for over a year now. The RDR model started with a simple template, but has evolved into a full human-driven engineering workflow. There are a number of stages in the flow, all derived from process, management, and engineering literature, as well as 30 years of experience writing software and managing software engineering teams.

Every step has an artifact a human is either expected or can review and refine to ensure they are building the correct thing, while also ensuring the whole project maintains its conceptual integrity as it evolves.

The challenge is that even though the flow is well defined and orchestrated by a human, the flow elements are just large LLM prompts that internally make localized decisions and look up facts across related documents.

This is decidedly non-deterministic. Guo et al., in The Deterministic Horizon, identify a deterministic horizon as the step depth past which unaided model accuracy collapses. And name the mechanism state-space decoherence, divergence that accumulates from extended reasoning rather than being resolved by it. This decoherence is aggravated by a prompt performing tasks that can be externalized.

So, instead of writing an application that executes prompts in a deterministic way, I need to inject determinism into the prompts I execute in order to improve their ability to actually solve the problem they are tasked with and not lose accuracy or focus because they had to grep 100 files for a single fact.

I think we should call this what it is: Determinism Injection.

Determinism Injection is the inverse of code orchestrating small prompts, the approach behind frameworks like Alibaba’s Blueprint First, Model Second. The prompt is the orchestrator, and generic or purpose-built tools perform the lookups, state transitions, and writes.

This is a clear play on Dependency Injection, where capabilities are polymorphically supplied to adapt an application’s behavior to a given context. Determinism Injection is a precursor step that isolates a deterministic behavior and externalizes it, which allows contextual binding of tools that will satisfy the expectations.

Intrastate is a tool for resolving state transitions. I call it a stateless state-machine. It has no backing datastore, it is handed context for a given flow it’s intended to execute, it may look up additional facts, then determines the next state. Since an LLM is driving it, if there is any ambiguity or failure, it generally has enough context to recover. But the net effect is that we lower prompt size, token spend, and wall-clock duration. And gain both (more) deterministic results and higher reasoning quality.

Retrofit is another tool for Determinism Injection. Instead of relying on a prompt to generate SQL schemas and migration scripts, the LLM stops at intent. It hands its intent (RenameColumn, DecomposeTable, etc) to retrofit, and when asked, retrofit can render a SQL dump (with or without data), or migration scripts (or their reverse for rollback).

Arcaneum was my first such tool, started over a year and a half ago, to compensate for hallucinated assertions. It is a RAG search tool for both full-text and semantic indexes. Instead of filling my context by grepping all my documents, I use it daily for searching and retrieving literature, documentation, source-code, and LLM harness transcript histories. Every design decision I make is grounded in research. And every assertion has a valid citation.

The kata and roborev projects are also examples of this model. The first is an inline issue tracker for LLM agents, and the second is an asynchronous code reviewer. Work that is actually quite hard for an LLM to manage (remember beads?).

Introducing Retrofit

Thu, 02 Jul 2026 00:00:00 +0000

In my last couple roles I worked with teams spending an outsized amount of their time either avoiding making changes to their SQL database schemas or cleaning up debt in their schemas and production databases.

The non-avoiders were performing simple tasks like fixing the types on primary keys (yes, they were running out of numbers), renaming columns, linking tables with foreign keys, or deduplicating columns across tables and the data within those tables (how many email columns do you have, company wide?).

Each change required days or weeks of planning, testing against snapshots, and finally scripted zero-downtime changes against production, only to be babysat by one or more senior engineers over a weekend.

The avoiders eventually just hired consultants, like myself, to assist with the fixes (yes, they were running out of numbers).

It felt like there was a better way.

As a developer I’ve been refactoring code for decades. But I’ve also run away from many projects that required a SQL backend, in part for the compounding complexity and a lack of simple refactoring flow.¹

And as a developer and engineer, I usually attempt to find and build better ways to solve repeating challenges.

In the early 90’s I was building interactive web sites for clients in C++ (and frequently SQL). I felt there must be a better way than C++ (or perl), so I created iPerform, a database-backed web scripting language for Netscape and MS IIS web servers.²

A decade later I was consulting on a large geo-spatial project in Europe. The team was force fitting a relational database model into what amounted to a document model. My bias against working with relational databases may have colored my opinions, but this felt overly complex and wrong. It was definitely very difficult to develop against.

At about the same time Apache Hadoop broke out of Apache Nutch and I was one of the first to kick the tires.³

With some of my former client’s challenges in mind (not exactly the document model mismatch) and the intent to bypass SQL entirely, I realized Hadoop needed an alternate API, so I created Cascading (a suspiciously relational model as an API) to solve those problems in a better way than Map-Reduce could.⁴

Having started my career build SQL driven applications and having evolved through and contributed to the NoSQL movement, now I find myself back to facing SQL backed applications, or interfacing with them more often.

So I’ve spent the last few months reflecting, researching, and designing what I hope is a better way to design, build, and importantly, maintain database-backed applications. Especially applications that have accumulated debt and need to be modified to unlock more value from them. And doubly so for applications being operated by second or third generation engineers who fear making structural changes.

We’ve spent all our time helping developers write code faster, but the actual bottleneck is the database.

“The trend I’m hearing more and more often is that this bottleneck is happening at the database.”

– Nicole Forsgren, PhD, DORA co-creator, ComputerWeekly (2018)⁵

Developers need to refactor, collaborate, and release database schemas with the same level of trust as they do with source code. This requires schema changes to be safe by construction.

“Schema changes (i) require costly data migration, (ii) burden the DBA with tedious and error prone maintenance for queries, and (iii) cause unacceptable system downtimes.”

– Carlo A. Curino, Hyun J. Moon, and Carlo Zaniolo, Graceful Database Schema Evolution: the PRISM Workbench (2008)⁶

With the emergence of LLM coding agents, the problem becomes more acute. LLM agents are great at making code changes, but I do not want one writing scripts to modify my database.

So if an LLM is given a tool that intermediates changes to the database, and provides a query interface to the underlying semantic and structural model, I believe the LLM will be much more effective at refactoring an application and it’s underlying data model, as will developers.

With more trust in the schema and data changes, the smaller the database release bottleneck becomes, the more value developers can ship to users in a given amount of time.

This is why I created retrofit.

Migration tools typically manage what your schema is. retrofit captures how it changes.

retrofit is a CLI (command line interface) that maintains a text log of logical schema change operators, not hand-written DDL. The operators are logical, thus a column rename is one operation with one logical outcome, not as raw DDL without any semantic meaning or history. There is value in quickly seeing and understanding how a table or column has changed over time and the dependencies between, for example, tables and views.

AddColumn, RenameColumn, CastColumn, CopyTable, DecomposeTable, and MoveColumn are just a few operations out of about 70 retrofit supports. Each of these are a refactoring operation, many of which are reversible (data neutral), and the details on how they are implemented (executed) is deferred until the snapshot or migration scripts are rendered, on demand from the CLI.

With a log of change operations (an oplog), you can render a DDL snapshot of your database (with or without seed data in DML) at any point in the change history.

Or render migration scripts between any two points in the change history. And if the operations are reversible, then you can render a script that downgrades a database.

And if you (or someone else) changed your database in place, you can detect the drift and import the delta into the oplog to bring it up-to-date. It can even detect common refactorings, like a column rename, or table split or join, and mint them as a logical operator.

Sealing a set of operations allows you to mark a stable version in the oplog. The seal itself allows for quick detection of any edits in the log that may have been merged in by accident that could corrupt your production database if they were applied. If the seal is broken, you are prevented from rendering a schema snapshot or migration script until recovered (from git or user intervention).

Actually, since the oplog is safe by construction, you cannot add an operation that makes no sense, like renaming a non-existent (dropped) column or table.

You can see retrofit in action on retrofit.sh.

There is still a lot more work to do before I release it publicly. Plus I have a fair bit of off-the-grid vacationing between now and then, and a bit more coding work to do.

On the site you can sign up to be notified when it’s released. If interested in early access, you know how to reach me.

This being a primary contributing factor to the NoSQL movement. ↩︎
This was before there was Cold Fusion. Actually, when I started iPerform, browsers had no cookies and there wasn’t even a HOST header specified in the HTTP protocol. ↩︎
Hadoop was the poster child of the NoSQL movement. ↩︎
Funny thing, later I was one of the first to put SQL (on Cascading) on Hadoop, in party by sponsoring the development of Apache Calcite’s pre-apache predecessor. ↩︎
ComputerWeekly, Unblocking the Database Bottleneck. ↩︎
Carlo A. Curino, Hyun J. Moon, and Carlo Zaniolo, Graceful Database Schema Evolution: the PRISM Workbench, Proceedings of the VLDB Endowment, 2008. ↩︎

LLM Toolchain - June 2026

Mon, 29 Jun 2026 00:00:00 +0000

My toolchain is constantly evolving, but as of June 2026 I’ve landed on the following:

claude harness - latest model
codex harness - latest model
glm-5.2 via olama via claude harness
Arcaneum - local full text and semantic search
RDR skills
kata-flight skills
kata
roborev

Coming soon:

intrastate

Arcaneum is how I start every project, via deep research against academic literature, adjacent open source projects, reference materials, and any online documents. For academic research I have a full corpus named PapersFast (with ~5k papers) that is fast to index into and search across. Often I create focused corpora with a better embedding model for papers on a specific topic to help catch nuances across papers.

The RDR process has been evolving for a year. It’s a spec-first process that now has skills wrapping each stage of the process. It is not fully automated by design as I still do all the user-facing design work and drop into the specs to revise them.

kata-flight is new. Over the past month or so I’ve effectively created an ad-hoc state machine around issue tracking with kata. kata-flight formalizes that in a set of skills.

flowchart TD init["/kata-flight-init
(bind repo)"] -.setup.-> flight doctor["/kata-flight-doctor
(health check)"] ops["/kata-flow-ops
(dashboard / reaper)"] kinbox["/kata-inbox
(human inbox drain)"] flight["/kata-flight
batch orchestrator"] review["/kata-scope-review
review gate"] kship["/kata-ship
single-kata ship"] resolve["/kata-resolve
fix in worktree"] roborev(["roborev refine / respond"]) flight -->|"per wave"| review flight -->|"per kata"| kship kship -->|"resolve"| resolve kship -->|"refine"| roborev review -->|"design fork"| seed[/"kind:rdr-seed
(kata label)"/] review -->|"held / needs human"| inbox[/"inbox:*
(kata label)"/] inbox --> kinbox kinbox -->|"READY"| flight kinbox -->|"TO-SEED"| seed seed --> seedtri["/rdr-seed-triage"] seedtri -->|"/rdr-seed (Stage 1)"| rdrflow{{"RDR design flow
Stages 1-7 (external engine)"}} rdrflow --> impltri["/rdr-implement-triage
(Stage 8: build)"] impltri --> roretri["/roborev-triage"] impltri --> implland["/rdr-implement-land
(land)"] implland -->|"flight bug children"| flight pship["/prompt-ship"] -->|"refine"| roborev wsp["worktree-ship-pipeline"]:::lib -. read by .-> kship wsp -. read by .-> pship lland["lib-land-rdr"]:::lib -. read by .-> impltri lland -. read by .-> implland classDef skill fill:#EDE9FE,stroke:#7C3AED,color:#1E1B4B; classDef lib fill:#F5F5F4,stroke:#A8A29E,color:#1C1917,stroke-dasharray:4 3; classDef label fill:#FEF3C7,stroke:#D97706,color:#713F12; classDef ext fill:#DBEAFE,stroke:#2563EB,color:#1E3A8A; class init,doctor,ops,kinbox,flight,review,kship,resolve,seedtri,impltri,roretri,implland,pship skill; class seed,inbox label; class roborev,rdrflow ext;

It allows me to ship well-researched bug fixes and small features. But if a feature or bug hits a specific threshold, it funnels the issue into the RDR process for more intentional design. There is also a wrapper skill that will drive an RDR implementation (after a design is marked final) through any roborev fixes that spin off during the implementation.

Some projects use codex as the roborev reviewer, others use claude+glm-2.5 as the reviewer.

Note that I’m publishing these skills as a matter of transparency with no expectation of any adoption. It’s great to see what others are doing, so I’m sharing. I know everyone has their own bespoke workflows and skills, so hopefully others will publish their own flows.

I also use Arcaneum to index/search harness transcripts to find ways to improve the skills and prompts. As well as leverage current research on prompt engineering. This is an ongoing process.

I’m not a “token maxer”, but I am tracking my usage publicly on https://tkmx.odio.dev/u/cwensel.

It would be great if there was a productivity metric that showed that I was getting more capability for the same token usage. It would mean my RDR/Flight skills/prompt changes were improving my token usage.

One major issue I’ve found is that LLMs are bad at state transitions, they are always slow, but frequently make poor choices as to the next state regardless of how simple the decision criteria is.

Because of that I’ve started a new project named intrastate.

As of this moment it is nothing but a collection of final RDR instances. Tokens permitting I’ll kick off the implementation before I drop out for summer vacation.

Evolving the RDR Process

Fri, 29 May 2026 00:00:00 +0000

For the last year I have been experimenting with a specification-first LLM (agentic) development model I’ve been calling Recommendation Decisioning Records or RDRs. You can read about the genesis to get more context.

Arcaneum was written using early “versions” of the RDR template, as well were a few internal projects.

This year I’ve started a new project that is a lot more ambitious and realized I was using/creating a lot of ceremony (copy/paste) as I revised any given RDR, iterating it through to the final stage (the stage right before implementation). This became quite tedious. So a couple of things happened.

First, using Arcaneum, I indexed a large amount of research on specification-driven development, both old and new, and devised a few prompts that would better formalize how I was, by hand, hardening the specifications. I tend to write a few at a time before implementation since they usually represent architectural layers, so hardening them in isolation and against each other was important. This worked quite well and led to fewer iterations or inconsistencies found within a given RDR during the implementation phase.

Of note, one primary benefit of the RDR process is moving from a NxM model to a Nx1xM model. Instead of lots of iterations against a large number of files, I iterate more against a single file, which is used to generate a large number of files or edits. And the seams between the RDRs are also validated as much as they can so they can be built in tandem.

I also formalized a “launch” implementation prompt (research based) to drive implementation with further hardening and TDD best practices.

After quite a few more rounds, I decided to index my Claude Code sessions in Arcaneum and mine them for additional ad-hoc patterns I was using to see if I could formalize even more of the process.

This landed me with additional steps to the process, which all can be found on the Github repo.

The README is a great place to start, but make note of the RESEARCH behind every element of the flow.

The process is still evolving, but the flow captures more data that can be used to continue to refine the process.

I’ve also experimented with the Superpowers skills on smaller projects. I like the worktree tool a lot and have directed a few skills to only work in worktrees now that it’s more stable and streamlined. But the specification-first model provided by Superpowers isn’t as robust for complex systems with lots of interacting layers. I’ll continue to use Superpowers for smaller or more throwaway projects, but not for projects that require a lot of upfront research and nuanced behaviors.

Arcaneum is part of my everyday workflow. From competitive analysis and focused research to iterating on research-driven RDR specifications. I’m using the stella embedding model for pdfs and md usually, but it depends on the size of the corpus. My general research corpus (PapersFast) uses minilm for embedding and has 3586 documents. But I usually re-index a subset (after searching PapersFast) into a new corpus using stella. For source code I use jina-code.

Keep an eye on the RDR repo, I do want to convert things to skills etc, but there are so many touch points that require human judgment (which is the point of it all) that I’m unsure it would be any easier to deploy and iterate with.

Arcaneum RDR Postmortem

Wed, 25 Feb 2026 00:00:00 +0000

One of my early goals for the RDR process was to enable postmortems so I can measure the drift between the original prompt and the final stable code. This way I could attempt to improve the process by identifying issues that repeat across RDRs.

You can see the postmortem docs in the Arcaneum repo. But I asked Claude to summarize the results.

From Claude:

Key Findings #

17 post-mortems. 208 drift instances. One meta-lesson.

The three biggest sources of plan-vs-reality drift were authoring-process problems, not domain-knowledge gaps — meaning they’re fixable by changing how plans are written.

Key Findings #

Code samples in plans are waste. Over-specified code was the #1 drift category (35 instances, 17%). Every single RDR included full class implementations that were completely rewritten. YAML schemas and config formats were never implemented. Code for deferred features was never used. The fix: specify interfaces and behaviors (signatures, contracts, data flow) rather than implementation code.

30-minute spikes would have prevented the biggest surprises. Unvalidated assumptions and framework API errors accounted for 45 drift instances across 13 RDRs. Documentation-only research was consistently wrong in detail — method signatures, filter syntax, model coverage all differed from docs. Searching dependency source code (not just reading docs) would have caught the majority.

GPU/memory management was the largest unplanned effort in the project. 8 of 17 RDRs required GPU acceleration or OOM recovery that wasn’t anticipated. One RDR described GPU integration as “low complexity: just add device='mps'” — the implementation required OOM recovery, GPU poisoning detection, NaN/Inf validation, Metal timeout handling, and systematic garbage collection.

“Day 2 operations” are not optional and not future work. List, delete, verify, backup, and info commands for every resource created were consistently deferred to “Future Enhancements” — then all were needed immediately. Day 2 ops represent 30-50% of the actual implementation effort for any feature that creates persistent resources.

Scope was systematically underestimated by 3-5x. CLI commands grew from 9 planned to 30+ built. A 50-line embedding wrapper became a 1,600-line GPU-aware system. A simple sync module became 140KB. The recurring cause: focusing on the happy path without counting error handling, Day 2 operations, and cross-cutting integrations.

Speculative performance targets were universally ignored. 13 of 17 RDRs included specific numeric targets (e.g., “100 PDFs/min”, “query latency < 50ms”). No benchmark infrastructure was created. No target was validated. They consumed authoring effort and created false precision.

CLI namespaces designed in isolation conflict with each other. 8 RDRs made independent naming decisions that required renames (fulltext → indexes, find → search semantic, inject → store), breaking backwards compatibility.

Existing infrastructure was routinely duplicated. Multiple RDRs designed new components when equivalent modules already existed in the codebase. A 5-minute audit of existing code would have prevented redundant designs.

The Meta-Lesson #

Specify interfaces and behaviors, not implementations. Verify APIs against source code, not documentation. Plan for the full scope — Day 2 ops, error handling, and cross-cutting concerns — not just the happy path.

— Summarized from the Arcaneum RDR post-mortem synthesis, Claude Code conversation

With this feedback, and postmortems from other projects, I updated my templates in the RDR repo.

Possibly over time I can come up with some sort of index that represents the drift and see if it decreases.

I also would be interested in determining if there are multiple types of RDRs that might require different templates. For example RDRs that specify subsystems that support actual application features (like build tools), where application features would be defined in a different template. Currently I’m using my one-size-fits-all template.

Arcaneum via RDRs

Wed, 31 Dec 2025 00:00:00 +0000

Arcaneum is a set of tools to index pdf documents, source code repositories, and markdown files. It can be installed as a Claude Code Marketplace plugin.

It is intended to create both semantic search and full text search indexes for use by an LLM – when rendering code or creating a feature specification. But currently I’ve had my hands full just getting the semantic search working reliably. This is a me problem, not a technology problem.

Prior to building Arcaneum I started using what I call a specification first LLM driven development model via RDRs. In my day job, I write a single RDR per feature. But for Arcaneum, I decided to write as many RDRs as possible before starting to render a single line of code. Check out the RDR index.

I don’t write Python, but Arcaneum is in Python, knowing there were a rich set of supporting libraries. Also, I wanted to see how far I could get not reviewing code, but just by using the app and adjusting the app at an LLM level of abstraction. If I did read the code, I wouldn’t have any opinions on it, so why bother.

The RDR model worked great getting my thoughts in order and during subsequent development. My primary challenge has been performance and memory management of the chunking and embedding pipeline. This is somewhat aggravated by my lack of reasonable tests. As a Java person, I complete every RDR with a suite of tests. But this wasn’t something I pushed hard on with Arcaneum.

I use Arcaneum every day I write code and on the days I’m just doing research.

One of my primary flows is to clone a git repo, index it, and create a document analyzing the code base over a different lenses. For example, how complete is the JWT support in a given framework. Or compare two frameworks footprint’s in relation to size of transitive dependencies and number of documented CVEs over time. I even ‘diff’ two versions of a standards document (this is not a replacement of your compliance team….).

I’m about to pick up my Clusterless project again and refresh parts of it. In preparation, I indexed the full aws-cdk repo last week, it took almost 30 hours using the jina-code embedding model – most repos take a new minutes to an hour depending on the embedding model used. Every line of Clusterless and Tessellate to-date are human crafted. I’m cautiously looking at using an LLM moving forward (using the RDR in strict mode).

It’s important to note that semantic search isn’t complete without a complimentary full text search interface. So far I’ve been mirroring my pdfs documents (mostly standards documents) with markdown copies in a parallel directory (not true of my full academic paper corpus of +3k files), so my LLM can grep for lines to use as citations etc.

Also, mactop is great for watching GPU utilization.

Recommendation Decisioning Records

Tue, 30 Dec 2025 00:00:00 +0000

Like many developers, I’ve been experimenting with LLM code development. Over the past 6 months or so I’ve landed on a model that I call Recommendation Decisioning Records or RDRs.

When prompting code directly via an LLM like Claude Code, I found myself in a corner more than once. This was especially true for complex features. Being stubborn, I felt I could prompt myself out of it without having to discard the code I had spent the last few hours generating. I learned a lot over the past few hours and all of it was in my head or locked in the codebase. The idea of throwing the code away felt wasteful.

I almost never prompted myself out of the corners.

Architectural Decision Records are a great way to document a decision. They are created, discussed, revised, and then agreed on. When I write an ADR I’m also writing the code to better understand the problem. My ADRs are documentation, but also the journey. They capture why I landed on the decision.

An RDR is a prompt. It is the journey to a decision. It is a specification. It has the background as to why I want things the way I want them. It captures the hard won discoveries about the problem I couldn’t originally articulate or technical details that I had to search through research and source code to find.

An RDR is also not code and is subsequently more concise.

This model is context engineering. The problem I want to design and solve is bigger than the LLM context I can work with to find a reasonable implementation. My context and subsequent purpose start to drift as I start coding. Because the moment I start coding, I discover new problems or options, and every side-quest throws me off track.

So the solution was to not start coding, but to start specifying and defining and researching.

An RDR is the specification for a single cohesive concept.

When faced with a related set of new features or designs, I create multiple RDRs, before coding, in order to maintain conceptual integrity across the design and reduce coupling between the components. You can see an example here in my Arcaneum project RDRs.

How to get started?

Copy the relevant bits from the RDR repository. Then ask Claude to create a new RDR using the README.md and TEMPLATE.md for guidance for a given problem statement. For example, “I need a logging utility that will mask PII and emit structured logs”. Once the file is created, I’ll ask to research the source code ecosystem for any existing libraries or utilities. In this particular case, I might realize I’m conflating two issues. PII masking and structured logging causing me to create two RDRs instead of one where I crisply define PII masking in the first and then how to leverage that solution best in the second RDR.

Some notes.

Ensure you use the RDR to identify multiple options/solutions to the problem statement. I find as I get to know the problem better, an originally less attractive option may float up as the best solution.

I lock down an RDR before implementation. If I got the design wrong on the RDR, I throw the code out and revise the RDR itself. I’m not terribly strict about this, but if you find yourself meandering away from the original RDR, reconsider.

I never delete an RDR, but if I find it no longer relevant, I mark it unimplemented or abandoned. I’ve also reverted an RDR after implementation.

Mentioned above, stack RDRs that are related and cross reference them for conceptual integrity.

I use Arcaneum to index source code dependencies that I may be leveraging or deciding to use in a design option. This prevents API hallucinations, which I find common. Use the RDR to capture any hidden flags or APIs that are relevant.

RDRs do end up with a fair amount of code in them to capture API usage etc. This is ok, but I’ve been trying to bias toward pseudo-code vs actual code where I can to reduce the token count.

The final solution, the actual implementation, will inevitably drift from the RDR. The goal of the RDR is to prevent the drift happening too early in the cycle, because once it begins it’s too hard to get back into focus. At some point I’ll begin capturing the drift in a post-mortem document to accompany the RDR so that I can better craft future RDRs via an improved template. This is one reason I recommend to lock the RDR before implementation (in a single git commit) so the gaps and methods to prevent the gaps can be reasoned about.

AWS API Gateway with AppRunner

Fri, 23 Feb 2024 00:00:00 +0000

AWS AppRunner is a reasonably new service that aims to simplify deploying and managing web based applications and REST endpoints.

New, in the sense that the AWS CDK does not offer stable L2 constructs yet for the service.

New also, in the sense that some capabilities available on the service are under-documented and difficult, if not impossible, to enable via the CDK.

Custom domains over AppRunner are one such capability.

Another such capability is having the AWS API Gateway front a given private AppRunner instance.

Without a long winded overview and explanation, if you find yourself with an AppRunner instance on a private VPC that is intentionally not publicly accessible, it is possible to place an API Gateway in front of one or more AppRunner instances.

First, making an AppRunner instance not publicly accessible requires dropping down to the L1 constructs.

Service service = ...;

// in order to make an app runner service private, we need use L1 constructs to set the network configuration
CfnService.NetworkConfigurationProperty networkConfiguration = CfnService.NetworkConfigurationProperty.builder()
 .ingressConfiguration(CfnService.IngressConfigurationProperty.builder()
 .isPubliclyAccessible(application.publiclyAccessible()) // this api is not on the L2 construct
 .build())
 .egressConfiguration(CfnService.EgressConfigurationProperty.builder()
 .egressType("VPC")
 .vpcConnectorArn(vpcConnector.getVpcConnectorArn())
 .build())
 .build();

CfnService cfnService = (CfnService) service.getNode().getDefaultChild();

Objects.requireNonNull(cfnService).setNetworkConfiguration(networkConfiguration);

When this happens, the AppRunner no longer allocates a domain name for access within the VPC. This requires an InterfaceVpcEndpoint.

Service service = ...;
String region = ...;
Vpc vpc = ...;

InterfaceVpcEndpoint endpoint = InterfaceVpcEndpoint.Builder.create(service, "InterfaceEndpoint")
 .vpc(vpc)
 .subnets(subnetSelection)
 .securityGroups(securityGroups)
 .service(new InterfaceVpcEndpointService("com.amazonaws.%s.apprunner.requests".formatted(region)))
 .privateDnsEnabled(false)
 .open(true)
 .build();

CfnVpcIngressConnection ingressConnection = CfnVpcIngressConnection.Builder.create(service, "EndpointConnection")
 .vpcIngressConnectionName(name.with("IngressConnection").lowerHyphen())
 .ingressVpcConfiguration(CfnVpcIngressConnection.IngressVpcConfigurationProperty.builder()
 .vpcId(vpc.getVpcId())
 .vpcEndpointId(endpoint.getVpcEndpointId())
 .build())
 .serviceArn(service.getServiceArn())
 .build();

You can get the domain name from ingressConnection.getAttrDomainName(). This will be required below.

Next up, we have to put a Network Load Balancer listener in front of the InterfaceVpcEndpoint, to do that we are required to make a target group of the endpoint IPs. Sadly, we can only ask the endpoint for endpoint.getVpcEndpointNetworkInterfaceIds(), ENI ids, and even those have to be lazily resolved.

This is where, the new to me construct, AwsCustomResource comes in.

Service service = ...;
String region = ...;
Vpc vpc = ...;

NetworkLoadBalancer balancer = NetworkLoadBalancer.Builder.create(this, "NetworkLoadBalancer")
 .loadBalancerName("nlb")
 .vpc(vpc)
 .internetFacing(false)
 .crossZoneEnabled(true)
 .vpcSubnets(subnetSelection)
 .securityGroups(securityGroups)
 .ipAddressType(IpAddressType.IPV4)
 .build();

// look up ips for the enis
// we cannot ask the endpoint for the number of enis via getVpcEndpointNetworkInterfaceIds() as it returns a single array token
// not representative of the actual number of enis
// we assume one ip per subnet given to the endpoint
List<String> outputPaths = IntStream.range(0, subnetSelection.getSubnets().size())
 .mapToObj("NetworkInterfaces.%s.PrivateIpAddress"::formatted)
 .toList();

// we need to use a custom resource to get the private ip addresses of the network interfaces
AwsCustomResource customResource = AwsCustomResource.Builder.create(this, "EniLookup")
 .functionName(physicalName)
 /// do not do this. will cause the stack to hang on deployment, and prevents an easy deletion
 // .vpc(vpc).vpcSubnets(subnetSelection)
 // actually installs during the event, let's rely on the installed version
 .installLatestAwsSdk(false)
 .timeout(Duration.minutes(5))
 // by default onCreate calls onUpdate, so we only need to define onUpdate
 .onUpdate(AwsSdkCall.builder()
 .physicalResourceId(PhysicalResourceId.of(physicalName))
 .service("EC2")
 .action("describeNetworkInterfaces")
 .parameters(OrderedSafeMaps.of(
 "NetworkInterfaceIds", endpoint.getVpcEndpointNetworkInterfaceIds()
 ))
 .outputPaths(outputPaths)
 .build())
 .policy(AwsCustomResourcePolicy.fromStatements(List.of(
 PolicyStatement.Builder.create()
 .actions(List.of("ec2:DescribeNetworkInterfaces"))
 .resources(List.of("*"))
 .build()
 )
 )
 )
 .build();

List<IpTarget> ipTargets = outputPaths.stream()
 .map(customResource::getResponseField)
 .map(IpTarget::new)
 .toList();

NetworkTargetGroup targetGroup = NetworkTargetGroup.Builder.create(this, "TargetGroup")
 .targetGroupName("app-runner")
 .port(443) // the port apprunner presents
 .protocol(Protocol.TCP)
 .vpc(vpc)
 .targetType(TargetType.IP)
 .targets(ipTargets)
 .build();

NetworkListener networkListener = balancer.addListener(name.with("Listener").camelCase(), NetworkListenerProps.builder()
 .protocol(Protocol.TCP)
 .port(1001) // arbitrary, but don't duplicate across multiple listeners on this balancer
 .defaultTargetGroups(List.of(targetGroup))
 .loadBalancer(balancer)
 .build());

You will need the listener ARN, networkListener.getListenerArn(), in a later step.

Now we create the API Gateway V2.

HttpApi httpApi = HttpApi.Builder.create(this, "HttpApi")
 .apiName("api")
 .createDefaultStage(true)
 .disableExecuteApiEndpoint(true) // turns off the default endpoint, optional
 .build();

You may find it useful to customize the logging.

LogGroup logGroup = LogGroup.Builder.create(this, Label.of("LogGroup").with(id).camelCase())
 .logGroupName("/aws/apigateway/" + name.lowerHyphen())
 .removalPolicy(RemovalPolicy.DESTROY)
 .retention(RetentionDays.FIVE_DAYS)
 .build();

((CfnStage) httpApi.getDefaultStage().getNode().getDefaultChild()).setAccessLogSettings(CfnStage.AccessLogSettingsProperty.builder()
 .destinationArn(logGroup.getLogGroupArn()
 .format("$context.identity.sourceIp - - [$context.requestTime] \"$context.httpMethod $context.routeKey $context.protocol\" $context.status $context.responseLength $context.requestId")
 .build());

In this last step, we integrate the API Gateway to the balancer listener via a VpcLink over a RESTful route. Obvious, right?

String host = ...; // the domain name from the ingressConnection
String listenerArn = ...; // from the listener

// if VpcLink.Builder.create is used instead, api gateway returns 503
VpcLink link = httpApi.addVpcLink(VpcLinkProps.builder()
 .vpcLinkName("vpc-link")
 .vpc(vpc) // where the apprunner service resides
 .subnets(subnets)
 .securityGroups(securityGroups)
 .build());

NetworkListener listener = NetworkListener.fromNetworkListenerArn(this, serviceName.camelCase(), listenerArn)
HttpNlbIntegration integration = HttpNlbIntegration.Builder.create("app-runner", listener)
 .method(HttpMethod.ANY)
 .parameterMapping(new ParameterMapping()
 .overwritePath(MappingValue.custom("/${request.path.proxy}")) // see path below
 .overwriteHeader("Host", MappingValue.custom(host)) // since this is over tls, need to set the host header
 )
 .vpcLink(vpcLink)
 .secureServerName(host) // since this is over tls, we need to set the server name
 .build();

String path = "%s{proxy+}".formatted("/api/"); // translate /api/.... to /....
AddRoutesOptions apiRoute = AddRoutesOptions.builder()
 .path(path)
 .methods(List.of(HttpMethod.ANY))
 .integration(integration)
 .build();

Without setting secureServerName(), a 503 error will be returned.

Without setting .overwriteHeader("Host", MappingValue.custom(host)) a 404 error will be returned. This particular issue is quite maddening since the back-end hosted service isn’t returning the 404, it’s returned from AppRunner (envoy) because it doesn’t recognize the host header passed by the client to the API Gateway.

Of note, by testing API endpoints with curl -sD - you can see the returned headers. If the server header has the value envoy, I’m going to say you can assume AppRunner is in the request path. envoy also is returned for 200 responses. It would be wise to replace this with the public facing domain name.

Network security issues are harder to sort out. For those you’ll see a 503 response, and if you check the logs in CloudWatch, look for the $context.error.responseType value of INTEGRATION_NETWORK_FAILURE. You’ll need to add this variable above to the logging format – see this list of other useful logging variables.

For simplicity I left out a number of constructs and configuration, but it’s reasonable to drop everything into the same VPC, subnets, and security group, in order to get things working.

Also note I also have some clusterless-commons API calls littered about in the above code.

Github issues of note:

CloudWatch Log Export

Wed, 20 Dec 2023 00:00:00 +0000

AWS services send their logs to CloudWatch whether they are configured to do so or not. AWS StepFunctions pushes all transition events to CloudWatch under the log group /aws/vendedlogs/states/*. Even if your code is wrapped in their service, your logs will show up there as well. AWS Batch will send stdout logs to log group /aws/batch/*.

Note, if you are using the AWS CDK or CloudFormation, you will need to declare these log groups directly in order to configure the retention period and removal policy as they will be created for you automatically. See LambdaLogGroupConstruct for an example.

You can write your code to push your logs elsewhere, but for provider services, CloudWatch will be the interface unless you export your log group into AWS S3 via the CreateExportTask.

Clusterless now provides an Activity (think cron job) specifically for exporting a declared CloudWatch log to an S3 bucket, aws:core:cloudWatchExport.

Unfortunately, there is a quota on this task: One active (running or pending) export task at a time, per account. This quota can't be changed.

This doesn’t mean the task is unusable, but it now becomes tricky. If there is already a task in play, a LimitExceededException will be thrown. It is unfortunate that this quota limit isn’t promoted up into the docs.

Now the question becomes, how many CloudWatch log groups can be exported via cron like processes (via a Clusterless Activity) within an account?

An Activity requires as a parameter the frequency in which to create the export task, say every 5 or 10 minutes. The Activity will set the from and to timestamps for the previous interval of arrived logs. If the frequency is 10 minutes, then the prior 10 minute interval is exported.

Assuming the task creator/submitter (an AWS Lambda in the Clusterless case) will retry failed task attempts, how long should the retry wait – until trying again so that it’s more likely to succeed and not cause any API rate limiting. This is in part a function of how long a task takes to complete, and how many competing Activities there are.

The describe-export-tasks aws cli command will export all export task metadata. This example here uses data from only one Activity (exporting one log group), where the log data is quite small.

aws logs describe-export-tasks | jq '.exportTasks[].executionInfo | .completionTime - .creationTime' | ministat -n
x <stdin>
 N Min Max Median Avg Stddev
x 2663 1549 608539 2326 7376.5332 37206.281

As can be seen above, the median (p50) is 2.3 seconds. But a 10 minute duration was also encountered.

Unfortunately, any given Lambda attempting to submit a successful create export task command only has (up to) 15 minutes in which to have a success. If another Activity was exporting a different log group, and it was submitted milliseconds before the current Activity, the current Activity may need to wait (continue retrying) for 10 minutes before it can succeed and exit. This would be an expensive blocking operation.

Currently Clusterless does nothing clever to maximize the number of possible log groups that can be exported in an account. Clusterless only uses an exponential backoff retry policy starting at 3 seconds up to a maximum wait of 5 minutes. This all bounded into the Lambda configured timeout. That is, retries will continue until the Lambda function was configured to stop (timeout) so that log data can be emitted and the Lambda invocation marked as a failure.

A more clever arrangement can be devised in order to reliably support multiple log group exports at without costly blocking operations. Reach out if this would be of value.

Naming Things in AWS

Wed, 13 Dec 2023 00:00:00 +0000

Things in AWS frequently have names. Things like AWS S3 buckets or AWS Lambda Functions. And when using the AWS CDK, Constructs have names so that they can be referenced elsewhere in code or error messages.

I rely on four scopes to name things, what scope applies to what resource is only discovered in docs or trial-and-error.

For example, S3 bucket names are global. If two users each in unique accounts in unique regions attempt to create a bucket with the same name, the first to try wins. Sometimes these names are called “physical ids”.

On the otherside of the spectrum, if you are creating a new AWS CloudFormation stack, it must have a unique name in the account and region (us-west-1). The name could be used in another region concurrently.

The naming scopes I work with are:

global: e.g. S3 buckets
account: e.g. Elastic Container Registry (ECR)
region: e.g. Virtual Private Cloud (VPC)
stack: e.g. any Construct

I sometimes throw in an orthogonal name for the current lifecycle stage of the deployment. For example, “dev”, “uat”, or “prod”. This further prevents crossover or issues within a shared account (for those at companies that limit the number of AWS accounts granted to developers).

To further confuse matters, names have a kind of de facto format. When naming a Construct, CamelCase is frequently used. And when naming a bucket, lower-hyphen format is used.

If you consider many names are contextual, making them hierarchical, it makes sense to have a simple API that lets you build a name, then apply it where it’s needed.

The project clusterless-commons provides a Label interface for building labels.

Label label = Label.of("lower").with("case");
Assertions.assertEquals("LowerCase", label.camelCase());
Assertions.assertEquals("lowerCase", label.lowerCamelCase());
Assertions.assertEquals("lower-case", label.lowerHyphen());
Assertions.assertEquals("lower/case", label.lowerHyphenPath());

Sometimes you may not want any formatting applied, the Fixed class locks an element of the final label.

Label label = Label.of("DataRepo").with(Fixed.of("us-west-2"));
Assertions.assertEquals("data-repo-us-west-2", label.lowerHyphen());
Assertions.assertEquals("data-repo/us-west-2", label.lowerHyphenPath());

When using AWS regions in a label, it may be easier to simply use Region.of("us-west-2"), this way a custom Construct and require region information on the constructor by declaring a parameter type as Region.

There are a few other useful Label derivatives in the javadoc.

If you are willing to adopt the base classes ScopedApp, ScopedStack, and ScopedConstruct, naming things becomes a little simpler.

The class ResourceNames has a few helper methods that return a Label with the requested scope.

ResourceNames#globalUniqueLabel()
ResoruceNames#accountUniqueLabel()
ResourceNames#regionUniqueLabel()

Each of these take a Construct and attempt to find the parent ScopedApp for the relevant metadata.

Respectively, you can end up with names like:

Globally unique:

Label region = Region.of(Stack.of(scope).getRegion());
Label account = Fixed.of(Stack.of(scope).getAccount());
Label stage = ScopedApp.scopedOf(scope).stage();

return stage.upperOnly()
 .with(name)
 .with(account)
 .with(region);

Account unique:

Label region = Region.of(Stack.of(scope).getRegion());
Label stage = ScopedApp.scopedOf(scope).stage();

return stage.upperOnly()
 .with(name)
 .with(region);

Region unique:

Stage stage = ScopedApp.scopedOf(scope).stage();

return stage.upperOnly()
 .with(name);

Note, most names may have a size limit. Lambda function names are limited to 64 characters. Label currently does nothing to enforce this, but it could be added. But Label does support abbreviations via Label#of("LongValue", "abb");.

Links:

Speedup AWS CDK Updates

Mon, 11 Dec 2023 00:00:00 +0000

A great feature of the AWS CDK (inherited from AWS CloudFormation) is the ability to update a stack in place. This is accomplished by only sending changes across to CloudFormation (the changes can be viewed by calling diff instead of deploy).

Unfortunately, a number of artifacts may register as having changed and be re-deployed, even though the only difference between the artifact might be a timestamp on a file. Or the order of the metadata may have changed.

If you are using the Java CDK bindings, many Construct builders take java.util.Map instances that contain configuration information. Unfortunately the java.util.HashMap does not retain the order of items added, nor does it guarantee an stability of that order across executions (or JDK versions). java.util.LinkedHashMap does retain the order, so when creating Constructs that expect a Map of configuration values, remember to use LinkedHashMap.

That said, java.util.Map#of(...) is a very convenient convenience function for creating inline Map instances without all the boilerplate of Map<String,Object> config = new LinkedHashMap(); //etc, but Map#of returns a HashMap instance.

Map<String,Object> config = Map.of(
 "key1," "value1",
 "key2," "value2"
);

The project clusterless-commons provides a OrderedMaps#of(...) method that duplicates the Map#of method, but returns a LinkedHashMap instead.

Map<String,Object> config = OrderedMaps.of(
 "key1," "value1",
 "key2," "value2"
);

Additionally, the sister class OrderedSafeMaps provides an #of(...) method that will drop a key if the associated value is null.

Map<String,Object> config = OrderedSafeMaps.of(
 "key1," null, // ignored
 "key2," OrderedSafeMaps.ifPresent(value2, String::toLowerCase), // mutate if not null
 "key3," OrderedSafeMaps.ifPresent(value3, String::toLowerCase, defaultValue),
);

You can read more about these classes here.

Links:

Cascading 3.0 Query Planner

Thu, 18 Sep 2014 19:51:10 +0000

Original post: https://www.cascading.org/2014/09/18/the-cascading-3-0-query-planner/

Cascading 1.0 when released, represented a huge milestone. An enterprise friendly Java API, not a syntax, and fail fast planner allowed developers to build robust, maintainable, data-oriented applications that could execute reliably on Apache Hadoop for hours or days.

Cascading 2.0 made a nod towards cluster platform independence with the addition of an in memory ‘local’ mode that proved applications could be built without direct API access to the underlying platform. This provided much needed isolation across the evolving Hadoop platform and improved developer experience. Write once, run on any vendor provided platform.

Cascading 3.0 proves we can abstract away the underlying platform by providing additional platforms users can execute against, not just MapReduce. Apache Tez is the first such addition. Tez is the latest parallel computation fabric from the Apache group. It builds on Hadoop, keeping the Hadoop File System for instance, but uses a model that removes inefficiencies that are baked into the MapReduce model.

But we should point out the Cascading 3.0 abstraction is not just over the platform APIs, but also the semantics of the given platform like MapReduce or a DAG like model (like the one provided by Apache Tez). This is achieved by the Cascading “query planner”, it maps from one model to another.

Another way to restate this is that the Cascading query planner is not ‘mapping’ MapReduce onto other models, but mapping the Cascading model directly onto the underlying platform model, be it MapReduce or a DAG representation.

This makes the Cascading object model and their associated semantics an Intermediate Representation (or Intermediate Language) for data parallel cluster computing. Where new models can be created on top, over Cascading, and new ones can be mapped below, underneath Cascading. All the while retaining the qualities of both. LLVM provides this functionality to the compiler world. For example, Clang’s C, C++, and Objective-C; the Julia language (a technical computing language); and the Swift language from Apple all rely on the LLVM provided Intermediate Representation (IR).

To keep things less ivory tower, we usually call Cascading a framework.

But in practice, Cascading as an IR is quite common. Cascading is the basis for Lingual, Pattern, Scalding, and Cascalog, each of which are higher order syntaxes or languages that are feature rich, reliable, and very useful unto themselves. Cascading is also being embedded in a number of commercial products that expose a different (more vertical) model to the end user.

How is this done? In large part, it’s due to the brand new query planner we have spent the balance of this year developing and testing. First with our ‘local’ mode and the MapReduce model, which were made available earlier this year in a Cascading 3.0 WIP release. And during the last couple of months, adding more degrees of freedom to support a model with, well, more degrees.

We could have just used the MapReduce planner directly on Tez, but the point of the DAG model in Tez is to remove the inefficiencies inherent to MapReduce. For example, no more ‘identity mapper’; added support for multiple outputs; no prefixing data with join ordinality; suppressing sorting when not needed; removing HDFS as an intermediate store between jobs; etc.

Over the years we realized we need to rethink how the prior planners mapped Cascading into new platform models, but more importantly, how we can move control of adding new Cascading primitives, optimizations, or whole new platforms back to the Cascading user. The answer was somewhat obvious when looking at similar systems — create a rule language users can use to create rules and to create new rule sets.

But there are some reasonably complicated difficulties we had to address. Recognizing that what a Cascading user creates is itself a DAG (Directed Acyclic Graph) of elements that the user would like to execute remotely is key.

From there, the key difficulties we had to address boiled down to:

Allowing rules to be abstract enough so that they can target patterns of nodes and edges in the user DAG without having to account for all possible permutations
Isolating the structural and compatibility testing of the target DAG from the structural transformation of the DAG
Allow for high levels of code reusability
Allow rules to offer syntactic context on errors
Allow rules to leverage all available meta-data, including the structural qualities of the DAG itself
Present a simple ‘language’ that allows users to develop custom rules

Most of the above translates to having “rules” that can assert a DAG is a valid structure, transform a DAG into a new DAG, or partition a DAG into sub-DAGs.

In terms of MapReduce, we want to define rules that verify aggregations always follow a grouping; identify and group “map” side and “reduce” side elements; insert temporary files between the “reduce” side elements and the “map” side elements; break the graph up into physical jobs.

The root of the solution is to rely on a way to match sub-DAGs (or sub-graphs) within a larger graph. That is, we need a regular expression like “language”, not for strings and text, but for graphs of nodes and edges. In the literature this is called Sub-Graph Isomorphism Matching.

So we created what we call the Expression Graph API, a basis for building higher order complex logic allowing us to reason about or manipulate large complex graphs.

Fundamentally what we have is an API that allows a ‘match graph’ (think regular expression string) to be constructed that then can be applied to a larger graph to see if there is a match or, more generally useful, similarity between both structures. And if so, we can then perform an Assertion, Transformation, or Partition against the matched elements.

Consider the expression graph below, a simple DAG of two elements with some meta-data:

It will match the following elements in gray and not those with a dotted outline.

The above expression graph knows it is looking for a circle that is also a ‘join’ between two incoming elements. If there is no ‘join’, then don’t match.

But the world isn’t that simple. We had to go further to allow for matching of sub-graphs of distinguished elements.

That is, some elements in a graph can be ignored in a given case, but not in others. Thus the elements that matter, the distinguished elements, must be the only ones that match the graph expression. But the transformation must be applied to the full sub-graph bound by any matched distinguished elements.

Consider the following expression graph, a circle or a triangle will match:

It will match the gray elements in the following graph:

These would be the distinguished elements. Matching them allows us to remove all the other elements, or actually, mask them from subsequent matches, resulting in a contracted graph of the form:

Now we can apply a new expression graph (not shown) to find our target sub-graph of distinguished elements, the result is as follows:

If we isolate the new sub-graph, above in gray, we can expand it to include it original intermediate nodes and edges, that is, unmask previously masked elements.

Finally we can provide yet another expression graph to assert or transform this final graph within the original full graph. In the graph above, we could provide a new expression graph looking for a square with horizontal lines, and if found we an insert a new element after it, along the dotted edge.

Alternately, we can extract the new sub-graph from the full original graph effectively letting us partition the larger graph into smaller ones.

We have only touched on the basics, but the language is still evolving while already quite powerful.

One last value of having assertions, transformations, and partitioning implemented as basic building blocks for re-usability; users can create new “rule sets” to optimize specific applications, or create a set of “rule sets” with a voting strategy to choose the best resulting execution plan based on cost or some other metric.