Trouble Is My Workflow · YokerHood

The rain was hitting the asphalt outside with the dull, persistent rhythm of an unhandled exception in an infinite loop. It was past eight, the coffee in my mug was cold enough to preserve fish, and the only light in the office came from three flickering monitors and an incandescent exit sign that buzzed like an angry hornet in a tin can. It was 1998.

That was when the architect walked in.

You know the type. Rimless glasses, an ironed flannel shirt that had never seen manual labour, and the kind of earnest, predatory smile usually reserved for people selling whole-life insurance or enterprise service buses. He dropped a four-inch design document onto my desk. It landed with the sullen thud of a wet phonebook.

“We need a generic workflow engine,” he announced, resting two manicured hands on the edge of my desk. “Something dynamic. Decoupled. Highly extensible. A unified platform to orchestrate our business processes across distributed microservices.”

I looked at him through the steam of the fresh cup he hadn't offered to buy me.

“You don’t need an engine,” I said. “You need an honest finite-state machine, and you’re too fancy-pants to admit it.”

He flinched as if I’d insulted his mother, adjusted his glasses, and began to mumble about directed acyclic graphs. I lit a cigarette (these blessed years when you could smoke in the office,) pointed at the door, and waited for him to leave before I opened the file.

The Autopsy of a Buzzword

In our trade, “workflow” is the word people use when they want to charge six figures for a product or excuse three quarters of architectural delinquency. It sounds respectable. It sounds like something an executive can nod at during a quarterly business review without having to know what a pointer does.

Strip off the sales brochure, wipe the mud off the radiator grille, and look at the chassis underneath. What is a generic workflow, really, when nobody is trying to sell you a cloud subscription?

At its bare brass tacks, a workflow is nothing more than five distinct elements cohabiting in a trench coat:

A Finite Set of States. Your process is always somewhere . Not everywhere at once, not halfway between two ideas like an indecisive juror, but occupying exactly one discrete coordinate in its lifecycle.
A Discrete Set of Events. Things happen. An alarm goes off, an HTTP webhook knocks on the door at two in the morning, a human reviewer clicks a green button, or a payment processor shrugs and gives up.
A Transition Topology. The rules of the alley. If you are standing in State A, there are only specific doors unlocked. You may step through Door 1 to State B, or Door 2 to State C. You cannot phase through the brick wall into State Q just because the product manager had a sudden vision in the shower.
A Context Payload. The baggage the traveler carries. The pull request number, the customer’s delivery address, the current temperature in Minneapolis. The luggage travels with the workflow, but it does not drive the car.
Invariants Governed Over Time. Rules that do not bend, no matter how hard the wind blows or how loud the VP of Sales screams into the receiver.

If that list sounds familiar, it should. That is the formal definition of an extended Finite State Automaton. It was worked out back when computers were the size of municipal water heaters and programmers wore ties to work.

Yet, for reasons that keep therapists and cloud vendors in business, the modern software industry spends billions of dollars pretending this identity does not exist.

How We Murder State in the Dark

The average engineering team approaches a workflow the way a cheap criminal approaches a bank vault: without a blueprint, armed with a crowbar, and hoping nobody asks hard questions until the money is spent.

There are three classic ways this crime is committed in modern codebases.

① The Polite Script (The Straight-Line Delusion)

The engineer begins with a pipeline. In Elixir, they write:

def run(input) do
  input
  |> validate()
  |> charge_card()
  |> pack_box()
  |> notify_customer()
end

It looks elegant. It reads like poetry written by a civil servant. And it is completely divorced from the physical universe.

In the real world, the credit card is maxed out. In the real world, the warehouse is on fire. In the real world, the customer changed their mind five minutes after clicking the button and wants their money back while the parcel is sitting on a loading dock in New Jersey.

The moment reality kicks the door down, the clean pipe fractures into a sprawling tangle of nested case statements, early returns, and rescue blocks. What was supposed to be a straight highway becomes a snake pit.

② The Database Stew (Status-Driven Denial)

When the script breaks, the team “architects” a database solution. They take a PostgreSQL table and give it a status column.

Then they realize an order can be paid for, but pending fraud review. So they add fraud_review_pending: boolean . Then someone discovers an order can be partially refunded while in transit, so they add partially_refunded: boolean and transit_hold: boolean .

Within six months, the schema looks like an uninspected electrical switchboard:

schema "workflows" do
  field :status, :string, default: "pending"
  field :paid, :boolean, default: false
  field :reviewed, :boolean, default: false
  field :shipped, :boolean, default: false
  field :cancelled, :boolean, default: false
  field :refunded, :boolean, default: false
  field :on_hold, :boolean, default: false
end

Count the representable combinations. Seven statuses multiplied by six boolean flags gives you hundreds of distinct states your database will swallow without blinking.

Of those hundreds, perhaps six correspond to something that could happen to a physical object without breaking the laws of thermodynamics. The other hundreds are ghosts: orders that are simultaneously shipped: true , paid: false , and cancelled: true . Rows like that sit in your database like counterfeit bills in a cash register—silent, illegal, and waiting to ruin someone’s morning when the audit comes due.

③ The Amnesiac Message Queue

The third approach is the microservice special. The team splits the workflow into six workers listening to a message bus.

Worker A does a job, writes to the database, and emits an event. Worker B reads the event, reads the database, checks five conditional flags, does another job, and emits another event.

Nobody has a map of the process. The entire lifecycle is smeared across twenty-four Git repositories like grease on a diner counter. To answer a simple question— “What happens if the customer cancels while we are waiting for the courier?” —you need to interview four engineers, read twelve YAML files, and consult an astrologer.

There is a better way. In fact, there has always been a better way. Let’s look at three real-life cases from the desk blotter.

Case File: The Task Lifecycle (The Finite Trudge)

Consider the most familiar trudge in our corner of the city: moving a feature from an engineer’s brain to production.

The PM thinks it’s a line: state task → code → test → commit → review → merge.

Anyone who has survived a single sprint knows that line is a lie. Real engineering is full of hairpin turns and backward loops.

Here is what the lifecycle actually looks like when you shine a hard light on it:

stateDiagram-v2
    [*] --> Stated: define_requirements
    Stated --> InDevelopment: pick_up
    InDevelopment --> Committed: run_tests_and_commit
    Committed --> UnderReview: open_pull_request
    UnderReview --> InDevelopment: request_changes
    UnderReview --> Approved: lgtm
    Approved --> Merged: merge_to_trunk
    Merged --> [*]
    
    Stated --> Abandoned: discard
    InDevelopment --> Abandoned: discard
    UnderReview --> Abandoned: discard
    Approved --> Abandoned: discard
    Abandoned --> [*]

Notice what is happening here:

The Review Loop is First-Class. When the senior engineer leaves twenty-seven comments asking why you didn't use Enum.reduce_while/3 , you don’t crash the pipeline or manufacture an error state. You transition from UnderReview back to InDevelopment on the event request_changes . It is an expected, legitimate transition.
The Eject Button Exists Everywhere. A task can be discarded from almost any stage. Marketing can cancel the initiative while you are halfway through writing the tests. If you model this with explicit states, Abandoned is a clean terminal state that frees resources and logs the autopsy. If you model it with flags, you end up with orphaned branches and pull requests that sit open until the heat death of the universe.
Illegal Moves Are Inconceivable. Can you merge a task directly from InDevelopment without an approval? Not unless there is an arrow connecting them. The machine does not ask you to remember the policy; the topology is the policy.

Case File: Order Fulfillment (The Transactional Gauntlet)

Now take commerce. An order is placed, money changes hands, goods move across the map.

stateDiagram-v2
    [*] --> Placed: checkout
    Placed --> Authorized: authorize_payment
    Placed --> Failed: payment_declined
    
    Authorized --> Packing: claim_inventory
    Authorized --> Cancelled: cancel_order
    
    Packing --> Shipped: dispatch_courier
    Packing --> Cancelled: inventory_shortage
    
    Shipped --> Delivered: confirm_receipt
    Shipped --> LostInTransit: carrier_lost
    
    Delivered --> Refunded: return_items
    Cancelled --> Refunded: refund_authorization
    
    Failed --> [*]
    LostInTransit --> [*]
    Refunded --> [*]
    Delivered --> [*]

Look closely at the branches:

You cannot refund what was never authorized. In the old boolean stew, a bug in the webhook consumer could mark an order as refunded: true even if paid: false . In an FSM, :refunded can only be reached from states where money was actually held or captured ( Authorized , Delivered , Cancelled ).
Cancellation semantics change based on state. If the customer cancels while in Placed , it costs nothing. If they cancel while in Packing , the warehouse has to restock the shelf. If they cancel while in Shipped , you can't cancel at all—they have to wait for delivery and initiate a return. An FSM makes these rules self-evident because each state only listens to the events that make sense for its location.
Dead ends are accounted for. Packages get lost ( LostInTransit ). Cards get declined ( Failed ). These are not unhandled exceptions; they are terminal states.

Case File: The Eternal Beat Cop (The Infinite Loop)

Some people think workflows have to terminate—that if a process doesn't reach an end state, it’s defective, like a watch with no hands.

They haven't spent enough time watching the street. Half the machinery keeping the modern world turning runs in an infinite, unblinking loop: kiosks, telemetry collectors, sensor monitors, background pollers.

Take a classic: receive a weather forecast → show it → wait fifteen minutes → repeat.

stateDiagram-v2
    [*] --> Fetching: init
    Fetching --> Displaying: forecast_received
    Fetching --> Backoff: network_error
    
    Backoff --> Fetching: retry_timer
    
    Displaying --> Waiting: render_complete
    Waiting --> Fetching: refresh_timer

A novice looks at that loop and gets nervous. “Won’t the stack overflow? Won’t the memory swell up like a drowned corpse?”

In a traditional imperative language with recursive calls or unchecked event listeners, quite possibly. But in an honest, process-backed finite-state machine (such as an OTP GenServer running under Finitomata ):

The Stack Never Grows. State transitions are tail-recursive loops or GenServer callback returns ( {:ok, :displaying, payload} ). The machine sheds its past with every step. It can cycle two billion times without consuming an additional byte of RAM.
Time Is Just Another Event. Notice refresh_timer and retry_timer . Time is not a rogue cron job breaking in through the fire escape and mutating rows in your database behind your back. Time is an internal event generated by the machine itself.
Self-Healing Is Built-In. When the weather API returns a 503 Gateway Timeout, the machine doesn't throw a fit and die; it slips into the Backoff state, sets a short timer, and tries again.

It is the eternal beat cop walking the neighborhood: check the door, see it's locked, wait under the streetlamp, walk the block again. Rain or shine, day or night.

The Detective’s Rules: How to Model a Workflow Properly

If you want your workflows to hold up when the district attorney starts asking questions, you need discipline. Here are the rules I live by:

① The Blueprint Precedes the Brick

Before you write a single line of business logic, declare the topology. If you cannot draw your workflow on a napkin using simple state-and-event syntax, you do not understand the business problem.

Using Finitomata , the declaration is not an afterthought or an export—it is the source code itself:

defmodule WeatherStation.FSM do
  @fsm """
  fetching --> |forecast_received| displaying
  fetching --> |network_error| backoff
  backoff --> |retry_timer| fetching
  displaying --> |render_complete| waiting
  waiting --> |refresh_timer| fetching
  """

  use Finitomata, fsm: @fsm, timer: 60_000
end

The string above is compiled directly into a validated state machine. If an event arrives that isn't declared for the current state, it doesn't corrupt your data; it gets rejected with an explicit error.

② State Is the Coordinate; Payload Is the Luggage

Keep your states small, finite, and enumerable. A state is an atom: :idle , :under_review , :waiting .

The data associated with that state—the reviewer’s notes, the temperature reading, the authorization token—belongs strictly in the payload .

Never, under any circumstances, allow data to disguise itself as a state:

Wrong: :in_review_by_sarah , :in_review_with_three_edits
Right: State: :under_review , Payload: %{reviewer: "Sarah", edit_count: 3}

State determines what can happen next . Payload provides the context for how it happens . Conflating the two is the root of all evil.

③ Events Knock; They Do Not Dictate

External actors do not change your state. An HTTP request does not get to say order.set_state(:shipped) .

External actors emit events . The state machine receives the event, verifies that the transition is legal from its current coordinate, executes the business callback, and moves itself.

If the shipping department sends a :dispatch event for an order that is still :placed (unpaid), the machine laughs in their face, logs the attempt, and stays right where it was.

④ Failure Is a Destination, Not a Panic

In cheap dime-store novels, people get hit over the head, lose consciousness, and wake up fine twenty minutes later. In production systems, exceptions don't work that way. An unhandled exception kills the thread, corrupts state, and leaves the on-call engineer staring at a pile of ash.

Every state that touches an external boundary—a network call, a disk write, a human approval—must account for failure as a first-class transition. Model the failure states: :payment_declined , :review_rejected , :backoff . Give failure a roof and a bed, and it won't burn your house down.

⑤ One Process, One Life

If you’re running on the BEAM, every active workflow instance should be its own isolated process.

Order #1042 gets a GenServer. Pull Request #88 gets a GenServer. Weather Station #4 gets a GenServer.

When an entity is backed by an isolated process:

Incoming events are naturally serialized in the process mailbox. The concurrency race condition—where two web requests simultaneously attempt to modify the same record—vanishes by design.
If one workflow instance encounters a catastrophic error, it crashes alone, supervised, leaving the other fifty thousand running instances untouched.
State lives in fast, in-memory Erlang terms, backed to disk only at transition boundaries.

Epilogue: The Long Goodbye to Spaghetti

The rain had stopped by three in the morning. The streets of the city were black mirrors reflecting the streetlights.

The four-inch design document from the architect was still on my desk, right next to the empty coffee mug. I opened it to page one, took out a red marker, and struck through thirty-seven pages of “dynamic plugin orchestrators” and “configurable JSON execution pipelines.”

In their place, I drew six circles, eight arrows, and a dozen clear labels.

A workflow is not a mystical creature requiring enterprise middleware. It is a finite automaton in a damp trench coat, waiting for you to stop lying to yourself.

Give it a name. Draw its borders. Lock the doors that shouldn't be opened. You’ll sleep better at night, and when trouble comes knocking at your door—as it always does—you’ll know exactly which state you’re in.