Event-Driven Pipelines on AWS Lambda and Step Functions

Most recurring data work starts life as a cron job. It runs at 2am, it mostly works, and nobody looks at it again until it doesn't. The trouble starts once you have more than one of these jobs and they depend on each other. At that point cron stops being a scheduler and turns into a liability. Here's why I moved our recurring workloads to an event-driven model on AWS instead, and how the pieces actually fit together.

Why not just cron?

A schedule answers the question of when. Production pipelines actually care about after what. A file lands, a message arrives, an upstream job finishes: those are the real triggers. Modeling them as time offsets is just guesswork, and guesswork tends to fail silently.

Event-driven pipelines flip that around. Each step reacts to something that actually happened, not to a clock. Once it's set up this way, a few things get noticeably better:

• No idle polling. Work starts the instant its input exists.
• Natural retries. Failed events go to a queue and get retried, or parked in a dead-letter queue.
• Observability without extra effort. Every transition is an event you can trace.

The building blocks

The stack I reach for is deliberately small:

• EventBridge: the nervous system. Rules route events to whatever needs to react to them.
• Lambda: stateless units of work. Pull, transform, validate, done.
• Step Functions: orchestration once a job grows past two or three stages, picks up branching, or needs state a human can actually read.
• SQS + DLQ: buffering, plus a way to isolate failures between stages instead of letting them cascade.

A minimal state machine

Once a pipeline needs branching or retries, I pull the orchestration out of Lambda entirely and put it into a Step Functions state machine. The definition is declarative, so it becomes just another thing reviewers can actually read in a pull request:

{
  "StartAt": "Ingest",
  "States": {
    "Ingest":    { "Type": "Task", "Resource": "arn:...:ingest",
                   "Retry": [{ "ErrorEquals": ["States.ALL"], "MaxAttempts": 3 }],
                   "Next": "Transform" },
    "Transform": { "Type": "Task", "Resource": "arn:...:transform", "Next": "Validate" },
    "Validate":  { "Type": "Choice",
                   "Choices": [{ "Variable": "$.rows", "NumericGreaterThan": 0, "Next": "Publish" }],
                   "Default": "Alert" },
    "Publish":   { "Type": "Task", "Resource": "arn:...:publish", "End": true },
    "Alert":     { "Type": "Task", "Resource": "arn:...:alert",   "End": true }
  }
}

Every one of these resources gets provisioned through Terraform. Functions, rules, IAM, the state machine itself, all of it lives in version control and ships through CI, same as everything else.

My rule of thumb: once a workflow has branching, or more than two failure modes, it belongs in a state machine. Not in a chain of Lambdas calling each other and hoping for the best.

What it buys you

The payoff is honestly just calm. When something breaks, the failed execution sits right there in the console with the exact input that caused it, no digging required. Reprocessing means re-emitting one event. And because everything is code, spinning up a new environment is a terraform apply, not a runbook somebody wrote two years ago and forgot to update.

Cron still earns its place for genuinely time-based tasks, and I'm not trying to kill it off. But for anything that's actually reacting to the world, events are the more honest model, and AWS gives you the primitives to build them without running a single server yourself.