Serverless Event Driven orchestration?

The beauty of Serverless architecture is the its simplicity. You can do a lot of functionality based on a simple event got triggered from a single function. Basically, this, lets say workflow, gets triggered when for example a file was committed into a specific S3 bucket.

But, when it comes to complex workflows, its getting more challenging to maintain a healthy, successful workflow that accomplish the needful. The simple way is Lambda function Destination, which gets triggered when the main function execution completed. Of course you can trigger is when the function got completed successfully or failed (with keeping in consideration code skills to handle the function failures). But this is not an ideal scenario for complex workflow.

Lets mention several services and the sequence of evolve to achieve the best orchestration on my opinion:

SQS:

Simple Queue Service is one of the great options you need to keep in consideration. Simply it serve a waiting station lets say for the functions. You can send the event payload into the queue and then control many aspects for the execution like:

• Waiting time before consuming the message: How long should the event wait before it gets available for a function to consume it.
• FIFO: whether your workflow should maintain first in first out for any sequence execution needed.
• DLQ: if your function failed to consume the message, then send it to an investigation queue, which will hold the message for you to do your code or your sequence check. This is really good point for SQS.

But, making SQS as a trigger point for other Lambda functions can be overwhelming as you need to maintain multiple aspects for that. What can be done?

StepFunctions:

Stepfunction is a service that can work as your workload composer. This service is great to coordinate the sequence of function execution in your workload. Why this is a great service?

• State Machine: Order the execution of your functions in the order you need. What should be executed first and what should come next. Each function Payload feeds the other function to proceed with your execution.
• Parallel Execution: If one function payload suits multiple functions input, and the workflow can have multiple functions to be executed at the same time then this is a BIG plus for StepFunctions.
• A lot of compatible services: You can integrate Many services in your workflow as the compatibility is available. In fact, you can integrate DLQ from SQS into your workload as well.

This is great, but can this get better?

EventBridge:

EventBridge is an Event Bus service that has some points in common with StepFunctions but can serves wider range of workflows. Lets see why this is really good service:

• Event Routing: this is the main routing point for the service. This simply can redirect the payload into the desired destination even if the destination wasn’t Lambda function. Thanks to the set of services that are compatible with it.
• Schema Registry: This is the part that is responsible about identify, classify and trigger rules in EventBridge that can leverage the Routing service to redirect payloads into destinations. New schema version for a specific function? No problem, schema versioning is in place.
• Event Archive: This is really important feature and let me tell you why? This can make you reconcile missing events or missing execution that failed for any reason like concurrent limit exceeded or network interruption.

To summaries:

Serverless orchestration is something grow with you. You will not find the ideal scenario from day one or year one. As you can see from the sequence I mentioned the services in that not a single service is going to be an ideal solution for your workflow. Invest time in understanding these services, mirror it over your workflow, and construct what you need. I know its intimidating to leverage this much of cool services in one application but this is something you should get away from. Do not couple your workflow or at least if needed, do not do it too much.