When you login to a website like Facebook, where do your eyes go first? The top of your feed? The stories banner?

Probably not.

Your eyes go straight to the toolbar to see if there is a little red badge showing you notifications or new friend requests. We can’t help it, we love notifications.

Every little notification we see fires a signal in our brains that gives us a sense of satisfaction. Honestly, we’re addicted to notifications. So what better than to implement that with our WebSocket?

If you’ve been following along, this is part four in an introductory series to WebSockets. Each part builds on the one before, so if you’re joining us now, I’d highly recommend reading the first three parts.

• Part One - Building a WebSocket
• Part Two - Securing your WebSocket
• Part Three - Documentation Using Async API Spec

Today we’re going to expand on what we’ve built to create user notifications.

Deploy To The Cloud

Just like the three previous parts, the work is done ahead of time. To deploy the updates that add user notifications, run the following command in a terminal in the root of the repo on your local machine.

git fetch
git checkout part-four
npm run deploy

This will checkout this portion of the walkthrough and deploy it to AWS. The updates included to support user notifications are:

• A lambda function to search for and post to specific user connections
• Dead Letter Queues (DLQs) for event delivery/processing failures
• CloudWatch alarm for notifying when a failure has occurred
• Updates to our Async API Spec that define new input and output events

Fortunately, adding user notifications to what we already had was easy. Our data structure allowed for a give me connections for a specific user access pattern, so the changes were simple.

Find The User Connection

With the WebSocket we built in a previous post, we added a lambda authorizer to the $connect endpoint that saved the userId of the connecting user to the database.

We can use this information to find the open connections for a user in order to send them push notifications. When a new Send User Push Notification EventBridge event comes in, we can query the database for all connections for that userId and send them a message.

Using the connection details to receive a push notification

In our data model, we save the user id as the pk in our GSI so we are able to run a query just on the user id that was provided in the incoming event.

{
  "pk": "<connectionId>",
  "sk": "connection#",
  "GSI1PK": "<userId>",
  "GSI1SK": "user#",
  "ipAddress": "<connecting ip address>",
  "connectedAt": "<epoch connected at time>",
  "ttl": "<time to live before connection is removed>"
}

The Global Secondary Index (GSI) allows us to query for all records in the database that have a GSI1PK of the userId. This allows us to get the connections to the WebSocket so we can pass along the message from the event. See lambda function.

Types of User Notifications

There are two different types of user notifications we can implement with our WebSocket connection: actionable and feedback notifications. Each notification serves a unique purpose when it comes to user experience.

Actionable Notifications

If we clicked on a notification in Facebook and it did nothing, it probably wouldn’t have such an addicting effect. But if you click on it and you’re taken somewhere to perform an action, then the notification has value.

The push notifications we’re sending to users via our WebSocket contain two pieces of data:

• A message to display to the user
• A callback the user can follow to perform an action

The message will inform the user something happened, the callback will let them do something with it. Take this event as an example:

{
  "message": "The XYZ report has finished processing and is ready for review.",
  "callback": "https://www.gopherholesunlimited.com/jobs/736ajdff7/results"
}

With this message, the user can see that the report they requested has finished processing. When they go to the callback url, they will see the results specifically related to that message.

Feedback Notifications

On the other hand, feedback notifications provide status updates. They would not add a little red badge next to a notification icon in a user interface.

If you were waiting on a long running process, a feedback notification would provide you with updates on where it is in the process. An example would be a percentage complete on an upload or a message saying what the job is doing. If we use the report example, a feedback user notification should look like:

{
  "message": "Calculating average time between status changes..."
}

Notice there is no callback in this notification. It is strictly informative and provides no action.

The intent of a feedback notification is to assure the end user the system is doing something.

It is always a good idea to remind the user that something is going on in the background so they don’t think something went wrong.

Error Handling With Push Notifications

Event Driven Architectures (EDAs) are notoriously difficult when it comes to error handling. Since we’re using serverless services, we’ve doubled down on the need for enhanced observability.

To help keep track of the state of our WebSocket, we have implemented Dead Letter Queues so we can drop events in a single location when something goes wrong. There are two types of issues we can experience in our WebSocket:

1. Event delivery failure
2. Event processing failure

Event delivery failures occur when EventBridge fails to put the event in our SQS queue or when the event fails to transfer from SQS to the processing lambda. This could be caused by a system outage or improper configuration. Regardless of what the root cause is, we send it over to a DLQ for us to monitor and triage. Once it is in the DLQ, we can investigate what is going on in the system and attempt to fix it.

To send an event delivery failure to a DLQ, we update our EventBridge rule to target the queue on a delivery failure.

Event processing failures occur in our code. We either have a bug in the code, the event doesn’t have the data we expect, or maybe a service we are calling in the code has been throttled or is experiencing a failure. Once again, we push these errors to a DLQ so we don’t have to be ready the instant something goes wrong.

To send an event processing failure to a DLQ, we make use of Lambda destinations to route the event on a failure.

How to Know When There Are Errors

Sending errors to a DLQ is one thing, but how do you know when something needs your attention?

To know when things are wrong, we have implemented a CloudWatch alarm that watches the DLQ. Whenever the DLQ has 1+ item in it, a SNS topic will fire and notify the concerned parties of a system failure.

Once the dead letter queue has been cleared of all items, the alarm will turn off and continue watching for the next incident.

What to Do With Errors in a DLQ

When an error lands in a DLQ, you have two options: take manual action and automatically retry. Consider starting with manual action exclusively while you get used to troubleshooting event-driven errors. Figure out what the patterns are in resolving them before you try to automate.

Once you identify errors that can be retried and how you would go about fixing them, then you should start building infrastructure around that process. The LEGO team has an incredible video on how they automatically retry event delivery failures in their system.

Conclusion

User notifications are important in any system. Keeping the end users engaged is a critical part of retention for a product. If the app silently fails (or succeeds) on any long running process, the user would be required to explore the app to discover what is happening.

With WebSocket API, adding these types of notifications are easy. Now it is up to you to implement it in your app. You can include them in your AWS Step Functions as part of a workflow, or make them a destination when an async lambda function completes.

Coming up in our WebSocket series is an example on how to incorporate what we’ve built into your existing application. We’ve built the WebSocket, now it’s time for us to use it.

I encourage you to deploy this stack into your AWS account and play around with it. At this point, we’ve built a production ready WebSocket microservice. So get it out there and start using it!

Happy coding!

Update

The WebSocket series is complete. If you would like to continue on, please refer to the following:

Part Five: Adding a Custom Domain