Lob's website experience is not optimized for Internet Explorer.
Please choose another browser.

Engineering
January 11, 2023

Improving Developer Experience For sqs-consumer

by 
Sishaar Rao

Lob offers direct mail automation via a suite of APIs designed for developers. Even though we now offer a low-code solution to automate direct mail (via our Campaigns product), we will always be a developer-first company. We are excited to announce an open-source project that gives back to our developer community.  

TLDR: We’ve released @lob/sqs-consumer, an open-source library that makes it easier to work with AWS SQS; this includes support for AWS SDK V3, updated dependencies, a new Jest test suite, and fixes to major issues (ex: enables support for FIFO queues).

Background

AWS SQS, or Simple Queue Service, is a piece of infrastructure offered by Amazon. Like the name suggests, it’s a queue, or a way to make our data stand in line and wait to get processed. When there are multiple actors feeding data in and just as many reading data out, you need a faucet of sorts to control that flow. It’s a very powerful tool for processing data in a controlled manner.

The Problem

Owned by the British Broadcasting Corporation, “sqs-consumer” is an existing open-source library in GitHub for working with SQS. This library provides an easy way to configure the multitude of SQS settings, all with minimal code. It’s like a remote control that allows you to define only the pieces that you really care about when working with SQS—popping messages from the queue, deleting them, setting time-out intervals, and changing that depending on errors, and so on.  It has been a pretty popular library—we’ve used it in Lob API and other services—but Lob and the open-source community have major concerns. 

The bbc/sqs-consumer library is no longer being maintained. Pull requests involving major issues and features are being ignored, and most importantly, it’s not been upgraded to the new version of AWS SDK (V3). 

To fully understand the differences between V2 and V3, see this article, or the AWS docs, but from our perspective, the benefits of AWS SDK V3 are:

  • Modular Architecture: Only import the packages you need, reducing build times and bundle size
  • First-Class Typescript support: Provides well-defined types and documentation, improving Developer experience
  • First-Class Async/Await support: Enables us to modernize NodeJS codebases
  • Individual Client Configuration
  • Modular Middleware Architecture
  • Rewritten Streaming API

In short, upgrading to the latest version of the AWS SDK is necessary to improve the stability and maintainability of our platform.

We have services at Lob that need to be upgraded to V3 but they use sqs-consumer, which is stuck on V2. It would appear the current existing solution has been abandoned and is not being actively maintained, thus won’t ever be upgraded to V3 of the SDK. Finally, if we want to use V3 of the SDK, we’d have to write our own custom boilerplate code for SQS (reinvent the wheel). Arg!

The Solution: @lob/sqs-consumer 

It took a few dedicated days of work during a hackathon, but on October 28, 2022, we released @lob/sqs-consumer! It's an open fork of the previous package, but it makes a few key changes:

  • Upgrade underlying AWS SDK to V3
  • Upgraded all dependencies
  • Converted test suite from an outdated version of Mocha to Jest
  • Fixed challenges with FIFO queues

sqs-consumer got a glow up!

From zero to hero
From zero to hero

Migrating over and swapping to the lob/sqs-consumer library is very easy. All you need to do is create a new V3 client; everything else, your code, your configuration, it all stays the same. You can download this package here

Wrap up

This new-and-improved library removed major roadblocks and has allowed us—and developers—to move forward with development in AWS SDK V3. This is a huge win for our engineers, and we look forward to feedback and contribution from the developer community.

UPDATE: In December, after a year and a half of radio silence, the BBC started making some updates to the original package! Not saying we had anything to do with that, but... 😉

Special thanks to Software Engineer, Sishaar Rao, with help from Staff Software Engineer, Landon Barnickle, and Senior Software Engineer, Hanqing Chen.