Every developer wants to write better and efficient NodeJS applications. Applications that work reliably and do what it is made for.

Queues are something that is ignored by a lot of developers, at least from my experience.

This story is focused on using queues to make the NodeJS applications more reliable, error-free, and efficient. It's not rocket science and very easy to understand and use.

What is a Queue?

Fundamentally, a queue is just a list or an array of items. We push items in it from one side and pop one from another side when needed.

The items are ordered and processed on a first-come, first-serve basis (FIFO - First In, First Out) (not really necessary). We can also process items on a last come, first serve basis (LIFO) but that makes it a Stack.

I will only be talking about Queues here but Stacks can, and should, also be used depending on the applications need.

There are different kinds of queues. I will talk about some of them indirectly here and will write a separate blog post on types later.

Now we know what is a queue and most of you may already knew it, let's dive into another section.

Why and When to Use a Queue?

Using queues correctly in your application may provide a lot of benefits. And if not used correctly, it may have the opposite effect.

A queue can bring an order to your application. Depending on what library you are using or how you implement the queue, the benefits may vary.

The queue should not be used if the response is expected right away. It may cause timeouts. You can set the timeout to an hour or whatever, but I don't think a user will an hour to get an API response that long.

Generally, things that can be delayed or can be done later or on the side, are probably the best use cases to utilize a queue. For example, sending emails, processing files, generating PDF, batch processing, transcoding videos, etc, are some examples of the type of workload that can benefit from a queue. Similar to fire and forget.

Let's go through some example cases/benefits of using a queue. A queue can be as simple as an array to a separate service using persistent storage, such as RabbitMQ, Bull, etc.

Decoupling

The first thing a queue provides is decoupling, i.e. the producer (code that pushes the tasks to the queue) and the consumer (code that pops the tasks from the queue to process it). This makes it very easier to move the producer and consumer to different machines or maybe run as a different service.

Example: You were sending emails directly from your code, i.e. calling the email API or server directly in your code. Something similar to the below code

const signup = async (user) => {
   await validateSignup(user);
   await createUser(user);
   await sendSignupEmail(user.email);
}

While this may work most of the time, there is a chance that email API gets down, or just the request fails. It will most probably fail the entire request (depends if you have used try/catch correctly). Or maybe the email service is busy and taking too long. Meanwhile, the user is waiting. You should not wait here until the email is sent (or you should, you know better).

A better option will be to push the email request to a queue, where the email service will pick the event and send the email

const signup = async (user) => {
   await validateSignup(user);
   await createUser(user);
   await EmailQueue.push("USER.SIGNUP", { email: user.email });
}

This way you can easily move the code responsible for sending emails to different services when needed.

Scalibility / Workload Distribution

As we can easily move the producer and consumer to different services, it makes it easier to scale them individually. In most cases, it's the consumer that needs to be scaled.

Example: When a user uploads their profile picture, you want to generate multiple thumbnails of different sizes or maybe compress the images, or both?. Because we know it's not a good idea to show 5 MB DSLR images as the avatar on the top-right account dropdown.

You can process the image on your main app, but that's really a bad idea (maybe not for 10-20 users or more). A good solution will be to move the processing code to another service.

1. The user uploads an image
2. You push the URL to the image processing queue
3. Image processing code picks a task from the queue
4. Process and uploads the images
5. Updates the DB or fire an event to update the DB

This way if lots of users are uploading images and your image processing service is under load, it can be easily scaled individually.

But how many time a user uploads their profile picture? And do we really need another service for profile pictures? Figure it out.

Reliability / Retry

Another benefit of using a queue is that you know which tasks were completed successfully and which ones failed. Without a queue, it may be difficult to find out. Obviously, I assume the queue you are using supports this feature.

You can also retry to process the failed requests very easily this way. It will require some extra coding and some queue library has this feature built-in. It helps reduce the amount of setTimeout calls you make to retry in your main code.

Priority

Well, everything has a priority, right? Right? Of course, some things are important than others. Like enjoying your weekends with family/friends or riding a bike instead of staring at the computer for 20 hours. Well, obviously you can because it's your life. But maybe not in applications.

An email containing the OTP or password reset link is far more important than sending a welcome email template from the CEO.

A queue (assuming supports priority) can be used to prioritize some tasks over others, which is a great way to prioritize important things.

Rate limit

Our email service was working fine but suddenly we are getting 429. Yikes! Faheem did not use a queue to send emails and thought sending emails directly will work fine. It did because the API provider has its own queue.

But you are not always lucky. Some services have a rate limit that you have to follow to keep things running smoothly. For example, AWS SES (email service) has a default quota of 200 emails per 24-hour period. While SendGrid has 600 requests/minute.

Depending on which service you use and for what purpose, you need to follow the rate limit. Using a queue that supports this feature can make this very easy.

And this is another example where you need to prioritize your tasks so non-important things don't eat your quota.

Why and When NOT to use a Queue?

Things that need to be done right away should not use a queue, such as login, logout, general API calls, or anything that can't be fire and forget.

It's also a good idea to keep things simple and try to keep your code modular and decoupled. So it can be easily moved to another service later.

You should not focus too much on these things from start. Well, you should but you know better as you know about the project/product more than me. When you get more users, more load, and you think it's time to scale/upgrade things, take a decision.

It's probably not a good idea to use RabbitMQ to send emails using SendGrid or another service. A simple queue will work and these kinds of services provide several benefits.

Some Libraries

You can always make your own queue. But I will personally not make my own.

• queue
• fastq
• p-queue
• bull
• better-queue
• bee-queue

Some Services/Apps

• RabbitMQ
• Kafka
• AWS SQS
• Google Cloud Tasks
• Azure

Thank you!

I have recently started (again) writing blogs and would love to get some serious feedback.

If you are reading until now, did you notice how priority queue does not follow FIFO. Its weird right? But its still a queue because FIFO is not a hard rule, and we can call them FIFO queue. Some times order is not really important.

I will soon make a video on this topic. If you want to read more of my blog or want to watch the video, you may follow me on Twitter .

Credits

Cover Photo by Cátia Matos from Pexels