top of page

How to Use AWS SNS/SQS to Build a Distributed Application?

Updated: May 8, 2023


Overview


Microservices are a way of building and organizing large software applications as a collection of small, independent services that communicate with each other over a network. The main goal of microservices is to make it easier to develop, deploy, and maintain large, complex applications by breaking them down into smaller, more manageable pieces.

There are several ways of communication between microservices:

  • RPC (Remote Procedure Call);

  • API (Application Programming Interface);

  • Message brokers.

RPC involves using a client-server model, where a client sends a request to a server and the server returns a response. This can be a simple and efficient option for communication between a small number of components, but it can become inflexible and unscalable as the number of clients and servers increases.

API involves exposing a set of functions or methods that can be called by other applications or services over a network connection. This can be a flexible and powerful option, but it can also require more effort to set up and maintain, as the API needs to be designed and documented carefully.


Message brokers are a good option when you need to decouple the sender and receiver of a message, support a large number of senders and receivers, or ensure the reliability and integrity of the messages being sent. They may not be the best option if you only need to communicate between a small number of components, or if you need a high level of control over the communication channel.


Why SQS/SNS?


When it comes to choosing a message broker for your application or system, there are many options to consider. Some of the most popular message brokers include Kafka, RabbitMQ, NATS, and AWS SQS/SNS. Each of these message brokers has its own strengths and weaknesses, and the right choice for your system will depend on your specific requirements and constraints. We will compare these message brokers in terms of their features, performance, scalability, and cost, to help you make an informed decision about which one is best for your needs.


Kafka is designed for high performance and horizontal scaling. It supports a wide range of use cases, including real-time data pipelines, streaming analytics, and event-driven architectures. Pros: Kafka is highly scalable and performant, and offers a wide range of features for configuring the delivery and behavior of messages. Cons: Kafka can be complex to set up and maintain, and may require more specialized skills and knowledge.


RabbitMQ is designed for flexibility and ease of use. It supports a wide range of messaging patterns and protocols, and can be used on-premises or in the cloud. Pros: RabbitMQ is easy to use and offers a wide range of features and customization options. Cons: RabbitMQ may require more setup and maintenance compared to some other message brokers, and may not be as scalable or performant as some other options.


NATS is designed for simplicity and high performance. It supports a wide range of use cases, including real-time messaging, event-driven architectures, and microservices. Pros: NATS is simple to use and offers high performance and scalability. Cons: NATS may not have as many features or customization options as some other message brokers.


AWS SQS is a fully managed, distributed message queue service that is designed for scalability and reliability. It supports a wide range of use cases, including message-based architectures, event-driven architectures, and microservices. SNS is a fully managed, distributed notification service that allows you to send messages to a wide range of endpoints (e.g. email addresses, phone numbers, AWS Lambda functions). Pros: AWS SQS and SNS are fully managed, meaning that AWS takes care of all of the underlying infrastructure and maintenance for you. They are also highly scalable and reliable, and offer a range of options for configuring the delivery and behavior of messages. Cons: AWS SQS and SNS are paid services, with costs that can vary based on the volume and type of usage. They may not offer as much control or flexibility as some other message brokers.


Code examples


Before you start it is worth noting that you need to configure the following items:

  • topic topic_card_created

  • queue queue_card_created

  • and queue subscriber queue_card_created to a topic topic_card_created

You can find more details on how to do it in the AWS tutorial.


Let’s imagine that something happened in one of the microservices (Service 1 in the picture), for example, a bank card was created, and now we need to notify another microservice (Service 2 in the picture) about this.

Let’s describe the structure of the event, which will contain some information about the bank card:

Then we will write a publisher - an entity that will publish a message to a certain topic. It is worth noting that we use the library written by Velmie to work with SNS / SQS - https://github.com/velmie/broker.




The OnCardCreated method will receive a structure with information about the card, wrap it in a message structure from the broker, and send it to the topic, which we will also pass to this method.

Now let’s write the last part from the diagram at the beginning of the chapter - the event handler, also known as the subscriber on the queue, which will be located in another microservice:


As you can see from the code above, you can add event handlers to the subscriber, subscribe to them, and also unsubscribe.


Now let's register it all in the main and check how it works.

Let's declare the necessary constants:

Next, let's create an aws config and initialize an aws session:

Aws account id is required for the publisher, we get it like this:


Register publisher:

Register subscriber:

Add an event handler for the new card to the subscriber. It will display the information that came in:

And send a message to the broker:

If the components work correctly, the following output will be displayed in the console:

INFO[0000] got new message. Body {"cardId":"1ffce375-ded4-42b8-af94-3a40ec5c2024","userId":"60437ef4-cd86-11ed-afa1-0242ac120002"}


The whole working example from the article can be found here:

How it can be used?


When we need the reliability and integrity of message processing, especially in cases of integration with third-party services, when message processing may take a long time, or when processing may fail and it needs to be repeated, then we use the “message locking” technique. Here is an example:

In this diagram, we see 3 services. This diagram shows how Service 1 notifies the integration service about some event through a message broker. If the message has been successfully processed after some time, then the integration service will send the message to Service 2. If not, then the message will be processed again after some time.


If we need distributed systems that need to scale horizontally and process a large volume of work efficiently, for example for notifications functionality, we can use “consumer pattern”.

In this diagram, you can see how several notification services are subscribed to the same queue. This gives us horizontal scaling and efficient work.


Also, if there is a need to process the same message to several services, you can make 1 topic, and several queues that will receive the same message and notify the services that are linked to them:



Conclusion


In conclusion, Amazon Simple Queue Service (SQS) and Simple Notification Service (SNS) are powerful and flexible message queueing services that can be used to decouple and scale microservices, distributed systems, and serverless applications. SQS offers a range of features and options that make it a great fit for a variety of use cases, including messaging, job processing, and event processing. Our experience with using SQS has been positive, and we have found it to be reliable and easy to use.


In addition, SNS allows you to send notifications to a variety of endpoints, including email, SMS, mobile push, and HTTP/S endpoints. It offers a highly scalable, flexible, and cost-effective way to distribute messages to a large number of recipients. We have found SNS to be a powerful tool for sending notifications and alerts to our users, as well as for integrating with other AWS services.

Whether you are looking to build a distributed application, process background jobs, send messages between services, or send notifications to users and other endpoints, SQS and SNS are worth considering. Their scalability, flexibility, and reliability make them valuable tools for building and running distributed systems and applications.


Written by Vlad Kulizhenko, Software Engineer at Velmie




Comentarios


bottom of page