Create SQS FIFO, and the IAM role to access it

In this first lesson of Scalable EC2 consuming servers for SQS series, we will create an SQS FIFO, and an IAM role to give EC2 applications access to the FIFO.

SQS offers currently two types of queues. The standard one, that guarantees almost unlimited throughput, and at least once delivery, but may not preserve order. At least once delivery means that the same message can be added twice to the queue. these two limitations make it harder to deal with standard queues, although it has a very powerful throughput. AWS suggests to handle this duplication or miss-ordering in the application.

The other type of queues that SQS offers is the FIFO queue, First In, First Out. This one provides high throughput, 300 request/second, which can be argued about depending on the application, but for our tutorial it will be more than enough. FIFO also guarantees that only one request got registered in the queue, also requests got listed in order, so you will not serve the third request before serving for the first two, or at least requesting the first two. I will use FIFO in this tutorial, because I want to keep it as simple as possible, and our problem really do not need such high throughput.

Create FIFO

After logging in you AWS console, head to the SQS service, and create a new queue. Enter the queue name, in my case I will pick ‘ScalingTestQueue’. and make sure that you clicked on the FIFO button. Since we are creating a FIFO, aws forces us to add the postfix .fifo to the name, so please make sure to add that as well.

Do not just click on quick creating, because I would like you to just look at the configurations. Do not worry, we do not need to change any, but it  worth the look.

The configurations to consider are:

  1. Default Visibility Timeout: by default, this parameter is 30 seconds, which means that after an application gets a message from this queue, this message will be invisible for other applications to receive for this 30 seconds period. If the application completes serving this message, it should delete this message. If the deletion do not take place, this is translated to “The application could not finish his job”, and the message is put back to be available for other consuming servers, or applications. You should be cautious when picking this value, as it should be larger than the processing time, but not too large to make sure this message got processed ASAP.
  2. Content-Based Deduplication: if set to true, AWS will remove the new message if there is currently a message has the same content. If this is set to false, or unchecked as default, then AWS will ask the message to have a unique key to deduplicate values based on.

Other configurations might be interesting based on your case. In ours, we do not need to change anything, just keep going on, and create the queue.

Create IAM Role

IAM is used to provide access to AWS services. Since we are using applications, we can use users to give access to the application, using key Id and secret key that can be obtained from IAM service. These keys should be placed in a configuration file placed in /home/user/.aws/credentials or in Windows c:\Users\{User Folder}\.aws\credentials . It is also possible to pass the credentials to the client object in the code. Using this method is not recommended, as you will have to change the image that you are using for creating the EC2 instances, and also you will have to change everywhere you used that key if that key got compromised.

AWS gives you Roles, a quick way to authorize applications built using AWS SDK in EC2 instances. It do not depend on keys, and thus will make no problem of compromising your secret keys. Also it needs no special programming or configuring in your code.

To create a role, please head to IAM service, click on roles, and create a new one.

  1. Select the trusted entity to be an AWS EC2 instance, then head to permissions, next.
  2. Enter in the search box SQS, and check on AmazonSQSFullAccess, then next.
  3. Set the name, in my case I will pick “ScalingTestRole”.

With this, we have created the SQS FIFO queue, and the IAM role that will be used to access the created queue.

Next, we will create Cloud watch Alerts.

Leave a Reply

Your email address will not be published. Required fields are marked *