Overview

Prerequisites

• AWS Account credentials (Access Key ID and Secret Access Key)
• These can be provided for your own Amazon Web Services (AWS) account
• These can also be provided by Flomenco for managing queues in your Flomenco owned instance. Please reach out to Flomenco Support if you would like to deploy and manage queues in Flomenco owned AWS accounts.

• Appropriate IAM permissions to perform SQS operations

Available Operations

Queue Management Operations

Create Queue

• Queue Name (Required): Name of the queue to create

• Queue Type (Required): Choose between Standard or FIFO queue

• Queue Attributes (Optional):
• Delay Seconds: Default delay for messages (0-900 seconds)
• Message Retention Period: How long messages are kept (1 minute to 14 days)
• Visibility Timeout: Default visibility timeout for messages
• Maximum Message Size: Size limit for messages (1KB to 256KB)
• Receive Message Wait Time: Long polling wait time (0-20 seconds)

Delete Queue

• Queue URL (Required): The URL of the queue to delete

List Queues

• Queue Name Prefix (Optional): Filter queues that begin with the specified value

• Maximum Results (Optional): Maximum number of results to return

Add Permission

• Queue URL (Required): The URL of your queue

• Label (Required): Unique identifier for the permission

• AWS Account IDs (Required): List of AWS account IDs to grant permission to

• Actions (Required): List of actions to allow (e.g., SendMessage, ReceiveMessage)

Purge Queue

• Queue URL (Required): The URL of the queue to purge

Get Queue Attributes

• Queue URL (Required): The URL of your queue

• Attribute Names (Optional): List of specific attributes to retrieve

Add External SNS Permission Policy

• Queue URL (Required): The URL of your queue

• SNS Topic ARN (Required): The ARN of the SNS topic

• Policy Document (Required): The permission policy document

Message Operations

Send Single Message

• Queue URL (Required): The URL of your queue

• Message Body (Required): The content of your message

• Delay Seconds (Optional): Time to delay message visibility

• Message Attributes (Optional): Key-value pairs for message metadata

• Message Deduplication ID (Optional, FIFO only)

• Message Group ID (Optional, FIFO only)

Send Message Batch

• Queue URL (Required): The URL of your queue

• Messages (Required): Array of messages, each containing:
• ID: Unique identifier for the message
• Message Body: Content of the message
• Delay Seconds (Optional)
• Message Attributes (Optional)
• Message Deduplication ID (Optional, FIFO only)
• Message Group ID (Optional, FIFO only)

Receive Message

• Queue URL (Required): The URL of your queue

• Maximum Number of Messages (Optional): Number of messages to retrieve (1-10)

• Visibility Timeout (Optional): Time messages are hidden from subsequent retrieve requests

• Wait Time Seconds (Optional): Duration for long polling

Delete Message

• Queue URL (Required): The URL of your queue

• Receipt Handle (Required): Receipt handle of the message to delete

Delete Message Batch

• Queue URL (Required): The URL of your queue

• Entries (Required): Array of message receipt handles to delete

Usage Examples

Example 1: Create a New Queue

1. Select "Create Queue" as the operation

2. Enter a name for your queue

3. For standard queue: my-queue

4. For FIFO queue: my-queue.fifo

5. Select queue type (Standard or FIFO)

6. Configure optional queue attributes

7. Execute the node

Example 2: Send a Message

1. Select "Send Message" as the operation

2. Enter your Queue URL

3. Add message body

4. (Optional) Add any Message Attributes

5. Execute the node

Example 3: Receive and Process Messages

1. Select "Receive Messages" as the operation

2. Enter your Queue URL

3. Set Maximum Number of Messages (e.g., 5)

4. Set appropriate Wait Time Seconds for long polling

5. Execute the node to retrieve messages

Example 4: Delete Processed Message

1. Select "Delete Message" as the operation

2. Enter your Queue URL

3. Provide the Receipt Handle from the received message

4. Execute the node to remove the message from the queue

Example 5: Add SNS Topic Permission

1. Select "Add External SNS Permission Policy" as the operation

2. Enter your Queue URL

3. Provide the Queue ARN (this can be obtained via the Get Queue Attributes Node)

4. Provide the SNS Topic ARN

5. Execute the node

Best Practices

• Choose appropriate queue type based on your use case:
• Standard queues for high-throughput scenarios where order isn't critical
• FIFO queues when message order and exactly-once processing are required

• Use meaningful queue names that reflect their purpose

• Configure queue attributes based on your specific requirements

• Always delete messages after successful processing to prevent duplicate processing

• Use appropriate visibility timeout values based on your message processing time

• For FIFO queues, ensure Message Group ID is provided when sending messages

• Implement error handling in your workflows for failed operations

• Implement dead-letter queues for handling failed message processing

• Use appropriate IAM permissions and security policies

Troubleshooting

Common Issues:

1. Authentication Errors
• Verify AWS credentials are correct
• Check IAM permissions for SQS actions

2. Queue URL Errors
• Ensure Queue URL is correct and queue exists
• Verify region matches your AWS configuration (set in Provider Credential)

3. Message Visibility Issues
• Adjust visibility timeout if messages reappear too quickly
• Check if other consumers are processing the same messages

4. FIFO Queue Specific
• Ensure Message Group ID is provided
• Verify Message Deduplication ID if using content-based deduplication

5. Queue Creation Issues
• Ensure queue name follows AWS naming conventions
• For FIFO queues, verify name ends with .fifo
• Check if queue name is already in use

Notes

• Queue names must be unique within your Flomenco instance and region (set in Provider Credential)

• Queue names are limited to 80 characters

• Queue names can only include alphanumeric characters, hyphens, and underscores

• Once created, you cannot convert between Standard and FIFO queue types

• Message retention period is set in your AWS SQS queue configuration

• Maximum message size is 256KB

• Long polling can help reduce costs and latency

• FIFO queues have additional requirements but guarantee exactly-once processing and ordered delivery