Amazon SQS Industrial Use Cases
Use Cases of AWS SQS and how Coursera, EBS, CapitalOne, NASA, BMW, RedBus use SQS as a service to maintain an excellent workflow.
Amazon SQS
Amazon Simple Queue Service…
Amazon Web services hold many services in its package. These services are widely used around the globe to lessen the amount of burden caused due to managing them. One such managed service is SQS, lets get a know-how of the benefits it has to render us.
What is SQS service?
Use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be available. SQS lets you decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system.
Amazon SQS is a tool in the Message Queue category of a tech stack.
Amazon Simple Queue Service supports tasks that process asynchronously. Instead of one application having to invoke another application directly, the service enables an application to submit a message to a queue, which another application can then pick up at a later time.
Insimple words, SQS can be defined as a service where one service sends messages in a queue and another service receives those messages. It can be used for a whole range of purposes.
SQS
Why SQS?
SQS is used by a developer to reduce the complexity of managing and operating a service. Using this we can store, send, receive and manage messages between two services Instead of sending messages directly we can send through the intermediate queue.
Queues can be used to handle malfunctions in the scalability of services. when a service depends on another service that can not scale to the same level. Having a queue between A and B is useful in such situations because the queue can handle peak demand from A, and send messages at a slower rate with which B can cope. Where A will send a message directly to B this may cause an issue if B System gets failure, corrupt, or lost data during loading data to solve this we are gone using SQS Service where this acts as Intermediator which stores data and sends to B so even B gets failure until that system set up this will be stored in SQS than sent to B.
There are two Queues in SQS Standard and FIFO SQS
Standard vs FIFO SQS
Standard and FIFO SQS
Message Queue Life Cycle
Amazon SQS automatically deletes messages that have been in a queue for more than the maximum message retention period. The default message retention period is 4 days. However, you can set the message retention period to a value from 60 seconds to 1,209,600 seconds (14 days) using the SetQueueAttributes action.
Message Queue Life Cycle
Companies Using SQS
Lots of companies reportedly use Amazon SQS in their tech stacks, including Pinterest, Amazon, BMW, NASA, EMS Driving Fuel IQ, Capital One, Redbus, Lyft, and many more.
I mentioned some companies case study below,
Recently Started Integrating SQS in their companies
KreditBee
By implementing a serverless strategy on AWS, KreditBee has increased its monthly loan disbursement by 25 times over a two-year period.
Expedia Group
Expedia Global Payments migrated a legacy system from Microsoft SQL Server to Aurora PostgreSQL, cost-effectively scaling to meet traffic and delivering near-real-time data to its users and internal teams.
THORN
On AWS, Thorn built Safer to detect, report, and remove child sexual abuse material from online content-hosting sites and validated its efficacy, flexibility, and scalability with beta customer Flickr, SmugMug’s photo-hosting site.
RepricerExpress
RepricerExpress reduced costs by $60,000 a year and obtained larger, higher-performing compute resources by migrating a Windows Server and SQL Server environment to AWS from Microsoft Azure.
SCB ABACUS
By leveraging AWS AI and Big Data technologies for its financial services platform, SCB Abacus has been able to make the loan process for customers seamless, update service features quickly, and comply with international standards.
Click here to see more companies success stories by using SQS
Top Companies Using SQS and their success stories
NASA Using SQS
NASA Established in 1958, the National Aeronautics and Space Administration (NASA) has been working around the world — and off of it — for almost 60 years, trying to answer some basic questions: What’s out there in space? How do we get there? What will we find? What can we learn there, or learn just by trying to get there, that will make life better here on Earth?
NASA
Use Cases
The NASA Image and Video Library provides easy access to more than 140,000 still images, audio recordings, and videos — documenting NASA’s more than half a century of achievements in exploring the vast unknown. The architecture includes Amazon SQS to decouple incoming jobs from pipeline processes and Amazon Simple Notification Service to trigger the processing pipeline when new content is updated.
NASA Use cases
Architecture
The NASA Image and Video Library is a cloud-native solution, with the front-end web app separated from the backend API. It runs as immutable infrastructure in a fully automated environment, with all infrastructure defined in code to support continuous integration and continuous deployment (CI/CD).
In building the solution, ManTech International took advantage of the following AWS services:
• Amazon Elastic Compute Cloud (Amazon EC2), which provides secure, resizable compute capacity in the cloud. This enables NASA to scale up under load and scale down during periods of inactivity to save money, and pay for only what it uses.
- Elastic Load Balancing (ELB), which is used to distribute incoming traffic across multiple Amazon EC2 instances, as required to achieve redundancy and fault-tolerance.
- • Amazon Simple Storage Service (Amazon S3), which supports object storage for incoming (uploaded) media, metadata, and published assets.
- • Amazon Simple Queue Service (Amazon SQS), which is used to decouple incoming jobs from pipeline processes.
- • Amazon Relational Database Service (Amazon RDS), which is used for automatic synchronization and failover.
- • Amazon DynamoDB, a fast and flexible NoSQL database service, which is used to track incoming jobs, published assets, and users.
- • Amazon Elastic Transcoder, which is used to transcode audio and video to various resolutions.
- • Amazon CloudSearch, which is used to support searching by free text or fields.
- • Amazon Simple Notification Service (Amazon SNS), which is used to trigger the processing pipeline when new content is uploaded.
- • AWS CloudFormation, which enables automated creation, updating, and destruction of AWS resources. ManTech International also used the Troposphere library, which enables the creation of objects via AWS CloudFormation using Python instead of hand-coded JSON — each object representing one AWS resource such as an instance, an Elastic IP (EIP) address, or a security group.
- • Amazon CloudWatch, which provides a monitoring service for AWS cloud resources and the applications running on AWS.
NASA AWS Architecture
Environmental Monitoring Solutions (EMS) Using SQS
Environmental Monitoring Solutions (EMS) is based in Victoria, Australia. Launched 25 years ago, the company specializes in solutions that help petrol retailers gather and analyze data on the performance of their petrol stations. Its solutions provide remote monitoring and 24/7 support services — helping customers boost sales, reduce maintenance expense, and decrease the risk of accidents. Today, EMS operates with a team of 30 personnel.
EMS solving challenges using SQS
The Challenge
The stations need to operate highly efficiently because profit margins are small. Yet, at the same time, they have to offer great customer experiences, ensure employee safety, and minimize their environmental impact. Sometimes accidents do occur, and a typical EMS customer is likely to incur annual costs of AU$15 million (US$12.13 million) for cleaning up underground petrol tank leaks or vehicle fuel tank contamination. To help customers maximize efficiencies while addressing the need for service excellence, safety, and environmental protection, EMS developed Fuelsuite, which enables customers to switch from legacy technologies that are largely manual and unify station management — significantly reducing costs. With Fuelsuite, customers can monitor inventories, deliveries, and prices. The solution also raises alarms in event of possible environmental incidents, such as underground tank overfills. After the successful launch of Fuelsuite, EMS focused on product development. It looked to connect sensors in the stations’ underground tanks and pumps and, regardless of the configurations of those sensors, collect all their data at 30-second intervals. The data would then be aggregated on a cloud-computing infrastructure and displayed via a web-enabled interface in Fuelsuite in near-real-time. Russell Dupuy, founder and managing director of EMS, says, “Our job was to find an IT partner and cloud-computing provider that could help us re-engineer our Fuelsuite technology and deliver an innovative off-the-shelf product that was user-friendly and easily customizable.”
The Benefits
Atpresent, more than 1,000 petrol stations in Australia are benefiting from the IoT-enabled Fuelsuite solution. Station operators get near-real-time data on the performance of their stations, including how much petrol is being sold and how much is in the underground storage tanks. It also includes data on the pressure inside the hoses connecting the petrol pumps to the automobiles and on the temperature and petrol level inside the underground tanks. “With our AWS IoT–enabled Fuelsuite solution, customers manage their petrol stations proactively rather than reactively — gaining a complete picture of petrol station performance to dramatically improve efficiencies and detect fuel leaks early to minimize environmental impacts,” says Dupuy. This how the SQS Service is helping EMS
For more details check this EBS using SQS
How RedBus used SQS as a service from AWS for expanding their business :
The redBus is an Indian travel agency that specializes in bus travel throughout India by selling bus tickets throughout the country. Tickets are purchased through the company’s Website or through the Web services of its agents and partners. The company also offers software, on a Software as a Service (SaaS) basis, which gives bus operators the option of handling their own ticketing and managing their own inventories. To date, the company says they have sold over 30 million bus tickets and has more than 1750 bus operators using the software to manage their operations.
RedBus Uses SQS services
The Challenges
The biggest problem was that the infrastructure could not effectively handle processing fluctuations, which had a negative impact on productivity. Additionally, the procurement of servers or upgrading the server configuration was an extremely time-consuming endeavor. Over time, redBus realized that a better solution was imperative — a solution that offered scalability to handle the company’s processing fluctuations. redBus looked to Amazon Web Services (AWS) for a solution.
The Benefits
AWS the company with its many magnificent characteristics, perhaps the most significant to redBus is the ability to “instantly replicate the whole setup on-demand for testing by creating and destroying instances on demand for experimentation, thereby reducing the time to market.” Less time to market translates to increased profitability and success.
The travel agency anticipates expanding the AWS solution to include Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) for monitoring, alerts, and intercommunication. SQS helps to provide more security by securing the sensitive data of users of redBus. In addition, the increasing amount of users of redBus are benefitted from the reduction in latency provided by SQS.
How BMW used SQS as a service from AWS for expanding their business :
The BMW Group is using AWS for its new connected-car application that collects sensor data from BMW 7 Series cars to give drivers dynamically updated map information. BMW Group is one of the leading manufacturers of premium cars and mobility services in the world, with brands such as Rolls Royce, BMW, and Mini. BMW built its new car-as-a-sensor (CARASSO) service in only six months leveraging Amazon Simple Storage Service (Amazon S3), Amazon Simple Queue Service (Amazon SQS), Amazon DynamoDB, Amazon Relational Database Service (Amazon RDS), and AWS Elastic Beanstalk. By running on AWS, CARASSO can adapt to rapidly changing load requirements that can scale up and down by two orders of magnitude within 24 hours. By 2018 CARASSO is expected to process data collected by a fleet of 100,000 vehicles traveling more than eight billion kilometers. -BMW Resource From Mayank Agarwal
BMW Uses AWS
watch this for details on how BMW solves a challeng using SQS.
Coursera Uses SQS
Watch this video to find out how Coursera uses Amazon (AWS) to manage and integrate fast communication power over 300 free online courses from Stanford, Duke, Princeton, the London School of Economics, and other universities to more than 3 million students worldwide.
Coursera uses AWS
Watch this for more details on how Coursera overcome challenges by using SQS
CapitalOne Uses SQS
Capital One
Capital One is a leading information-based technology company that is on a mission to help its customers succeed by bringing ingenuity, simplicity, and humanity to banking.
The Challenge
CapitalOne Challenge
Before SQS Integration
Existing Queue Integration
Selection behind SQS Service
SQS selection criteria
Application Queue Before SQS and After SQS
CapitalOne Application Queue
The architecture of SQS Integration with Capital One application
Capital one and SQS Integration
Benefits Of AWS SQS
Benefits of SQS
Companies Use SQS
680 companies reportedly use Amazon SQS in their tech stacks, including
Some of the companies using SQS
Click here to see more companies success stories by using SQS
Amazon SQS Integrations
Apache NiFi, MongoDB Stitch, SignalFx, LocalStack, and Cloudcraft are some of the popular tools that integrate with Amazon SQS. Here’s a list of all 16 tools that integrate with Amazon SQS.
Integrated tools
Benefits and features of SQS Service
Security: You control who can send messages to and receive messages from an Amazon SQS queue. Protection in transit and at rest. Transmit sensitive data in encrypted queues. Send messages in a Virtual Private Cloud.
Server-side encryption (SSE): It lets you transmit sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).
Availability: Amazon SQS uses redundant infrastructure to provide highly-concurrent access to messages and high availability for producing and consuming messages.
Scalability: Amazon SQS can process each buffered request independently, scaling transparently to handle any load increases or spikes without any provisioning instructions.
Reliability: Amazon SQS locks your messages during processing so that multiple producers can send and multiple consumers can receive messages at the same time.
