Bright Wolf Blog, Partners

5 Best Practices for Running Scalable IoT Architecture Solutions on AWS

By following this list of best practices, your IoT solution can leverage the scalability of Amazon Web Service (AWS) infrastructure and let your team focus on the most important part of your business: your customers.

Whether you’re building a connected solution for asset management, predictive maintenance, yield optimization, or other business goal, the ultimate value returned to your organization will be determined by how well the system meets your domain-specific requirements and individual needs of your customers.

By properly leveraging AWS infrastructure and services as outlined here, the IoT solution you build today will continue to “just work”, no matter how big or how fast your business grows.

1. Be ready for the data tsunami on Day 1

It’s going to hit eventually, and whether the flood of incoming data comes from a sudden surge in business, consistent monthly growth, or even a malicious attack, your system should be prepared to handle it right from the start. Fortunately, there are ways to architect your system that allow AWS components to ensure all of your data is always reliably processed. By understanding and adopting proper AWS IoT architecture upfront, you free your team to focus on building application intelligence (where the value is created from the data) and can move ahead confidently with your market strategy no matter how ambitious it may be.

iot data tsunami

2. Pay attention to the critical link

The ability to quickly and reliably ingest data from your IoT devices is crucial to the success of your IoT initiative. AWS IoT Rules allow you to trigger many different actions upon receipt of a message. While all AWS services adapt to different data flow properties, not all services are designed to be used as the single point of entry into the system. Some have behaviors that could prevent all of your data from being processed as desired.

For example, in high volume applications you should consider buffering or queueing incoming data before invoking downstream services like AWS Lambda, so that in the event of subsequent failures the application still has the ability to recover.

With unpredictable spikes in data load, in the time it takes for a Lambda function to spin up there could already be hundreds of thousands of messages in the pipeline. Going to Lambda directly in this example could result in message loss. But if our data is buffered in Amazon Simple Queue Service (SQS) first, downstream systems have more processing time for any large spikes of incoming data.

aws iot rules

3. Get a life(cycle)

IoT devices are notoriously difficult to manage over the lifespan of initiatives that often take several years just to make it into production. Once deployed, devices need to be trackable and maintainable while facing challenges from varying business needs, staff turnover, and hardware/software degradation. Unsurprisingly, Amazon recently added features to AWS IoT core that address these concerns head-on.

Configurable endpoints in conjunction with multi-account registration provide some powerful abstractions that enable you to communicate with IoT devices using environment-specific accounts and device-specific endpoints. These abstractions free developers and operators from the daunting challenge of organizing disparate AWS IoT resources from the same AWS account with the same IoT endpoint.

Real world IoT deployments often face networking challenges that are a result of the device users’ legitimate concerns around enterprise security. While data collection is an important piece of the puzzle, it is often not the most difficult piece of the security puzzle. IoT devices need regular maintenance to ensure that they remain reliable and secure over the long run. Providing access to devices that tend to be hundreds if not thousands of miles from the home base of operations teams is challenging. The addition of secure tunneling to AWS IoT Core gives teams a powerful capability for accessing devices that are protected by private networks.

4. Sometimes the best way to scale is not to scale at all

While we’ve shown you how you can architect your solution to leverage the scaling power of AWS, sometimes you don’t want to process all of your machine data in the cloud.

aws iot amazon greengrass

One way to reduce traffic to the cloud is to run AWS Greengrass at the edge. Greengrass can intelligently process and filter data locally, eliminating the need to send all device data upstream. Greengrass also allows you to leverage the power of machine learning through the ML Inference engine. Models are developed in AWS SageMaker (or independently) and are run on the device. This approach allows you to leverage the volume and complexity of cloud-side data with the flexibility of device-side execution. 

While AWS infrastructure allows your system to automatically scale when it needs to, it also provides the ability to process data from your devices whenever and however you want to.

5. Choose the ingest that is best

Just because you are building an IoT system doesn’t mean all your different types of data have to go through AWS IoT Core. AWS IoT supports both MQTT and WebSockets, and there are appropriate settings for each service based on recommended use cases. If you ever find yourself bumping up against AWS IoT service limits, there’s a good chance you’re just not using the right service for the particular task.

MQTT is perfect for small payloads of sensor data typical in IoT systems. In a legacy system architecture with log files or XML file dumps, however, it may make more sense to parse the files and send selective values up via MQTT in small chunks. Let’s say you want more than a few values out of the file, it could in fact be more appropriate to send file data via Kinesis or directly to S3 and process it later. MQTT was intended for enabling communications on extremely small embedded devices, why use your device’s limited power for parsing when it can be offloaded to the cloud?

The story is the same for historical and 3rd party data. Many legacy devices do not have the ability to run newer SDKs that enable MQTT and WebSocket protocols. However, many of these systems can make HTTP requests or use plain TCP sockets for communication. By leveraging services such as Amazon API Gateway and Lambda, you can build custom REST APIs that allow you to ingest data into an SQS queue, an S3 bucket, or an RDS database.  By taking the step upfront to choose the right AWS tool for the right job, you can maximize the scalability your overall solution provides and realize more value from IoT on AWS.

Putting it All Together

Bright Wolf has used these best practices to deliver IoT solutions on AWS across Manufacturing, Oil & Gas, Cold Chain Transportation, Healthcare, Agriculture, Smart Building, Heavy Equipment, and other industries. If you’re looking to get started on a rapid prototype or are facing challenges with the architecture of an existing system, we can help. Let’s talk about your project, and we’ll share more guides and case studies to accelerate your digital strategy and implementation initiatives.

bright wolf aws iot architecture

About Bright Wolf

Bright Wolf helps industrial enterprises increase business value by transforming operations and organizations with digital strategy, technology, solution delivery, and team enablement.

Industrial IoT Newsletter
Protected by reCAPTCHA, Google Privacy Policy and Terms of Service apply.
Featured in…

IoT OneCIO ReviewIoT Agenda Manufacturing.net IoT Evolution IoT Inc IoT Central IoT for All Industry Today

Learn how Bright Wolf can help your team

Bright Wolf Services

Digital strategy, architecture, development, integration, and operations

IoT Platform Accelerators

Connect equipment and generate value in the cloud faster with AWS and Azure solution starters

Client Success Stories

Learn how Bright Wolf clients are optimizing operations & creating business value for customers