Recently, I worked on a project for a mobile backend solution, which required me to integrate with multiple 3rd party providers, and also connect with IoT devices in order to process data on the field. I would like to share my learnings from this project about how to design and architect large applications with IoT.

For the confidentiality of my client, I cannot show-case the entire application in detail, instead, I will explain this solution using a cut-down version as per the design diagram below.

The scope of the backend system includes

  • Authenticated APIs for the mobile application
  • Input from the camera to be processed
  • Display data on the monitor from cloud
  • Handle payment via credit card
  • Data persistence and storage

The first thing about solution design is to determine the technology to use. For this project, I chose to implement it on top of AWS + Serverless framework not only because of powerful services available (API Gateway, Lambda, etc) on AWS, but also the great eco-system to support IoT development. AWS IoT Greengrass supports the ability to execute Lambda functions on the device, and Greengrass core provides secure connections between edge devices and the cloud to seamlessly manage devices in the field. The fact that Greengrass supports Node.js is another good reason for this choice as most developers are more familiar with JS over Python.

The next thing to do is to split the backend solution into multiple microservices. There are a couple of things that have been taken into consideration when designing services for this application.

  • Single business feature per microservice. In my experience, some developers may over partition services resulting in too many tiny services which increases the complexity of integration and maintenance. This presentation from AWS re:Invent 2019 demos some architecture patterns and best practices, I recommend you to take a look at.
  • Size properly. Size each microservice so it can be managed and developed by single dev team.
  • Avoid coupling between service: Services should have high cohesion along with loose coupling. Over-coupling on code base or infra resources will increase difficulties for testing and monitor services.

Based on above principles, this application is designed into multiple microservices as below:

  • Authenticated REST APIAPI Gateway and Lambda functions were used ato create backend APIs. In addition, it also uses Cognito as an authentication provider to manage users, which can be directly integrated with mobile apps using Cognito SDK (available for both iOS and Android) to register and authenticate a user.
  • Payment ServiceStripe was selected to handle payment. In the mobile app, it implements Stripe UI directly to register a user’s credit card, you can customize UI to do so as well. In the backend, a payment service is designed using a Lambda function to integrate with Stripe SDK to authorise and process payment.

One of the key things I learned from designing an IoT solution is to decouple the services from Edge to Cloud but scope them as one single application. When designing an IoT solution, list down all the requirements on one piece of paper, think through all data flows from one place to another, separate the actions on the device with actions on the cloud in order to decouple the services with the least dependencies. For this project, IoT related requirements are designed into multiple services to architect a two way solution to communicate between the edge device and the cloud:

  • IoT Edge App: An application runs on the IoT edge to connect edge devices with cloud services. Multiple Lambda functions are deployed into a single IoT Edge to connect with a camera and a monitor in the field. To start with, the camera will detect an image of an order and analyse the image to predict the content of an order. With IoT rules and actions defined in Greengrass IoT Core, the edge device is able to send the camera prediction result to the order processor in AWS via an MQTT message. When defining a CloudFormation template for an IoT core, one of best practises is to set core ID as input parameter and name resources dynamically using this ID, e.g IoT-${env:BUILD_ENV}-${env:CoreId}. This solution is recommended especially when the same stack needs to be deployed into multiple IoT devices as it easily identifies and manages a group of IoT resources in a different location.
  • Order Processor: This service was created to process data from IoT Edge, which will verify prediction results from the camera against the database and handle the payment of an order if it is valid. Upon completion, it will trigger the monitor controller service to pass the order status from AWS back to the device, as well as notify the mobile user via SNS push notification service.
  • Monitor Controller: This service is created to control monitors via an IoT topic, which is subscribed to by monitor handlers on edge devices. Therefore, this service can pass the order status from the cloud to the edge in order to be displayed on the monitor at the end.

After finalizing the design of the microservices, the next step is to decide how to organize these microservices, setup a repository and the CI/CD chain in order to kick off development tasks. Mono-repository is quite popular these days but it does not fit well with this application as the deployment chain for IoT edge services is quite different to the microservices hosted in the Cloud. Hence, I decided to create multiple repositories for this large application by organizing microservices with the same CI/CD into one repository.

Microservices provide flexibility to customise the CI/CD chain for each microservice. Creating and maintaining a CI/CD for each service will increase operational time a lot especially for a small team with short development time. In practice, it can be solved by creating a mono-repo for services with the same CI/CD and specifying trigger directories in serverless CI/CD. By doing this, I can deploy services only if specific files have changed. Leveraging the benefit of the Serverless framework, this application is split into 3 repositories in total:

  • Infrastructure Repo: This repo defines fundamental resources for this application, e.g VPC, security group, subnet, RDS instance. This repo is rarely to be changed after deployment.
  • Cloud Service Repo: This repo contains the source code for all microservices to be deployed to the Cloud, including the APIs, payment service, order processor and monitor controller. Note that although the order processor and monitor controller are part of the IoT solution, these two services are defined in this repo so that they can leverage the same CI/CD to deploy the service to the cloud, as well as access to common modules (e.g database module) shared with other microservices. Regarding sharing code and modules between services, Lambda layers support this feature very well and keep the deployment package small. This article gives a good demo on how to setup Lambda layers.
  • IoT Edge Repo: This repo defines the resources for IoT core, group, rules, topics, and all the Lambda functions to be deployed to edge devices.

Jin Zheng

Senior Software Engineer,Sydney

Jin Zheng

Senior Software Engineer,Sydney