Build an SMS-Equipped Twilio Weather Station with an ESP8266, Amazon AWS IoT and Lambda

November 07, 2017

Twilio Powered Weather Station

Today, we're going to look at how to build an ESP8266 weather station which uses Twilio Programmable SMS to send weather updates.  Our station will reply to incoming SMS messages with weather reports as well as send daily scheduled outgoing SMS weather reports.  We'll use AWS IoT as a persistent preference store and MQTT broker and a combination of AWS Lambda and API Gateway to send and receive messages with Twilio.

Prerequisites

We will cover each of these in more detail as we go along, but to build the whole station you'll need:

  • Twilio Account
  • AWS Account
  • ESP8266 (Good options)
  • DHT11 (or 22) Humidity Sensor (Like this)
    • For a bare sensor, a 4.7k or 10k resistor is needed as well. (Included with the Adafruit product)
  • BMP180 (or 085) Barometric Pressure Sensor (Like This)
  • Male to Male Jumper Wires (Like these)
  • Solderless Breadboard (Like this)

Depending on your choice of ESP8266 development board (or breakout), you may also need:

And of course, our code: 

Editor: we moved the repository when migrating this post to the Twilio Blog. The files you need to complete this tutorial are now pasted in full in this post. They were all in the same directory.

 

You'll need:

  • TwilioLambdaHelper.cpp
  • TwilioLambdaHelper.hpp
  • TwilioWeatherStation.cpp
  • TwilioWeatherStation.hpp
  • twilio-weather-station-esp8266-iot.ino

Additionally, you will need one Python script to forward messages from Lambda to your ESP: lambda_awsws.py

Why a Weather Station?

We've written four guides about using Twilio (and in two cases, an ESP8266) with the AWS ecosystem:

This tutorial wraps all of them up with an end-to-end application that does something useful - watches the weather wherever you want it to.  Remote monitoring is a major IoT application, and hopefully this station gives you some ideas for your own product.

This build allows us to put something together which incorporates the requirements of many applications you might end up building next.  We're going to look at receiving messages via Webhooks from Twilio, sending messages via the Twilio Python Helper Library and AWS Lambda, persisting device state with AWS IoT, and connecting everything over MQTT topics.

Weather Station Architecture

Our architecture is unchanged from our Receiving and Replying to MMS and SMS messages with a ESP8266 and AWS post.

Sending Messages:

Sending Messages with AWS IoT, Lambda, and the ESP8266

Receiving Messages:

Receiving SMS and MMS Messages from Twilio with AWS IoT and Lambda

In addition, we'll be adding a handler for 'Help' and 'Set Preference' messages, which will come in over SMS.  These will return TwiML to Twilio at the Lambda step in the above diagram.  Any successful preference changes will happen in the background on the MQTT topic.

Sign Into a Twilio Account

Our weather station will communicate with you - or whomever you give the number - completely over SMS.  We'll eventually have messages moving both ways - originating from the ESP8266 and pushed to you, as well as messages initiated by you to the 'weather station'.

Log into your Twilio account now - and if you don't have one here is a link to sign up for a free Twilio trial.  Keep that browser tab handy - you will need to refer back to it later.

Find or Purchase a SMS Capable Number

For this demo, you'll require a SMS capable number you already own (or purchase now).

Go to the Twilio Console and select the hash tag/Phone Numbers ('#') section on the left side.  Navigate to your currently active numbers.  

Under capabilities, you'll want to look for one with SMS capabilities, as shown in the below image:

Active SMS Capable Numbers

 

If you don't yet have a number with the SMS icon, you'll need to buy one.  Navigate to the 'Buy a Number' link and click the SMS checkbox:

Buy a number with sms capabilities

 

After purchasing, leave the Console tab open and you'll be ready to head to Amazon Web Services.

Log Into - or Sign Up For - Amazon Web Services

A lot of the weather station's infrastructure will use AWS products.  AWS will handle passing messages to and from the ESP8266, persisting state in a power loss for the ESP, helping the end user change settings, and handling webhook requests from Twilio.

Mocking an API With API Gateway

In the last section, you set up a framework for the ESP8266 to stay connected to AWS.  We also need a plan for the outside world to talk to our weather station.

When you send a text message to a Twilio number, Twilio forwards it to a webhook at a URI you define.  In order to expose a resource with AWS, we use the API Gateway service.

We first went covered this process in our receiving and replying to SMS and MMS messages using Amazon Lambda guide, but we will describe it briefly here.

  1. From the API Gateway console, use the 'Create API' button.  
  2. Name and describe your API a friendly name and description (for your own reference), then create it.  
  3. Create a new resource 'message' (at '/message')
  4. Create a 'POST' method on message.
  5. Select 'Mock' as your 'Integration type'.

Mock API Integration Type

 For now, that's it for API Gateway.  We'll be returning later when Lambda is ready.

Highly Suggested - Set Up a New IAM User

While you can use your main AWS account's credentials for the weather station, best practice with the ESP8266 is to create a new IAM user with IoT permissions.  Go to the IAM console, select 'Users' in the left pane and click the blue 'Add User' button.  

Name your user and click the box to add Programmatic Access:

Adding an IAM User for AWS IoT

Click through to the 'Permissions' step and select 'Attach Existing Policies Directly'.  Add every IoT permission, as we did in this picture:

AWS IoT IAM User Permission

Download the CSV with the user credentials from the success screen.  Those will be used in the settings at the top of the .ino file the ESP8266 uses.

Using Lambda To Send Outgoing Messages

While the ESP8266 does have the capability to use the Twilio API directly to send messages (with a few shortcuts, like verifying a certificate fingerprint), we're going to use the AWS infrastructure to send messages via a Lambda function.  To demonstrate the flexibility of this approach, we're going to use the same setup as we did in the sending SMS and MMS Messages with the ESP8266 and Lambda article.

We'll discuss what you need to do briefly here, but for the full details of each step please visit that article.

Having Outgoing Messages Trigger Lambda

  1. Create a new Lambda function in the same region as you've set up AWS IoT (use the blank template).  Configure it to use AWS IoT as a trigger.  
  2. Check the box for 'Enable Trigger' and name it something memorable.  'IoT Type' should be 'Custom IoT Rule'.
  3. The SQL statement you should use to trigger Lambda is:
    SELECT * FROM 'twilio' WHERE Type='Outgoing'
  4. Here's how you do that from the configuration screen:

Integrating Amazon Lambda and AWS IoT
  • Hit 'Next' to go to the function stage.
  • Leave a Blank Function

    1. Change the runtime to 'Python 2.7'.  
    2. Set the 'Handler' field to 'twilio_functions.iot_handler' in preparation for our upcoming code upload.  
    3. Name the function 'myTwilioSendMessage' and use whatever description you desire.
    4. Use 'Create new role from template(s)' using the AWS IoT Button Permissions, and naming the role something memorable.  

    You're now ready - create the blank function!

    Change the SQL Version

    Return to the AWS IoT console briefly (perhaps even in a new tab) - we've got to do one more step so AWS and the ESP8266 play nicely.

    1. Click the 'Rules' link in the left sidepane.  
    2. You should then see the new rule you've created for the send Lambda function; click it to see details.
    3. In the 'Rule query statement' section, click the 'Edit' link, and change the 'Using SQL version' to '2015-10-08':

Amazon SQL Version Selector in AWS IoT

 

Hit the 'Update' button and your ESP8266 will be ready to fire your rule!

Add some Python Code to Lambda

It's time to add some code.  

  1. On your computer, create a new folder and install the Twilio Python Library manually inside.  
  2. From the GitHub repository, bring in everything from the 'Lambda Function Send SMS' directory (see our earlier guide for detailed help on adding external packages, and read the companion article if you want more details on that function).  Zip the contents of that directory up.
  3. Inside Lambda, select 'Upload a .ZIP File' on the 'Code' tab.  Using 'Upload', select the zip file you just created and upload it by saving.  
  4. Double check you're using the Python 2.7 runtime, and that the Handler is set to 'twilio_functions.iot_handler'.

You won't be able to edit the code inline with the newest helper library, so if you have issues make your updates offline then upload again.

Set the Environmental Variables

In Lambda, Environmental Variables are set on the 'Code' tab.  From the Twilio Console (you do have the tab still open, right?), find your authorization token and account ID to populate:

  • AUTH_TOKEN
  • ACCOUNT_SID

And with that, you should have the plumbing set to send messages from the weather station.  You can double check the 'Triggers' tab to make sure that IoT is triggering this function based upon the SQL query we already entered.

If that checks out, we can move on...

Using Lambda to Build a SMS-Tree

We're going to model the user facing portion of the Weather Station as a SMS-Tree, much like the Phone Trees Twilio makes so easy to create.  Philosophically, our SMS-Tree will be stateless and idempotent - that is, performing actions will not require multiple steps and running the same command multiple times will leave the device in the same state.

For authorization and security we are employing a few mechanisms:

  • We use the Python Twilio Helper Library's assistance in to verify a Webhook originates from Twilio (see our detailed writeup).
  • We use a 'Master Number' that is allowed to make changes to the settings (although other users can see status other than the master_number).

Create a User Facing Lambda Function

Using similar steps, create a new blank Lambda function named 'twilioWeatherStation' with no trigger.  The 'Handler' should be set to 'twilio_functions.twilio_webhook_handler', and create a zip file which contains the code in the 'Lambda Function Weather' directory and the Twilio Python Helper Library.  For details on loading external libraries into Lambda, see this article.

This function needs five environmental variables:

  • AUTH_TOKEN - Twilio Auth Token, from the Console
  • AWS_TOPIC - 'twilio' without quotes
  • THING_NAME - whatever you named the thing, we used 'Twilio_ESP8266_Weather_Station'
  • AWS_IOT_REGION - the region of the Thing above
  • REQUEST_URL - This will be the request URL of API Gateway, and will exactly match Twilio - so leave it blank for now.

Under role, select 'Create a New Role From Template', selecting 'AWS IoT Button permissions'.  Give that role a descriptive name - such as 'twilio_weather' - and continue.  Next we'll give that role some special permissions.

Allowing Lambda to Post to MQTT Topics and Update the Device Shadow

Since this Lambda function is both publishing to MQTT topics as well as updating and checking the Thing Shadow, it is going to need extra permissions.  We're going to enable them through IAM using an inline policy.

  1. Go to IAM by using the 'My Security Credentials' in your name pulldown:
Go to Amazon IAM With The Menu Near Your Name
  • Click on the 'Roles' link in the left sidebar.
  • Find the role you assigned to this Lambda function and click on it.
  • Click on the 'Create Role Policy' button under 'Inline Policies':
Create Inline Role Policy in Amazon IAM
  • add an inline policy which allows this function free reign in IoT:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "iot:*"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
  • Now your Lambda function can perform every action it needs to in IoT - and for this weather station, we'll be using quite a few.

    Next let's do a bit of a deep dive and highlight some of the features of the Lambda server code.

    What Are Our SMS-Tree Functions?

    Our SMS-Tree will help the user, let the user change settings, and send the user a weather report.

    Help

    We build a number of 'Help' responses to guide our users when updating settings.  There will be a general help message, individual help on each specific setting, and an overview of the currently set settings.

"""
Handle 'help' and 'set' for a weather station.

Here we show the infrastructure for a weather station run on AWS IoT, API
Gateway, and Lambda.  We handle 'set' and 'help' messages directly in Lambda
and either change settings or return assistance.

As a webhook comes in, we'll verify the webhook is from Twilio then extract the
key information and send to our IoT device(s) subscribed to the 'twilio'
topic.  Devices on that channel will then react, either confirming a changed
setting or replying with the weather.
"""
from __future__ import print_function

import json
import os
import boto3
import urllib
import time
import datetime
import calendar
from datetime import timedelta
from twilio.request_validator import RequestValidator
from twilio import twiml

# The six preferences supported in the demo application
topic_list = ["alt", "tz", "m_num", "t_num", "alarm", "units"]


def ret_int(potential):
    """Utility function to check the input is an int, including negative."""
    try:
        return int(potential)
    except:
        return None


def handle_help(body, from_number):
    """
    Handle Help for incoming SMSes.

    Remind the user how to interact with the weather station, to be able to
    change settings on the fly.  We also demonstrate reporting back the
    Device Shadow from AWS IoT, and redact the sensitive information
    (For this demo App, it is 'Master Number')
    """
    r = twiml.Response()
    word_list = body.split(' ')

    if (len(word_list) < 2 or
            (word_list[1].lower() not in topic_list and
                word_list[1].lower() != "cur")):
        our_response = \
            ":: Help (var)\n" \
            "alt - Altitude\n" \
            "cur - Current Set\n" \
            "m_num - Master number\n" \
            "t_num - Twilio Number\n" \
            "alarm - Alarm\n" \
            "units - Units\n" \
            "tz - Timezone"
        r.message(our_response)
        return str(r)

    if word_list[1].lower() == "alt":
        our_response = \
            ":: Help alt\n" \
            "Set altitude in meters (integers):\n" \
            "set alt 50\n"
        r.message(our_response)
        return str(r)

    if word_list[1].lower() == "tz":
        our_response = \
            ":: Help tz\n" \
            "Set timezone adjust in minutes:\n" \
            "set tz -480\n"
        r.message(our_response)
        return str(r)

    if word_list[1].lower() == "m_num":
        our_response = \
            ":: Help m_num\n" \
            "Set master number:\n" \
            "set m_num +18005551212\n"
        r.message(our_response)
        return str(r)

    if word_list[1].lower() == "t_num":
        our_response = \
            ":: Help t_num\n" \
            "Set Twilio number:\n" \
            "set t_num +18005551212\n"
        r.message(our_response)
        return str(r)

    if word_list[1].lower() == "alarm":
        our_response = \
            ":: Help alarm\n" \
            "Set alarm hours:minutes, 24 hour clock:\n" \
            "set alarm 15:12\n"
        r.message(our_response)
        return str(r)

    if word_list[1].lower() == "units":
        our_response = \
            ":: Help units\n" \
            "Set units type:\n" \
            "set units imperial\nor\n" \
            "set units metric"
        r.message(our_response)
        return str(r)

    if word_list[1].lower() == "cur":
        aws_region = os.environ['AWS_IOT_REGION']
        client = boto3.client('iot-data', region_name=aws_region)

        aws_response = client.get_thing_shadow(
            thingName=os.environ['THING_NAME']
        )
        from_aws = {}
        if u'payload' in aws_response:
            our_response = ""
            from_aws = json.loads(aws_response[u'payload'].read())

            if u'state' in from_aws and u'desired' in from_aws[u'state']:
                desired = from_aws[u'state'][u'desired']
                if u'tz' in desired:
                    our_response += 'tz: ' + str(desired[u'tz']) + '\n'
                if u't_num' in desired:
                    our_response += 't_num: ' + str(desired[u't_num']) + '\n'
                if u'm_num' in desired and from_number == desired[u'm_num']:
                    our_response += 'm_num: ' + str(desired[u'm_num']) + '\n'
                if u'm_num' in desired and from_number != desired[u'm_num']:
                    our_response += 'm_num: (not this number)\n'
                if u'alarm' in desired:
                    our_response += 'alarm: ' + str(desired[u'alarm']) + '\n'
                if u'units' in desired:
                    our_response += 'units: ' + str(desired[u'units']) + '\n'
                if u'alt' in desired:
                    our_response += 'alt: ' + str(desired[u'alt']) + '\n'
            else:
                our_response = \
                    "No shadow set, set it through AWS IoT.\n"
                r.message(our_response)
                return str(r)
        else:
                our_response = \
                    "No Thing found, set it up through AWS IoT.\n"
                r.message(our_response)
                return str(r)

        r.message(our_response)
        return str(r)

    r.message("Did not understand, try '?' perhaps?")
    return str(r)


def handle_set(body, from_number):
    """
    Handle Changing preferences with incoming SMSes.

    This function will parse (crudely) an incoming SMS message and update
    the current device shadow state with the new setting.  We do some very
    basic checking, - in production you'll want to do much more extensive
    parsing and checking here.

    Additionally, we demonstrate very basic security - only the Master Number
    can update this Thing.
    """
    r = twiml.Response()
    word_list = body.split(' ')

    aws_region = os.environ['AWS_IOT_REGION']
    client = boto3.client('iot-data', region_name=aws_region)

    aws_response = client.get_thing_shadow(
        thingName=os.environ['THING_NAME']
    )

    # Check this person is authorized to change things.
    time_zone_shadow = 0
    if u'payload' in aws_response:
        from_aws = json.loads(aws_response[u'payload'].read())

        if u'state' in from_aws and u'desired' in from_aws[u'state']:
            desired = from_aws[u'state'][u'desired']
            if u'm_num' in desired and from_number == desired[u'm_num']:
                # This person _is_ authorized to make changes.
                # Put any extra logic here you need from the shadow state.
                time_zone_shadow = int(desired['tz'])
            elif u'm_num' in desired and from_number != desired[u'm_num']:
                # This person _is not_ authorized to make changes
                # Put any handling logic here
                our_response = "UNAUTHORIZED!"
                r.message(our_response)
                return str(r)

            # If no Shadow or no master number set, we'll fall through.
            pass

    # Trap invalid 'sets' and make sure we have a 0, 1, and 2 index.
    if len(word_list) != 3 or word_list[1] not in topic_list:
        our_response = \
            "Should be exactly 3 terms:\n" \
            "set <term> <preference>\n" \
            "Perhaps see help with '?'?"
        r.message(our_response)
        return str(r)

    print(word_list[0], word_list[1], word_list[2])

    # Set altitude
    if word_list[1].lower() == "alt":
        # Clean HTML Characters
        word_list[2] = word_list[2].encode('ascii', 'ignore')
        if ret_int(word_list[2]) is None:
            our_response = \
                "Altitude must be an integer, in meters.\n"
            r.message(our_response)
            return str(r)
        else:
            new_alt = ret_int(ord_list[2])
            from_aws[u'state'][u'desired'][u'alt'] = new_alt
            our_response = \
                "Updating altitude to " + str(new_alt) + "m.\n"
            r.message(our_response)

            client.update_thing_shadow(
                thingName=os.environ['THING_NAME'],
                payload=json.dumps(from_aws)
            )

            return str(r)

    # Set timezone
    if word_list[1].lower() == "tz":
        # Clean HTML Characters
        word_list[2] = word_list[2].encode('ascii', 'ignore')
        if ret_int(word_list[2]) is None:
            our_response = \
                "Timezone must be an integer, in minutes.\n"
            r.message(our_response)
            return str(r)
        else:
            new_tz = ret_int(word_list[2])

            from_aws[u'state'][u'desired'][u'tz'] = new_tz
            our_response = \
                "Updating timezone to " + str(new_tz) + " min.\n"
            r.message(our_response)

            client.update_thing_shadow(
                thingName=os.environ['THING_NAME'],
                payload=json.dumps(from_aws)
            )

            return str(r)

    if word_list[1].lower() == "m_num":
        new_mnum = word_list[2]

        if not word_list[2].startswith(u"+"):
            our_response = \
                "Number must start with '+' followed by county + local " \
                "code then phone number ie '+18005551212\n"
            r.message(our_response)
            return str(r)

        else:
            from_aws[u'state'][u'desired'][u'm_num'] = new_mnum
            our_response = \
                "Updating master number to " + str(new_mnum) + ".\n"
            r.message(our_response)

            client.update_thing_shadow(
                thingName=os.environ['THING_NAME'],
                payload=json.dumps(from_aws)
            )

            return str(r)

    if word_list[1].lower() == "t_num":
        new_tnum = word_list[2]

        if not word_list[2].startswith(u"+"):
            our_response = \
                "Number must start with '+' followed by county + local " \
                "code then phone number ie '+18005551212'\n"
            r.message(our_response)
            return str(r)

        else:
            from_aws[u'state'][u'desired'][u't_num'] = new_tnum
            our_response = \
                "Updating Twilio number to " + str(new_tnum) + \
                ".  Update webhook in Twilio console too!\n"
            r.message(our_response)

            client.update_thing_shadow(
                thingName=os.environ['THING_NAME'],
                payload=json.dumps(from_aws)
            )

            return str(r)

    if word_list[1].lower() == "alarm":
        split_alarm = word_list[2].split(":")
        if (len(split_alarm) != 2 or
                ret_int(split_alarm[0]) is None or
                ret_int(split_alarm[1]) is None):
            our_response = \
                "Alarm must be in XX:YY format, will adjust to local" \
                " timezone automatically.\n"
            r.message(our_response)
            return str(r)
        else:
            epoch_time = int(time.time())
            # ESP's library uses a _local_ timestamp
            epoch_time += 60 * time_zone_shadow

            print("Time is " + str(epoch_time))

            current_datetime = datetime.date.today()
            proposed_time = datetime.datetime(
                current_datetime.year,
                current_datetime.month,
                current_datetime.day,
                ret_int(split_alarm[0]),
                ret_int(split_alarm[1])
            )

            # proposed_time += timedelta(minutes=((-1) * time_zone_shadow))
            proposed_epoch = calendar.timegm(proposed_time.timetuple())
            print("Proposed: " + str(proposed_epoch))

            if proposed_epoch < epoch_time:
                # This already passed today, use tomorrow.
                proposed_time += timedelta(days=1)
                proposed_epoch = calendar.timegm(proposed_time.timetuple())
                print("New Proposed: " + str(proposed_epoch))

            our_response = \
                "Updating alarm to " + str(proposed_epoch) + "."
            r.message(our_response)

            from_aws[u'state'][u'desired'][u'alarm'] = proposed_epoch
            client.update_thing_shadow(
                thingName=os.environ['THING_NAME'],
                payload=json.dumps(from_aws)
            )
            return str(r)

    if word_list[1].lower() == "units":
        if word_list[2].lower() not in [u'imperial', u'metric']:
            our_response = \
                "Must be 'imperial' or 'metric' units, sans quotes."
            r.message(our_response)
            return str(r)
        else:
            new_units = word_list[2].lower()

            from_aws[u'state'][u'desired'][u'units'] = new_units
            our_response = \
                "Updating units to " + str(new_units) + "."
            r.message(our_response)

            client.update_thing_shadow(
                thingName=os.environ['THING_NAME'],
                payload=json.dumps(from_aws)
            )

            return str(r)

    r.message("Did not understand, try '?' perhaps?")
    return str(r)


def twilio_webhook_handler(event, context):
    """
    Main entry function for Twilio Webhooks.

    This function is called when a new Webhook coems in from Twilio,
    pointed at the associated API Gateway.  We divide messages into three
    buckets:

    1) 'Help' messages
    2) 'Set' messages
    3) 'Catch-all'/default, which we forward along to the weather station for
       an update of conditions.
    """

    print("Received event: " + str(event))
    null_response = '<?xml version=\"1.0\" encoding=\"UTF-8\"?>' + \
                    '<Response></Response>'

    # Trap no X-Twilio-Signature Header
    if u'twilioSignature' not in event:
        print("NO HEADER")
        return null_response

    form_parameters = {
        k: urllib.unquote_plus(v) for k, v in event.items()
        if k != u'twilioSignature'
    }

    validator = RequestValidator(os.environ['AUTH_TOKEN'])
    request_valid = validator.validate(
        os.environ['REQUEST_URL'],
        form_parameters,
        event[u'twilioSignature']
    )

    # Trap invalid requests not from Twilio
    if not request_valid:
        print("NOT VALID")
        return null_response

    # Trap fields missing
    if u'Body' not in form_parameters or u'To' not in form_parameters \
            or u'From' not in form_parameters:
        print("MISSING STUFF")
        return null_response

    body = form_parameters[u'Body']

    # Determine the type of incoming message...
    if ((len(body) > 0 and
            body[0] == u'?') or
        (len(body) > 3 and
            body.lower().startswith(u'help'))):
        # This is for handling 'Help' messages.

        # Note that Twilio will catch the default 'HELP', so we also need to
        # alias it to something else, like '?'.  We also add a help
        # response for each possible setting.
        # See: https://support.twilio.com/hc/en-us/articles/223134027-Twilio-support-for-STOP-BLOCK-and-CANCEL-SMS-STOP-filtering-
        # Or: https://support.twilio.com/hc/en-us/articles/223181748-Customizing-HELP-STOP-messages-for-SMS-filtering
        return handle_help(
            form_parameters[u'Body'],
            form_parameters[u'From']
        )

    elif ((len(body) > 1 and
            body[0].lower() == u's' and
            body[1] == u' ') or
          (len(form_parameters[u'Body']) > 3 and
            body.lower().startswith(u'set') and
            body[3] == u' ')):
        # This is for handling 'Set' messages.

        # Again, we also handle an alias of 's', and have a handler for each
        # preference.
        return handle_set(
            form_parameters[u'Body'],
            form_parameters[u'From']
        )

    else:

        aws_region = os.environ['AWS_IOT_REGION']
        aws_topic = os.environ['AWS_TOPIC']
        client = boto3.client('iot-data', region_name=aws_region)

        client.publish(
            topic=aws_topic,
            qos=0,
            payload=json.dumps({
                "To": form_parameters[u'To'],
                "From": form_parameters[u'From'],
                "Body": "Give me some weather!",
                "Type": "Incoming"
            })
        )

        # A blank response informs Twilio not to take any actions.
        # Since we are reacting asynchronously, if we are to respond
        # it will come from the weather station.
        return null_response

Both '?' and 'help' are used to retrieve help. Note that by default for a ten-digit phone number Twilio will handle any incoming SMS with a body of 'help'.  'help' messages with multiple words will still be passed through to Lambda.

Set

If the user is authorized, we allow her to change the settings which affect the behavior of the station.  We have included some very basic input validity checks to ensure, for example, that settings which require numbers are only set to numbers.

Triggering Lambda from API Gateway

Go back to the API Gateway console, and navigate to the API we were building up before.  We're going to use it now to trigger our new Lambda function.

The receiving and replying to SMS and MMS messages from Lambda article has the full steps you'll need to perform to have a working API for Twilio to call, but we'll summarize here.

  1. In 'Method Request', add X-Twilio-Signature HTTP Request Header to pass through to Lambda.
  2. In 'Integration Request', remove any Body Mapping Templates and add one for application/x-www-form-urlencoded with a template of:
    #set($httpPost = $input.path('$').split("&"))
    {
    "twilioSignature": "$input.params('X-Twilio-Signature')",
    #foreach( $kvPair in $httpPost )
     #set($kvTokenised = $kvPair.split("="))
     #if( $kvTokenised.size() > 1 )
       "$kvTokenised[0]" : "$kvTokenised[1]"#if( $foreach.hasNext ),#end
     #else
       "$kvTokenised[0]" : ""#if( $foreach.hasNext ),#end
     #end
    #end
    }
  3. In 'Integration Response', remove any Body Mapping Templates and add a new one in HTTP status code 200 for application/xml.  Use this two line template:
    #set($inputRoot = $input.path('$')) 
    $inputRoot​
  4. In 'Method Response', for HTTP Status 200 remove any existing response bodies and add one for application/xml.

Now, from the 'Action' pulldown menu, 'Deploy API':

Deploy an API in API Gateway

Create a new stage with a name of 'prod' and your choice of description. 

When you deploy, Amazon will assign a URL to the new /message route.  Copy the entire thing, including the /message.  Back in the weather station Lambda function's 'Code' tab, paste the exact URL into the REQUEST_URL environmental variable.

Next, even though we haven't even added our hardware yet(!), we're going to plug everything together and test it with Twilio.

Configure Your Twilio Webhook URL

In the Twilio Console browser tab, navigate to the Numbers Section in the sidebar (#).  Select the number you set up in the beginning of this tutorial.

Under 'Messaging' and in 'A Message Comes In', select 'Webhook' and paste the API Gateway URL into the text box (highlighted below).  Ensure 'HTTP POST' is selected.

Configure a Webhook

 

Backup Webhook URL

While in many applications you should plan for the failure of the first Webhook, for this weather station we're okay having a single point of failure.  

Just be aware - when you are building a Twilio application and the primary Webhook fails (sends back a 5xx HTTP response or times out), Twilio will failver to the backup web hook.  That extra piece of machinery is perfect for ensuring you maximize the number of 9s in your uptime.

Text Your Twilio Number for Help or Settings

As noted, we haven't added any hardware yet so the weather will have to wait... but we can try out the set/help machinery.

Try texting the number from your cell phone now, with a single question mark ('?') in the body.

If everything is working together, you should see the 'Hello, World!' of this application - a friendly (but terse, admittedly!) message from Lambda about the various help topics.

You can 'set' variables too, such as 'tz' for timezone (in minutes):

  • set tz -480
    (Pacific Standard time: -8 hours * 60 minutes).
  • set tz 540
    (Japan standard time: +9 hours * 60 minutes).

Hopefully you get something back!

If not, trace back through the chain, starting with the Twilio Debugger.  For each step in the chain, check the logs - after the debugger, check CloudWatch for any Lambda or API Gateway logs, and finally (for 'set' messages only), check for MQTT messages in the Test MQTT clients.

If it all works, most of the software is done - it's time to move to the board.  It's time - let's build the actual station itself!

The ESP8266 Weather Station - Board and Software

At this point, we're ready to put together the ESP8266 part of the chain.  If you haven't yet purchased a board, the repository for Arduino on ESP8266 has a nice list of tested boards. We'll be building the weather station on a solderless breadboard with pre-made jumper wires.  That in mind, we'd suggest sticking with a full sized development board until it's working... after that you can decide if you'd like to miniaturize the setup.

To develop this guide, we used a Sparkfun Thing and Sparkfun's Basic FTDI breakout. The Sparkfun Thing overloads the DTR pin for ease of programming. This causes problems with the hardware serial port when monitoring from inside the Arduino IDE. We find it easier to use SoftwareSerial for simple text debugging, but have left the choice of serial port (or none) as a setting in the code - you can use the USE_SOFTWARE_SERIAL and USE_HARDWARE_SERIAL to tell the preprocessor how to compile serial support in the code.

Arduino IDE

For the greatest ease of use, we developed the ESP8266 code for this tutorial in the Arduino IDE.  Help building the station using another toolchain is outside the scope of this tutorial.  While we encourage you to try, please use Arduino to get a working setup first.

Adding ESP8266 Support to Arduino

If you haven't added new board support to Arduino before, we'll walk through that now - feel free to skip ahead if the ESP8266 is already added to your setup.

In your preferences, add a new URL to the 'Additional Board URLs' section: "http://arduino.esp8266.com/stable/package_esp8266com_index.json".

Add Board to Arduino

Next, in the 'Tools' menu, select 'Boards Manager'.  It will automatically update - after a second, search for the ESP8266 and install the most recent version.

Adding Libraries to Arduino

We're relying on quite a few libraries for the guide today:

Using Arduino's Library Manager is possible for two of the libraries, but the others must be added manually. For a complete overview of library management on Arduino, see the official documentation.

Add Through Library Manager (Search)

  • Adafruit BMP085 Unified
  • Adafruit Unified Sensor
  • DHT Sensor Library
  • NTPClient
  • ArduinoJSON
  • WebSockets

Add Manually to Arduino

The easiest way to install these libraries in the Arduino IDE is to install from a downloaded zip file.

This can be done directly from the ZIP Library Installer in the Arduino IDE:

'Sketch' Menu -> 'Add .ZIP Library' -> select downloaded .zip file

Add ZIP Library to Arduino

Download links:

Compiling The Weather Station Code

We've bottled up much of the complexity of the weather station in the TwilioLambdaHelper (first seen in this article) and TwilioWeatherStation classes.  In those classes, we manage a lot of the heartbeat functions which need to be fired off on certain timers.  We also wrap up things like managing time and reading the sensors.

Both 's' and 'set' are used to perform a set command.

Report

If the message is not a 'Help' or a 'Set' message, we pass it through to the ESP8266.  The logic there is simple - for any SMS or MMS it receives, it will send back the last weather report.

/*
 * Twilio Weather Station, based upon our previous guides on Lambda.
 * 
 * This application demonstrates an ESP8266 Weather Station which sends
 * reports over SMS with Twilio.  It also has the ability to receive text
 * messages, and it will respond with the current conditions.
 * 
 * It also demonstrates receiving an SMS or MMS via AWS API Gateway, Lambda, 
 * and AWS IoT.  An empty response is returned at the Lambda level and the 
 * ESP8266 uses the same path as the sending route to deliver the message.
 * Persistant state is handled by AWS IoT's Device Shadows, which is where
 * we park our preferences.
 * 
 */
 
#include <ESP8266WiFi.h>
#include <WebSocketsClient.h>

// AWS WebSocket Client 
#include "AWSWebSocketClient.h"

// Embedded Paho WebSocket Client
#include <MQTTClient.h>
#include <IPStack.h>
#include <Countdown.h>

// Handle incoming messages
#include <ArduinoJson.h>

// Local Includes
#include <DHT.h>
#include <DHT_U.h>
#include "TwilioLambdaHelper.hpp"
#include "TwilioWeatherStation.hpp"


/* 
 *  Pin configuration - set to the pin the DHT sensor is connected to 
 *  and the type of DHT sensor.
*/
#define DHTPIN 0
#define DHTTYPE DHT11


/* IoT/Network Configuration.  Fill these with the values from WiFi and AWS. */
char wifi_ssid[]                = "YOUR NETWORK";
char wifi_password[]            = "NETWORK PW";
char aws_key[]                  = "AWS KEY";
char aws_secret[]               = "AWS SECRET";
char aws_region[]               = "AWS REGION";
char* aws_endpoint              = "AWS ENDPOINT";

/* 
 *  Configurable Preferences - you can set them here, but they are updated via
 *  SMSes from the master number.
 *  
 *  Don't worry about getting them perfectly right, they are updated from the
 *  device shadow state.  You can even skip them if you'd like, but they are 
 *  useful as a reference if you will be deploying multiple stations.
 */
// Set to the numer which can change preferences (your cell?)
char* master_device_number      = "+18005551212";
// Set to a number you own through Twilio (device default)
char* twilio_device_number      = "+18005551212";
// Time zone offset in minutes
int32_t time_zone_offset        = -480;
// Altitude in meters
int32_t location_altitude       = 60;
// Units in metric or imperial units?
char* unit_type                 = "imperial";
// Alarm in seconds since 1970
int32_t alarm                   = 1488508176000;
const char* shadow_topic        = \
        "$aws/things/<YOUR THING NAME>/shadow/update";

/* MQTT, NTP, WebSocket Settings.  You probably do not need to change these. */
const char* delta_topic         = "twilio/delta";
const char* twilio_topic        = "twilio";
int ssl_port = 443;
// NTP Server - it will get UTC, so the whole world can benefit.  However,
// there is no latency adjustment.  Of course, if we're off by a few
// seconds with a weather station it isn't that bad.
const char* ntp_server          = "time.nist.gov";


/* You can use either software, hardware, or no serial port for debugging. */
#define USE_SOFTWARE_SERIAL 1
#define USE_HARDWARE_SERIAL 0


/* Pointer to the serial object, currently for a Sparkfun ESP8266 Thing */
#if USE_SOFTWARE_SERIAL == 1
#include <SoftwareSerial.h>
extern SoftwareSerial swSer(13, 4, false, 256);
Stream* serial_ptr = &swSer;
#elif USE_HARDWARE_SERIAL == 1
Stream* serial_ptr = &Serial;
#else
Stream* serial_ptr = NULL;
#endif


/* Global TwilioLambdaHelper  */
TwilioLambdaHelper lambdaHelper(
        ssl_port,
        aws_region,
        aws_key,
        aws_secret,
        aws_endpoint,
        serial_ptr
);


/* Global TwilioWeatherStation  */
TwilioWeatherStation* weatherStation;


/* 
 * Our Twilio message handling callback.  In this example, Lambda is doing 
 * most of the filtering - if there is a message on the Twilio channel, we want
 * to reply with the current weather conditions the ESP8266 is seeing.
 */
void handle_incoming_message_twilio(MQTT::MessageData& md)
{     
        MQTT::Message &message = md.message;
        std::unique_ptr<char []> msg(new char[message.payloadlen+1]());
        memcpy (msg.get(),message.payload,message.payloadlen);
        StaticJsonBuffer<maxMQTTpackageSize> jsonBuffer;
        JsonObject& root = jsonBuffer.parseObject(msg.get());
        
        String to_number           = root["To"];
        String from_number         = root["From"];
        String message_body        = root["Body"];
        String message_type        = root["Type"];

        // Only handle messages to the ESP's number
        if (strcmp(to_number.c_str(), twilio_device_number) != 0) {
                return;
        }
        // Only handle incoming messages
        if (!message_type.equals("Incoming")) {
                return;
        }

        // Sending back the current weather to whomever texts the ESP8266
        lambdaHelper.list_message_info(message);
        lambdaHelper.print_to_serial("\n\rNew Message from Twilio!");
        lambdaHelper.print_to_serial("\r\nTo: ");
        lambdaHelper.print_to_serial(to_number);
        lambdaHelper.print_to_serial("\n\rFrom: ");
        lambdaHelper.print_to_serial(from_number);
        lambdaHelper.print_to_serial("\n\r");
        lambdaHelper.print_to_serial(message_body);
        lambdaHelper.print_to_serial("\n\r");

        String weather_string = weatherStation->get_weather_report();
       
        // Send a weather update, reversing the to and from number.
        // So if you copy this line, note the variable switch.
        lambdaHelper.send_twilio_message(
                twilio_topic,
                from_number,
                to_number, 
                weather_string,
                String("")
        );
}


/* 
 * Our device shadow update handler - we'll just dump incoming messages
 * to serial for the weather station.  The 'twilio/delta' channel
 * is where we consume the pared down messages.
 * 
 * We do _post_ to this channel however, to get the initial device
 * shadow on a power cycle and to update the alarm after it goes off.
 */
void handle_incoming_message_shadow(MQTT::MessageData& md)
{
        MQTT::Message &message = md.message;
        
        lambdaHelper.list_message_info(message);
        lambdaHelper.print_to_serial("Current Remaining Heap Size: ");
        lambdaHelper.print_to_serial(ESP.getFreeHeap());

        std::unique_ptr<char []> msg(new char[message.payloadlen+1]());
        memcpy (msg.get(), message.payload, message.payloadlen);

        lambdaHelper.print_to_serial(msg.get());
        lambdaHelper.print_to_serial("\n\r");
}


/* Setup function for the ESP8266 Amazon Lambda Twilio Example */
void setup() 
{
        WiFi.begin(wifi_ssid, wifi_password);
    
        #if USE_SOFTWARE_SERIAL == 1
        swSer.begin(115200);
        #elif USE_HARDWARE_SERIAL == 1
        Serial.begin(115200);
        #endif
        
        while (WiFi.status() != WL_CONNECTED) {
                delay(1000);
                lambdaHelper.print_to_serial(".\r\n");
        }

        lambdaHelper.print_to_serial("Connected to WiFi, IP address: ");
        lambdaHelper.print_to_serial(WiFi.localIP());
        lambdaHelper.print_to_serial("\n\r");

        // See note in TwilioWeatherStation.hpp - the reference to lambdaHelper 
        // is questionable in C++, but we include it here so you can see how 
        // the infrastructure has evolved from our previous examples.
        weatherStation = new TwilioWeatherStation(
                ntp_server,
                DHTPIN,
                DHTTYPE,
                time_zone_offset,
                location_altitude,
                alarm,
                master_device_number,
                twilio_device_number,
                unit_type,
                twilio_topic,
                shadow_topic,
                lambdaHelper
        );

        // Connect to MQTT over Websockets.
        if (lambdaHelper.connectAWS()){
                lambdaHelper.subscribe_to_topic(
                        shadow_topic, 
                        handle_incoming_message_shadow
                );
                lambdaHelper.subscribe_to_topic(
                        delta_topic, 
                        handle_incoming_message_delta
                );
                lambdaHelper.subscribe_to_topic(
                        twilio_topic, 
                        handle_incoming_message_twilio
                );
                weatherStation->report_shadow_state(shadow_topic);         
        }

        // Yield to the Weather Station heartbeat function.
        weatherStation->yield();
        
}


/* 
 * Our Twilio update delta handling callback.  We look for updated parameters
 * and any that we spot we send to the weatherStation for update.
 */
void handle_incoming_message_delta(MQTT::MessageData& md)
{     
        MQTT::Message &message = md.message;
        std::unique_ptr<char []> msg(new char[message.payloadlen+1]());
        memcpy (msg.get(),message.payload,message.payloadlen);

        // List some info to serial
        lambdaHelper.list_message_info(message);
        lambdaHelper.print_to_serial(msg.get());
        lambdaHelper.print_to_serial("\n\r");
        
        
        StaticJsonBuffer<maxMQTTpackageSize> jsonBuffer;
        JsonObject& root = jsonBuffer.parseObject(msg.get());
        
        if (root["state"]["alarm"].success()) {
                int32_t possible_alarm = root["state"]["alarm"];
                weatherStation->update_alarm(possible_alarm);
        }
        if (root["state"]["units"].success()) {
                String possible_units = root["state"]["units"];
                weatherStation->update_units(possible_units);
                //delete[] possible_units;
        }
        if (root["state"]["alt"].success()) {
                int possible_alt = root["state"]["alt"];
                weatherStation->update_alt(possible_alt); 
        }
        if (root["state"]["tz"].success()) {
                int possible_tz = root["state"]["tz"];
                weatherStation->update_tz(possible_tz); 
        }
        if (root["state"]["t_num"].success()) {
                String possible_tnum = root["state"]["t_num"];
                weatherStation->update_tnum(possible_tnum);
                //delete[] possible_tnum;
        }
        if (root["state"]["m_num"].success()) {
                String possible_mnum = root["state"]["m_num"];
                weatherStation->update_mnum(possible_mnum);
                //delete[] possible_mnum;
        }

        // Let AWS IoT know the updated state
        weatherStation->report_shadow_state(shadow_topic);
        
}


/* 
 * Our loop checks that the AWS Client is still connected, and if so calls its
 * yield() function encapsulated in lambdaHelper.  If it isn't connected, the 
 * ESP8266 will attempt to reconnect. 
 */
void loop() {
        if (lambdaHelper.AWSConnected()) {
                lambdaHelper.handleRequests();
        } else {
                // Handle reconnection if necessary.
                if (lambdaHelper.connectAWS()){
                        lambdaHelper.subscribe_to_topic(
                                shadow_topic, 
                                handle_incoming_message_shadow
                        );
                        lambdaHelper.subscribe_to_topic(
                                delta_topic, 
                                handle_incoming_message_delta
                        );
                        lambdaHelper.subscribe_to_topic(
                                twilio_topic, 
                                handle_incoming_message_twilio
                        );
                        weatherStation->report_shadow_state(shadow_topic);
                }
        }
        
        /* Time and weather checking heartbeat */
        weatherStation->yield();
}

Try compiling - and with that, you're ready to build the hardware and flash the board!

The ESP8266 Weather Station - Hardware

This part might be a little tricky since we will all be using different boards.  With the current setup, you will need to use I2C to connect the BMP Pressure sensor.  If you need to move the DHT sensor, update the DHTPIN setting in the code.

DHT11 Data PinESP8266 GPIO0
BMP180 SCL PinESP8266 SCL Pin
BMP180 SCA PinESP8266 SCA Pin

Depending on the DHT11 sensor you purchase, you will need to add a 4.7kΩ resistor between data and 3.3 volts.  These resistors start with Yellow, Violet and Red in a 4-band resistor, and it is drawn on the schematic.  Don't add a second one if it is already onboard.

Remember: Change the exact setup to fit the board and parts you useyou use, and change the pins in the code if you move them around.  We drew up a schematic and a possible breadboard layout with Fritzing - but this is based on what we purchased!

Schematic:

Note: this schematic is for the SparkFun Thing development board.  You might have to vary it for your own hardware!  Boards which don't relabel the pins should have a final circuit something like our diagram.

One colleague has a NodeMCU board and was succesful with the following setup (GPIO pins are in parentheses; they are not labeled on the board):

  • D1 (GPIO5): SCL
  • D2 (GPIO4): SDA
  • D3: DATA pin of the DHT11

 

 

AWS Weather Station Schematic

 One Possible Layout:

AWS Weather Station Breadboard

 

 

Plug it In and Upload Your Code

Triple check your connections, then plug everything in.  If you don't let out any magic smoke, you're well on your way!  (If you did, don't worry about it - it's a rite of passage with hardware - hopefully you've got extras.)

Use the 'Tools' menu in Arduino to select your ESP8266.  We have had the most success keeping our station connected with a speed of 160 MHz.  Choose the proper port and try a slower serial speed such as 115200 for uploading.

And with a compile and an upload hopefully everything just works the first time!  You've now got a weather station which updates every 3 minutes with barometric pressure, humidity and temperature.

Changing Weather Station Settings

There are at least three avenues to change the settings on the station, and you should set the initial state depending on what seems easiest:

Through the Shadow State Directly

  1. From the IoT Console, select 'Things' then click the Weather Station, and go to 'Shadow' on the left side bar.
  2. Under 'Shadow Document', click the 'Edit' link.
  3. Modify the keys and values directly (there are six, as seen in the Lambda code).
  4. Upon 'save', the changes will be published to the various MQTT topics.

Through an SMS from the Master Cell Phone

  1. Text into the weather station 'set <param> <value>' or 's <key> <value?'.
  2. If authorized and making a valid request, the changes will be published to the various MQTT topics.

Publishing to the Shadow State Update Channel

  1. Send properly formatted JSON to the /update topic.
  2. If 'desired' is in the JSON object body as detailed here, the shadow state will update.
  3. On update, the changes will be published to the various MQTT topics.

 

Remote Monitoring: We've Got Weather Updates! What's Next?

This is where you come into play!

We've built this remote monitoring application together which will keep a close eye on the weather.  The sky - whether blue or grey - is the limit, and the station is now yours to customize and make your own.  Add voice, add MMS support with weather icons, even add video (got video working on an ESP8266?  We're hiring...).

We've got a nice selection of Add-ons with some amazing partners as well.  You'll really enjoy how easy the integration is - perhaps see if anything would work well for your application.

Whatever you build, we'd love to hear about it.  Drop us a line on Twitter and show us what you've built!