The SR04 module is a popular ultrasonic distance sensor that is commonly used with microcontrollers like Arduino and Raspberry Pi. It can be used to measure distances from 2cm to 400cm with an accuracy of 0.3cm. In this blog post, we will explore how to use the SR04 module with a Raspberry Pi.

HC-SR04 Module

The HC-SR04 module consists of a transmitter and receiver, and works by sending out a high-frequency sound pulse and timing how long it takes for the pulse to bounce back off an object and return to the sensor. By knowing the speed of sound and the time it takes for the pulse to return, the module can calculate the distance between itself and the object.

Materials

Before starting the project, you will need to gather the following materials:

• Raspberry Pi (any model) - Amazon link
• HC-SR04 sensor - Amazon link
• Breadboard (optional)
• Female-to-female jumper wires - Amazon link

Making the Connections

The SR04 sensor has four pins: VCC, GND, Trig, and Echo. The VCC and GND pins are used to provide power to the sensor. The Trig pin is used to send the ultrasonic pulse, and the Echo pin is used to receive the reflected signal. In this tutorial we will use GPIO14 (board pin 8) for the TRIG pin and GPIO15 (board pin 10) for the ECHO as seen in this diagram:

Writing the Python code

With the SR04 module connected and the required packages installed, we're ready to write a Python script to read the distance measurements from the sensor.

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BOARD)

trig_pin = 8  # GPIO14 pin connected to the Trig pin on SR04 module
echo_pin = 10  # GPIO15 pin connected to the Echo pin on SR04 module

GPIO.setup(trig_pin, GPIO.OUT)
GPIO.setup(echo_pin, GPIO.IN)

def distance():
    # Send a 10us pulse to trigger the SR04 module
    GPIO.output(trig_pin, True)
    time.sleep(0.00001)
    GPIO.output(trig_pin, False)
    
    # Measure the duration of the pulse from the Echo pin
    start_time = time.time()
    while GPIO.input(echo_pin) == 0:
        start_time = time.time()
        
    end_time = time.time()
    while GPIO.input(echo_pin) == 1:
        end_time = time.time()
        
    # Calculate the distance based on the duration of the pulse
    duration = end_time - start_time
    distance = duration * 17150  # speed of sound in cm/s
    distance = round(distance, 2)  # round to two decimal places
    
    return distance

# Main loop
try:
    while True:
        dist = distance()
        print(f"Distance: {dist} cm")
        time.sleep(1)
        
except KeyboardInterrupt:
    GPIO.cleanup()

1. We first import the RPi.GPIO library and the time module. We also set the GPIO mode to BOARD, which means we refer to the GPIO pins by their physical pin numbers on the Pi board.
2. We define the GPIO pins connected to the Trig and Echo pins on the SR04 module.
3. We set up the Trig pin as an output pin and the Echo pin as an input pin.
4. We define a function distance() that sends a 10us pulse to the Trig pin to trigger the SR04 module. It then measures the duration of the pulse from the Echo pin and calculates the distance based on the speed of sound.
5. In the main loop, we call the distance() function and print the distance to the console. We also add a delay of 1 second between measurements.
6. We handle the KeyboardInterrupt exception by cleaning up the GPIO pins.

Run the code

After running your file you should see the distance measurements printed to the console every second. Note that you may need to adjust the GPIO pin numbers to match the pins you have connected to the SR04 module on your Pi.

Conclusion

In this tutorial, we learned how to leverage an SR04 sonar sensor module using a Raspberry Pi and control it using Python code. We hope this tutorial was helpful and inspires you to build your own robotics projects using Raspberry Pi.

Good luck and happy coding!

Raspberry Pi is a credit card-sized computer that can be used for various DIY projects, including robotics. One of the essential components of robotics is a servo motor, which can be used for a variety of purposes such as moving an arm, controlling a camera, or steering a robot. In this tutorial, we will explore how to connect a Raspberry Pi with an MG995 servo motor.

The MG995(R)

The MG995 servo is a type of motorized actuator that is commonly used in robotics and automation projects. It is a high-torque servo motor that can rotate to a specific position based on the electrical signals it receives from a controller.

The MG995 servo operates on a pulse-width modulation (PWM) signal, which is a type of digital signal that is commonly used to control motors and other devices. By varying the width of the PWM signal, the servo can be made to rotate to different positions, allowing for precise control over its movement. We will control the length of this signal with our Raspberry Pi.

Materials

Before starting the project, you will need to gather the following materials:

• Raspberry Pi (any model) - Amazon link
• MG995 Servo Motor - Amazon link
• Breadboard (optional)
• Male-to-female jumper wires - Amazon link

Making the connections

The MG995 servo motor has three wires: red (power), brown (ground), and orange (signal). To connect the servo motor to the Raspberry Pi, follow these steps:

• Connect the red wire of the servo motor to the 5V power source.
• Connect the brown wire of the servo motor to the ground of the Raspberry Pi.
• Connect the orange wire of the servo motor to a GPIO pin 13 of the Raspberry Pi.

Writing the Python code

Now that we have connected the servo motor and installed the required libraries, we can write the Python code to control the servo motor. Open a new file in the terminal window and type the following:

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)
GPIO.setup(13, GPIO.OUT)

p = GPIO.PWM(13, 50)  # PWM frequency is 50Hz
p.start(2.5)  # Initialization

try:
    while True:
        p.ChangeDutyCycle(5)  # Rotate the servo motor to 90 degrees
        time.sleep(1)
        p.ChangeDutyCycle(10)  # Rotate the servo motor to 180 degrees
        time.sleep(1)
except KeyboardInterrupt:
    p.stop()
    GPIO.cleanup()

In the code above, we first import the necessary libraries and set the GPIO mode to BCM. We then set up GPIO pin 13 as an output pin and initialize the PWM signal with a frequency of 50Hz and a duty cycle of 2.5 (which is equivalent to 0 degrees).

We then enter a while loop that will rotate the servo motor to 90 degrees and then 180 degrees with a delay of one second between each rotation. Finally, we handle the KeyboardInterrupt exception and stop the PWM signal and clean up the GPIO pins.

Run the code

After running your file you should see the servo motor should start rotating to 90 degrees and then 180 degrees with a delay of one second between each rotation.

Conclusion

In this tutorial, we learned how to connect an MG995 servo motor to a Raspberry Pi and control it using Python code. We hope this tutorial was helpful and inspires you to build your own robotics projects using Raspberry Pi.

Good luck and happy coding!

Getting started with react takes minutes but production deployments (although quick) can be intimidating. This post documents the step-by-step deployment using Ubuntu 18.04 LTS and NGINX.

Objectives

Time required: 10 minutes

Prerequisites

To follow along, you should have the following:

1. A basic understanding of how JavaScript and React
2. Somewhere to host your application (I will be using a GCP VM instance)

Throughout the article the following domain is used mydomainname.com, this needs to be replaced with the domain name or IP of your server. For SSL and HTTPS setup this guide can be followed.

Step 1 - build your application

Before deploying the application you must build the project which produces a directory with an optimized codebase. This is contained within a build folder and encapsulates everything the application needs to run. If the application is deployed somewhere other than your development system you can either build the code locally and migrate the codebase and build on the deployment machine or migrate just the build directory.

cd project-folder
sudo npm install
sudo npm run build

This creates a build folder directly within the working directory. Before moving to the next step, let's relocate the folder to a typical deployment location /var/www/. Create this /var/www/ folder if it does not exist and copy over the build directory.

sudo mkdir /var/www/
sudo scp -r ./build/* /var/www/build/

Next NGINX needs to be configured to serve this build.

Step 2 - install and configure NGINX

Install NGINX using the following command sudo apt install nginx. This creates an NGINX folder under /etc/nginx/. To host the web app, the default NGINX config file must be updated or a new one created. In this walkthrough, the existing default file is replaced. Change the content of the file /etc/nginx/sites-enabled/ to the following:

server {
    listen 0.0.0.0:80;
    server_name mydomainname.com;
    access_log /var/log/nginx/app.log;
    root /var/www/build;
    index index.html index.htm;
    try_files $uri /index.html;
    location / {
        try_files $uri $uri/ = 404;
    }
}

Note: /var/www/build must coincide with the location of the production build folder from the previous step.

Note: server_name value ( mydomainname.com above) must coincide with the server's access point. This can either be the IP address or domain name.

Step 3 - deploy

For the changes to take effect the NGINX service must be restart:

sudo service nginx stop
sudo service nginx start

The application should now be running and accessible using the server_name provided in the config file.

Good luck and happy coding!

This post covers the implementation of a 2-dimensional quadcopter using the Unity game engine and PID controllers for flight control. This was a recent implementation and the details are documented here in hopes that others find some of this helpful.

Expected Outcomes

Follow the walkthrough in this tutorial should hopefully provide the following:

1. A Unity environment simulating 2-dimensional quadcopter flight ready for experimentation.
2. A basic understanding of how PID controllers can be used in Unity (and elsewhere) as control systems.
3. Basics on rotorcraft 2-dimensional dynamics.

All the covered code as well as the working project can be found in this GitHub repository. If you don’t care about the walkthrough of how and why you can grab the code and skip right to The Simulation section.

Before getting started

A few decisions were made for this simulation.

Why Unity? Unity comes with a sophisticated physics engine and makes rapid development and experimentation quick and easy.

Why Only 2 Dimensions? The problem of rotorcraft control is complex and removing a dimension allows for significant simplification. Once control in 2 dimensions is mastered, 3 dimensions can be tackled.

Why the PID Controller? Experienced Unity developers might wonder why use PID control over classic pathing techniques. The goal is to use a controller that translates to the real world. PID controllers are the go-to control loops for self-regulating systems.

Basics of a Quadcopter in 2D

By starting with simulation in 2 dimensions instead of 3, the complexity is significantly reduced.

IMPORTANT: In this article, the vertical axis is referred to as y and the horizontal axis as x. This is done to keep consistency with the x-y 2D axis in Unity. In standard notation, the horizontal axis would be referred to as y and the vertical axis as z (or x => y and y => z).

Active Forces

The drone has 3 total forces acting on it at any given time:

1. A variable upward force for each propeller perpendicular to the frame and upwards
2. A constant force of gravity directly downwards regardless of the frame’s orientation.

As long as the quadcopter is defined as a rigid body with mass, Unity takes care of the gravitational force. The force (or thrust) of the rotors is what the control system has to manage.

Control

Acceleration of the simulated quadcopter can be controlled by changing the thrust of the rotors.

When the quadcopter is level with the ground, acceleration is only applied along the y axis. Increasing the thrust of the rotors in this orientation causes the quadcopter to accelerate up and decreasing thrust causes it to accelerate down.

When the quadcopter is tilted at an angle (we will call this angle phi or 𝜙) relative to the ground the force from the propellers begins to apply both horizontal and vertical acceleration. Maintaining thrust in this orientation increases horizontal velocity (y-axis).

or …

Lastly, in order to change the angular orientation of the quadcopter (or roll), the thrust between the rotors needs to vary in order to produce torque (or moment). This moment is proportional to the difference between the forces and the distance between the rotors.

NOTE: Quadcopters in 3 dimensions also need to control yaw, which is torque (or moment) arising from the motors spinning. Luckily with only 2D there is no third dimension for the drone to yaw around.

The dynamics above allow for a fairly simple control system where only the overall thrust for the quadcopter’s linear acceleration and the difference in forces between the rotors for rotation need to be determined. This is typically expressed in terms of u1 and u2, where u1 dictates total thrust and u2 the total moment.

These are the values that the PID controller needs to determine.

The Control System

The PID controller is a control loop approach that continuously supplies a system with an input (like thrust, voltage, resistance, etc.) and adjusts this input based on how well the system performs over time.

The general idea revolves around providing the PID controller with 3 coefficients — Position, Integral, and Derivative. These coefficients are used by the algorithm to determine how to scale the output control value as the system gets closer or further away from its goal. We won’t get into the details behind exactly how PID controllers work but getting more knowledge on the subject is highly recommended. I have found the series here to be an excellent introduction.

The system uses 2 PID controllers to solve the 2D quadcopter control problem;

Altitude controller — Provides the drone with a thrust value required to drive the quadcopter to a desired high.

Attitude controller — Provides the drone with moment values to help stabilize the aircraft.

Here is what the control loop looks like. Note that only the desired elevation (y-axis) is provided as an overall system input. The controller’s job is to drive the quadcopter to that altitude while ensuring a balanced orientation.

Position Driver — Represents the general control loop input — altitude.

Error Estimator — Component responsible for providing the PID controllers with the deviation between the desired state (altitude and neutral orientation) and actual values.

PID Altitude Controller — This PID controller uses the position error from the error estimator to derive a u1 value or the overall desired vertical thrust.

PID Attitude Controller — This PID controller uses the orientation error from the error estimator to derive a u2 value or the required torque.

Motor Mixing Algorithm (MMA) — This algorithm uses u1, u2 and combines them with the drone’s current orientation to determine the required total thrust for each rotor.

Plant / Drone — The physical (or virtual) drone that interacts with the environment and provides feedback to the system with actual state values.

The Code

With all of that out of the way let’s get to building the actual simulation. Unity is used due to its built-in physics engine. The GitHub repository can be found here containing a working instance of the simulation.

One might be able to get away without any Unity experience but getting at least some familiarity is highly recommended even if you don’t have any game development aspirations. Brackeys has some fantastic resources such as this one that can get you started.

The folder structure

The project was organized into the following file structure.

├───Prefabs
│   ├───Quadcopter - prefab of the complete quadcopter
│   └───Thruster - prefab for a thruster object
├───Scenes
│   └───Main - the only scene in the project
├───Scripts
│   ├───PIDController - PID script obejcts
│   ├───Quadcopter - script for the quadcopter object
│   ├───FlightController - script to bring everything together
│   └───Thruster - script containing control for rotor objects

The scene

All scene and camera default values are kept and the scene is only updated with the following objects:

Quadcopter — The quadcopter, contains a flight controller, a rigid body frame, and two thrusters.

Platform — A static rigid body providing the quadcopter with a stable liftoff surface.

Event System — Default event system object which was disabled as it is not used.

Main Camera — Default camera object.

The code

The repository with all the code can be found here. The script objects are described at a high level with a few notable lines of code called out.

Thruster.cs

This script is associated with each rotor and provides the quadcopter with thrust. At every physics engine tick, the rotor script computes the force it should be producing using a thrust coefficient and the simulated blade speed.

Rotor thrust can be updated by calling the setRevolutionTarget() function and passing the desired RPM value. The thruster then updates the RPM value continuously until the target is reached using the updateRevolutionRate() function.

Quadcopter.cs

This script represents the quadcopter object, it is mapped to the attached rotor objects and contains the Motor Mixer Algorithm code.

The u1 value is adjusted to account for gravity by multiplying by Cos(phi), where phi is the drone’s current angular orientation angle. The u2 value (moment input) is applied as a positive to the left rotor and a negative to the right to generate the required moment.

Note that the values as passed directly as units of force to the rotors. This can be done because of the general flexibility of PID control systems. The control loop scales the magnitude of the inputs based on how the system reacts.

PIDController.cs

This is the main PID control script and the core logic is packed into the GetPIDOutput() function. This method is called every time a new value needs to be estimated. p, i, and d are the gain magnitudes, and kP, kI and kD are the coefficients that can be tweaked through the FlightController.cs script to change the PID controller’s performance.

Here is a great resource to learn move about PID control logic.

The integral term is updated using the model’s previous integral value and is intended to provide the model with a type of “memory”. This is great at helping the system adapt to unknown variables but it can also get us into trouble. For example, if the drone is still some significant distance from the target destination but has reached its maximum velocity the integral term will continue to grow past the maximum. This will cause sub-optimal deacceleration as the system will have to undo the winded-up excess. A more detailed explanation can be found here and it discusses the concept of preventing this over-saturation by the clamping (or stopping accumulation) of the integral term if the PID output exceeds the system capability threshold.

FlightController.cs

This is the script that pulls everything together. The desired position and Thrust, and Roll PID coefficient can be set through this object.

At every physics engine update, this script computes the current error state, passes the error to the PID controllers, and feeds the resulting u1 and u2 values through to the drone Motor Mixer Algorithm.

The Simulation

The code in the repo should run as-is out of the box but how the parameters of the simulation can be manipulated are covered below.

Thrusters

The drone has left and right thrusters, these can be manipulated separately, however for best results their parameters should be identical. Each thruster also has a rigid body with a defined mass (0.3) and this rigid body is attached to the parent drone body using a 2D Fixed Joint.

The main Thruster.cs parameters to configure would be the thrust coefficient, maximum RPM, and the spinup rate.

Quadcopter

The quadcopter also has a rigid body with a defined mass (1) along with a simple collider to prevent it from passing through the surface platform.

The rotors and rotor count are set in the Quadcopter.cs script and the target position and PID control coefficients in the FlightControl.cs script.

Running the Code

When the scene is run, the quadcopter lifts off the ground, moves towards the specified altitude, and settles there.

*The red lines are debug lines showing thrust.

The P thrust values can be modified to change how quickly the drone reaches the desired high.

The D thrust value dictates how smooth our trajectory is and can be used to minimize overshooting at a cost of slower convergence.

The I thrust value allows the system to overcome unforeseen disturbances in the environment and have a smoother recovery. The drone uses the integral term value of 2.5 which allows it to account for the force of gravity that was omitted from the model altogether. Here is what it looks like with the integral term set to 0 — the drone fails to reach the target altitude of 10.

The roll PID coefficients provide the same role but for pitch. Here is what it looks like when the orientation and position of the drone are disturbed.

Conclusion

If you got this far you should either have a working 2D quadcopter Unity simulation or at least the knowledge of how to create one. In future articles, I hope to increase the complexity of this simulation, including automated motion planning and the adaptation to 3 dimensions.

Please let me know if I have made any errors or omitted anything. Feel free to post any questions as comments as well.

Happy coding!

Google’s App Engine service is a great way to get web applications up and running with minimal effort. In fact, I have an article on how you can spin up a Node.js application on this platform in about 10 minutes. This service is part of Google’s free tier and getting started with the App Engine is easy but one can struggle with understanding the different configuration options and the pricing structure. This article outlines in three sections what you need to consider when configuring and deploying your applications on the App Engine platform.

Hosting Options — Things to keep in mind when configuring and deploying your App Engine application.

Pricing —How App Engine charges are incurred.

Other Considerations — Some other considerations when deploying an App Engine application.

Let’s get started.

Hosting Options

When deploying an application you must provide instructions to the App Engine on what settings to use via a app.yaml config file. This section aims to help you understand the difference in deployment options and how they can be specified in this file.

Here is an example of a app.yaml file:

runtime: nodejs14
env: standard
instance_class: F1
service: my-node-app
automatic_scaling:    
    max_instances: 2    
    min_instances: 0    
    min_idle_instances: 0    
    max_idle_instances: 1

Runtime

When deploying an application the runtime language has to be specified. Documentation on support runtime environments can be found here and in the config file the runtime attribute is used — here is an example for a deployment using Node.js v14:

runtime: nodejs14
env: standard
instance_class: F1
service: my-node-app
automatic_scaling:    
    max_instances: 2    
    min_instances: 0    
    min_idle_instances: 0    
    max_idle_instances: 1

Environment

You may select from two different environment types — Standard and Flexible. This document outlines the detailed differences between the two options. In short, Flexible environments will cost you much more to run and are recommended only if you are using a language not supported by the Standard environment. Here is the list of supported languages:

• Python 2.7, Python 3.7, Python 3.8, Python 3.9 (preview)
• Java 8, Java 11
• Node.js 8, Node.js 10, Node.js 12, and Node.js 14 (preview)
• PHP 5.5, PHP 7.2, PHP 7.3, and PHP 7.4
• Ruby 2.5, Ruby 2.6, and Ruby 2.7
• Go 1.11, Go 1.12, Go 1.13, Go 1.14, Go 1.15 (preview)

There are other reasons to go Flexible but it should be an exception. You can find the full feature matrix here contrasting environment type capability. The environment type can be specified in the app.yaml file via the env attribute — the values are either flex or standard:

runtime: nodejs14
env: standard
instance_class: F1
service: my-node-app
automatic_scaling:    
    max_instances: 2    
    min_instances: 0    
    min_idle_instances: 0    
    max_idle_instances: 1

Instance Class

Like most other cloud services, when deploying an App Engine application you may select a class that will dictate allocated resources and functionality options.

It’s important to remember that Google’s free tier covers 28 hours of F1or 9 hours of B1 instance usage for free per day.

F instances are considered “Front End Instances” and B instances are considered “Back End Instances”. As you can see in the above table, the main difference is the scaling options available. The instance type can be specified in the app.yaml file via the instance_class attribute:

runtime: nodejs14
env: standard
instance_class: F1
service: my-node-app
automatic_scaling:    
    max_instances: 2    
    min_instances: 0    
    min_idle_instances: 0    
    max_idle_instances: 1

Service

The App Engine runs applications as services. Your first App Engine deployment is associated with the default service but anything you deploy afterward must be associated with a dedicated service name. If a specified service does not exist, a new one will be created, and if one exists the pushed application will take its place. Failing to specify a service in your app.yaml file will have your default service application overwritten. You can see the list of running services in your App Engine dashboard.

The environment type can be set in the app.yaml file via the service attribute :

runtime: nodejs14
env: standard
instance_class: F1
service: my-node-app
automatic_scaling:    
    max_instances: 2    
    min_instances: 0    
    min_idle_instances: 0    
    max_idle_instances: 1

Scaling Types

Instance scaling is a very important configuration to understand because it can significantly impact your costs and performance. As mentioned, F-type instances support automatic scaling and B-type instances can leverage either manual or basic scaling. You can find the matrix of the options here and the configuration flag details can be found here. Here is a summary.

Automatic Scaling — Instances are created and terminated based on requests and performance. Configuration can get a little complicated but I would recommend at the very least setting the min and max setting for total and idle instances using the max_instances, min_instances, min_idle_instances and max_idle_instances attributes. This should prevent incurring unexpected charges. There are other flags like target_cpu_utilization and max_pending_latency that the App Engine can use to optimize performance and trigger scaling changes. Here is an example running 0 to 2 instances and a restriction of at most 1 idle instance:

runtime: nodejs14
env: standard
instance_class: F1
service: my-node-app
automatic_scaling:    
    max_instances: 2    
    min_instances: 0    
    min_idle_instances: 0    
    max_idle_instances: 1

Basic Scaling —  Instances are spun up and down as the load on the application fluctuates. In the app.yaml file the maximum number of instances can be set using the max_instances attribute and the idle period using the idle_timeout attribute like so:

runtime: nodejs14
env: standard
instance_class: B1
service: my-node-app
basic_scaling:
  max_instances: 11
  idle_timeout: 10m

Manual Scaling — Spins up and maintains a set number of instances regardless of the load on the service. In the app.yaml file the number of instances can be set using the instances attribute like so:

runtime: nodejs14
env: standard
instance_class: B1
service: my-node-app
manual_scaling:
  instances: 5

Costs and Pricing

Google provides a set number of App Engine uptime hours in their free tier however, depending on your application is set up you may incur costs even if your application is up for a fraction of the allotted time. Understanding where these charges are coming from can sometimes be a challenge — the detailed documentation on costs can be found here but I will outline the key elements below.

Instance Uptime Charges

As of today, Google will charge you for instance uptime as follows:

You might end up getting charged and reading the below three caveats might save you countless hours of scratching your head, and aimlessly searching for answers online.

As the documentation outlines here, you will be billed for uptime plus an additional 15 minutes when an instance spins down.

If your instance receives four evenly spaced out requests (every 15 minutes) within an hour and only takes a second to process each before spinning down you will still be charged 60 minutes despite only using 4 seconds of uptime.

Your app might scale to multiple instances if usage spikes and scaling is enabled. In this scenario, you may see an hourly charge that is a multiple of the above cost table.

Application versions might also produce some unexpected charges. By default, the App Engine maintains all of your application’s version history and makes each version available mapped to a unique endpoint. If not used, these historic builds should only consume storage resources, but if requests are made against historic versions, they will spin up dedicated instances — this will of course come with additional utilization costs.

Quotas

The App Engine caps your application’s utilization of resources through quotas. There are three types of quotas; free, daily, and per-minute, if you are deploying hobby applications it is unlikely that you will come close to exceeding any of these quotas. If you do exceed a quota limit your application will be unavailable until the resources are reset at the end of the period, you can read more about this here.

Other Considerations

Google Cloud Storage

There are a few different ways for the App Engine to leverage GCS:

1. The staging.<project-id>.appspot.com bucket is used to stage some objects as applications are deployed
2. The us.artifacts.<project-id>.appspot.com bucket is used to store build artifacts
3. Some applications may use buckets for runtime object storage
4. Buckets can be used to store your source build files for App Engine deployments

GCS has a great free usage tier as outlined here.

Other Services

The App Engine might utilize a few other Google Cloud Platform services for general operation. The utilization of these services should be covered under the free tier but it’s important to understand what these services are in case you do start seeing charges. You can see the full list here.

Conclusion

Although the App Engine is a great way to get your application up and running on a highly scalable platform, understanding how to properly configure your deployments and costs can be a bit confusing. In this article, I shine some light on a few areas where beginners typically get bogged down. I hope my post was informative and saves you some headache and googling.

Good luck and happy coding!

Python is a popular language for all sorts of data processing today. Use cases range from web applications and machine learning applications all the way to hardware control on devices like the RaspberryPi. When it comes to these even systems and real-time data processing, leveraging Pub/Sub platforms can add modularity and scalability to your solutions — you can read more about this here.

Read about why I used Google Cloud Platform tools for my hobby projects here.

Objectives

In this article, I will walk through setting up a Python application to publish and consume data from Google’s Pub/Sub.

Time required: 15 minutes

Prerequisites

To follow along, you should have the following:

1. A basic understanding of how Python works
2. Python 3.x installed on your machine
3. A Google Cloud Platform account and a project

Let’s Do Some Coding!

GCP — Service Account Setup

First things first, let's get all the configuration done in GCP. A GCP Service Account and private key are needed to access the Pub/Sub service from a Python application.

The full list of your service accounts can be accessed here and a new service account can be added using this link. Give your account a name and id —  both can be the same but the id must be unique — I named mine python-tester.

Click create and add the Pub/Sub Publisher and Pub/Sub Subscriber roles to ensure that this account can both consume data from and publish data to your Pub/Sub topic(s).

From here you can click done.

Next, we need to generate a private key that our Python application will use when communicating with GCP. Find the service account you just created and select the Manage keys option.

Use the Add Key button to add a new JSON key.

Clicking Create should download the private key file to your default Downloads directory. If you open the file you should see something like this:

{  
  "type": "service_account",  
  "project_id": "...",  
  "private_key_id": "...",  
  "private_key": "-----BEGIN PRIVATE KEY-----...",  
  "client_email": "python-tester@...",  
  "client_id": "...",  
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",  
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "...",  
  "client_x509_cert_url": "..."
}

Make sure you keep track of this file as our Python application will need it.

GCP — Pub/Sub Topic Setup

Before we can push/pull data from Pub/Sub we need to create a topic. You can see all your active topics here. Create a new topic, give it a name and leave the default subscription option checked — I named my topic my-python-topic .

Make sure you leave the Add default subscription option checked and click Create Topic —  you should see the new topic appear in your topic list. Your default subscription will have the name of your topic with a -sub suffix, in my case it is named my-python-topic-sub .

Python — Writing the Producer and Consumer

Before writing code, you must have Python 3.x installed along with thegoogle-api-python-client and google-cloud-pubsub GCP libraries. You can install these with pip/pip3 using the following:

pip3 install --upgrade google-api-python-client
pip3 install --upgrade google-cloud-pubsub

Somewhere on your machine create a folder for your Python code.

mkdir pub-sub-test
cd pub-sub-test

Move your private key generated in the GCP — Service Account Setup section to this new folder. If you lost your key, you can generate a new one using the same instructions.

Create your main executable Python file in this directory — I am calling mine code.py and add the following content:

The GCP library expects an environment variable called GOOGLE_APPLICATION_CREDENTIALS to point to the private key. We set this value on line 2 with:

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="YYYY.json"

Make sure to replace YYYY.json with the path/name to your private key file.

PUB_SUB_PROJECT on Line 12 should be updated with the id of your GCP project, you can find the id on the Pub/Sub list page. It would be the value between projects and topics - projects/YYY/topics/my-python-topic.

Your project should now look like this:

├── pub-sub-test
│ ├── code.py
│ ├── YYYY.json

I tried my best to have the code be as self-explanatory as possible but essentially:

process_payload: A callback function that handles events consumed from Pub/Sub, any logic you want to apply to the payload should be added here.

push_payload: Takes a payload (JSON) and pushes it to the provided a Pub/Sub topic/project id combination.

consume_payload: Checks the provided subscription/project combination for new events and if data exists the callback function will be called for processing. The timeout period serves as an interrupt.

The rest of the code continuously pushes and consumes data until the program is terminated. You can run the code with python3 code.py and should see something like this in the terminal:

There you have it, a bare-bones implementation but this should be enough to get you jump-started with leveraging GCP’s Pub/Sub with Python.

Conclusion

Google’s Pub/Sub platform is great for handling large amounts of data and decoupling the various components of your architecture. In this article, I walked you through how Pub/Sub can be leveraged with Python applications. I hope that you were able to learn something from this post.

Good luck and happy coding!

Google Cloud’s App Engine allows you to deploy scalable web applications on a platform fully managed by Google. These applications can range from back-end services and API layers to front-end applications running on Angular and React frameworks. Google provides 28 daily hours of run time with this service for free so you can get away with some free hosting!

In this article, I will walk you through the deployment of a simple Node.js application on this App Engine platform.

Prerequisites

To follow along, you should have the following:

1. A basic understanding of how Node.js works
2. Node installed on your local machine
3. A Google Cloud Platform account and a project

Let’s Build and Deploy our Application!

Create the Project

Somewhere on your local machine initialize a new Node.js project.

mkdir helloworld
cd helloworld
npm init

Create a simple execution file index.js with the following content in the helloworld folder:

const express = require('express');
const app = express();

app.get('/', (req, res) => {
  res.send('GCP App Engine!');
});

const PORT = process.env.PORT || 8080;

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}...`);
});

Our application won’t do much and just returns a ‘GCP App Engine!’ string when called.

Add the following start script and express dependency to the created package.json file:

{
 ...
 "scripts": {
   "start": "node index.js"
 },
 "dependencies": {
   "express": "^4.16.3"
 }
}

Express is not required and I am using it to make deployment easier. The start script will be used by the App Engine to launch your application. Note that index.js matches the name of my main execution file created above.

Install dependencies (express) and make sure your app runs locally:

npm install
node index.js

Navigate to http://localhost:8080 and you should see:

The application is ready to go but we need to provide some information to the App Engine so that it knows how to deploy our code. We do this using a YAML file. The configuration in this file can get pretty complicated and there are a lot of options to configure but in our case, I will keep it simple. Create a file called app.yaml  in the helloworld folder and add the following content:

runtime: nodejs14
env: standard
instance_class: F1
automatic_scaling:
  min_idle_instances: automatic
  max_idle_instances: automatic
  min_pending_latency: automatic
  max_pending_latency: automatic

My Node.js runtime is version is 14.16.0, feel free to change the runtime if your version is different.  We will be using a standard environment and an F1 instance as these are covered by GCP's free quota.

At this point, you should have the below file structure and we are ready to start migrating our code to GCP.

├── helloworld
│ ├── index.js
│ ├── package.json
│ ├── app.yaml

Migrate your Code

Before deploying the application you will need to make the code available to GCP. You can do this by leveraging Cloud SDK on your local machine or by using the Cloud Shell as I will be doing in this walk-through.

I will be using a Cloud Storage bucket to stage my code before pushing it to the App Engine. Create a new bucket to house your source code (you can reuse an existing bucket if desired). For the region you should set Regional and the rest of the settings can be left default. In my case, I am creating a bucket called sample-code-repo and uploading the entire helloworld folder to the root.

Next, we need to get the code uploaded to the Cloud Shell VM, you can do this by opening the Cloud Shell terminal from any GCP console page or just click here.

To create the required folder structure on the Cloud Shell VM and sync the code from the bucket run the following commands, replacing sample-code-repo/helloworld with <source-code-bucket-name>/<app-folder-name> :

mkdir helloworld
cd helloworld
gsutil rsync -r gs://sample-code-repo/helloworld .

You will be asked to authorize the bucket access (pop-up) and once done running the ls command should confirm data replication:

If you have another preferred method to migrate code to Cloud Shell (e.g. git), feel free to use it. At this point, our project is ready for deployment.

Deploy the App

To deploy the application on the App Engine we need to head back to our Cloud Shell VM and run the following:

cd helloworld
gcloud app deploy

For your first App Engine deployment you will need to specify a region - I used us-central for mine.  The code will take a minute to compile and once done running gcloud app browse will output a link that you can use to access your now deployed application!

Conclusion

Google’s App Engine is a great platform for rapidly deploying applications online at no cost. In this article, I walked you through how this platform can be used for Node.js applications but the same result can be accomplished with Java, Python, PHP, and Go. I hope that you were able to learn something from this post.

Good luck and happy coding!

Ghost is a popular open-source blogging platform of which I am a huge advocate. Here is what a blog hosted on Ghost looks like. The platform is secure, lightweight, and very easy to use and customize. The Ghost team has a post on how their platform compares to WordPress here.

There is a managed hosting fee if you want Ghost to host the blog for you but in this article, I will walk you through setting up a Dockerized ghost blog on Google Cloud Platform (GCP). With this approach, your blog can be hosted absolutely free and the setup should take under 15 minutes. You can read up on Ghost here and why I use GCP as my cloud platform here.

Having a GCP account is a prerequisite and you can create it here for free. If you are using AWS, Azure, or another cloud vendor you can follow the same steps for everything except the compute instance setup.

Let’s get started!

Setup the Compute Engine Instance

Google’s Compute Engine is a service for deploying private Virtual Machines in the cloud. The GCP always free tier provides one of these VMs free for you to use and we will use this service to host our Ghost blog.

If you don’t yet have a project in GCP or if you want to create a new project to be associated with your Ghost blog you can do that here.

The Compute Engine creation process can be started here or by clicking on the create instance button on the VM instances view.

1. Specify a name for your instance (I will be using ghost-blog)
2. Under Machine configuration leaving the machine family as General Purpose, select N1 for series and f1-micro for Machine Type (free tier)
3. Change the Boot disk to the public image Ubuntu 20.10 or Ubuntu 20.10 Minimal with boot disk type set to Standard persistent disk
4. Ensure Allow HTTP traffic and Allow HTTPS traffic checkboxes are checked
5. Note the message in the top right indicating that the first 744 hours of f1-micro instance usage are free this month — if you don’t see this message you might have selected the wrong machine type or the wrong region (must be us-west1, us-central1, us-east1)

Here is what your configuration should look like:

Click Create and wait for the instance to spin up. You should be automatically redirected to your VM instance list and once you see a green checkmark appears next to your instance it is ready to go.

Note the external IP as we will need to further in this tutorial when we configure Ghost.

Click the SSH button under Connect to launch a session and get access to your new Compute Engine instance.

Get Your Instance Ready for Ghost

As mentioned above, I will be using Ubuntu 20.10 in this walk-through. If you are using a different OS, you will have to adjust your commands.

Update the Package List & Install Your Favourite Text Editor

I prefer nano but you can use whatever suits you. Update the package list and install your editor.

sudo apt-get update
sudo apt-get install nano

Add Swap Space

Small compute instances like the f1-micro come with limited memory (600MB in our case), which can slow things down and overload the CPU. If you are using a larger compute instance with sufficient memory you can skip this step, but I will start by adding swap storage to enhance performance. You want to add 2x of your available RAM in swap space — in my case, this is 1.2GB of swap for 600MB of RAM. Create your swap file:

sudo fallocate -l 1.2G /swapfile

Change permissions to only allow root access:

sudo chmod 600 /swapfile

Set up a Linux swap area on the file:

sudo mkswap /swapfile

Make sure the changes are permanent by modifying the filesystem table:

sudo nano /etc/fstab

Append /swapfile swap swap defaults 0 0 to the end of the file. It should look like this:

LABEL=cloudimg-rootfs / ext4 defaults 0 1
LABEL=UEFI /boot/efi vfat defaults 0 1
/swapfile swap swap defaults 0 0

*If you have never used nano before, you can use ctrl+x to close out of the file - enter y+enter when asked if you would like to save modified buffer.

Your instance now has an additional 1.2GB of swap memory.

Install Docker

We will be using the Ghost Docker image to deploy our blog. Docker has an install walkthrough that you can follow here but I will include the relevant instructions in this section.

Update your package list and run an upgrade to make sure everything is up to date:

sudo apt-get update
sudo apt-get upgrade

Install required packages:

sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release \
    software-properties-common

Add Docker’s official GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Setup the stable repository:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu groovy stable"

Run the update again and install docker:

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

You can check to ensure Docker is installed with the sudo docker --version command and see something like this in the output Docker version 20.10.5, build 55c4c88 .

You are now ready to pull and configure the Ghost image!

Configure and Deploy Ghost

There are a few different ways of deploying Ghost but as mentioned we will use Docker. There is a Ghost Docker image maintained on the Docker hub which makes things very simple for us.

Pull the latest Ghost docker image:

sudo docker pull ghost:latest

Create a directory to house your content and Docker config file and create said config file (make sure to change your IP/domain):

mkdir ghost_blog

echo '{
  "url": "http://yourdomain-or-ip",
  "server": {
    "port": 2368,
    "host": "0.0.0.0"
  },
  "database": {
    "client": "sqlite3",
    "connection": {
      "filename": "/var/lib/ghost/content/data/ghost.db"
    }
  },
  "mail": {
    "transport": "Direct"
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "systemd",
  "paths": {
    "contentPath": "/var/lib/ghost/content"
  }
}' >> ghost_blog/config.json

*Replace http://yourdomain-or-ip with the external IP we noted in the Setup the Compute Engine Instance section. If you have a domain pointing to the IP you may use the domain instead.

Create a shell script file for quickly spinning up your Docker container (make sure to change your IP/domain):

echo '# Set path variables
DATA_DIR="$PWD/ghost_blog"
CONTAINER_NAME="ghost_blog"

# Purge the existing container if running.
docker kill $CONTAINER_NAME
docker rm $CONTAINER_NAME

# Mount the volumes - content directory and config file
# and add the url variable for docker map as the public URL
docker run \
-d \
--restart=always \
-p 80:2368 \
-e url="http://yourdomain-or-ip" \
-v $DATA_DIR/content:/var/lib/ghost/content \
-v $DATA_DIR/config.json:/var/lib/ghost/config.production.json \
--name $CONTAINER_NAME \
ghost' >> run.sh

*Replace http://yourdomain-or-ip with the external IP we noted in the Setup the Compute Engine Instance section. If you have a domain pointing to the IP you may use the domain instead.

You should not have the following folder structure:

├── run.sh
├── ghost_blog
│ ├── config.json

To spin up your Docker container containing your Ghost image run:

sudo sh ./run.sh

You will see a cannot kill container error the first time you run the script. This is expected and will not cause any issues. Here is a sample output:

Navigate to your external IP and you should now see your blog instance (make sure you are using HTTP and not HTTPS).

To finish configuring your blog you will need to navigate to http://yourdomain-or-ip/ghost and follow the instructions to create your admin account.

Congratulations, you now have your very own Ghost blog page up and running!

Other Helpful Topics

Updating Ghost

With this setup, you can update your Ghost image by simply running sudo docker pull ghost:latest

Setting Up HTTPS for Your Blog

If you would like to take your blog to the next level you need to set up a proper domain name and SSL certificate(s). I have an article on doing this with NGINX and Certbot.

Persist Content Storage

With our setup, all of your content, configuration, and posts will live within the ghost_blog folder. To create a proper backup for your blog I recommend using git. You can initialize the ghost_blog folder as a git repository which allows for easy backups and restores.

IP and Domain Management

If your entry point to the blog changes to a new IP or a proper domain name you will need to update both the ghost_blog/config.json and run.sh files with the new IP/domain name. Once dun simply re-run the run.sh batch script.

Good luck and happy coding!

I enjoy using the Google Cloud Platform (GCP) for hobby projects (check out why I use GCP here) and Google’s Cloud Storage (GCS) product has made its way into my design several times. However, I quickly realized that other GCP services leverage GCS, creating buckets and filling them with objects.

At first, my use of GCS was light, and these system buckets didn’t bother me, but then I started seeing charges on my bill (albeit just a few cents), and decided it was time to understand what these buckets were for and how I could remove or reduce the charges.

Understanding these charges can be a challenge and what creates the system buckets is loosely documented. I have documented my experience here hoping that it will help others avoid similar frustrations.

Below includes my investigation processes, discoveries, and the steps I took to minimize my GCS charges. If you are just interested in the solution you can skip to The Solution section.

WARNING: Do not alter any buckets auto-generated by GCP (or their contents) without understanding their purpose. Some are tied to active processes and altering them can cause irreversible object corruption!

The Investigation

All this started when I began incurring GCS costs, and if you are here investigating your charges you know that there isn’t much detail in the billing dashboard. Here is what I see for February of 2021.

There is a charge of 5 cents for 1.45 GB month of storage under the US Multi-region SKU. I see two issues here:

1. The first 5GB of storage should be free (thank you free tier!)
2. My usage of Cloud Storage should not come close to 1.45 GB

The Google Cloud Storage pricing page addresses my first issue. The free tier only applies to certain regions.

Cloud Storage Always Free quotas apply to usage in US-WEST1, US-CENTRAL1, and US-EAST1 regions. Usage is aggregated across these 3 regions. Always Free is subject to change. Please see our FAQ for eligibility requirements and other restrictions.

Maybe I selected the wrong storage type for my buckets?

Cloud Storage Browser

Bucket details can be found on the Cloud Storage browser page:

In this case, I have 6 buckets in total. Highlighted in green is an active bucket used to house some IoT device data. I did not directly create the others and some have the location type of Multi-region.

That partially explains the charges, but it’s not clear what process created these buckets and what GCP uses them for. On that note…

WARNING: Do not alter any buckets auto-generated by GCP (or their contents) without understanding their purpose. Some are tied to active processes and altering them can cause irreversible object corruption!*

*Repeated intentionally due to its importance!

The next step is to analyze bucket space utilization and understand where the costs are originating.

The Monitoring Page

Although the GCS browser does not show total space utilization by bucket there are a few different ways of getting this information. I prefer the GCP monitoring page. Here are Google’s setup instructions when using the monitoring page for the first time:

If you have never used Cloud Monitoring, then on your first access of Monitoring in the Google Cloud Console, a Workspace is automatically created and your project is associated with that Workspace. Otherwise, if your project isn’t associated with a Workspace, then a dialog appears and you can either create a Workspace or add your project to an existing Workspace. We recommend that you create a Workspace. After you make your selection, click Add.

Once loaded, the easiest way to get an overview of GCS usage is to select it under the resource dashboard.

Expanding the legend of the Object Size graph on the resource dashboard provides a list of all buckets along with their current space utilization.

In this case, the us.artifacts bucket is responsible for 99.7% of my total storage. The main cost driver has been identified!

The Solution

Of the 5 auto-generated buckets in my Google Cloud Storage, 4 are multi-region and are incurring costs. I will outline what GCP processes are using each bucket for and how to minimize or eliminate the costs.

The Cloud Run Buckets

The <project-id>_cloudbuild and artifacts.<project-id>.appspot.com  buckets are utilized by the Google Cloud Run engine. When code is submitted to Cloud Run, the engine uses the cloudbuild bucket to stage build objects and the artifacts bucket as the artifact registry. It’s not critical for you to know exactly what these objects are, but you should know that they are typically not critical after deployments and there is no reason for them to be in a Multi-region bucket.

The good news is that GCP allows to overload defaults with the gcloud builds submit command. Here are the steps to ensure you incur no more GCS costs from your Clour Run deployments:

1. Create a new bucket with your desired regional storage (eg. gcr_store)
2. Create a default folder for the build objects in this bucket (eg. source)
3. Create a default folder for the artifact objects in this bucket (eg. artifacts)
4. Create a cloudbuild.yaml file in your deployment directory with something like the following (note the location mapping to the new artifacts folder and the gcr.io/cloud-builders/docker indicating what builder to use)

steps:
- name: 'gcr.io/cloud-builders/docker'  
artifacts:
  objects:
    location: 'gs://gcr_store/artifacts'
    paths: ['*']

5. Use the  --gcs-source-staging-dir flag to specify where build objects should be saved when building new Cloud Run applications and include your config yaml file

gcloud builds submit --gcs-source-staging-dir=gs://gcr_store/source --config cloudbuild.yaml

6. Delete your auto-generated <project-id>_cloudbuild and artifacts.<project-id>.appspot.com buckets

5. (Optional) Add a lifecycle rule on your new bucket to delete objects older than X days (eg. 7 days)

Once done you should no longer have a Multi-region bucket associated with your Cloud Run deployment process and if you ever find the size of your custom bucket is getting out of hand you can implement Step 6.

The Cloud Functions Bucket

The gcf-sources-<id>-<region>   bucket is used for the storage of Google Cloud Function (GCF) objects and metadata. This folder is deployed in the same region as your functions and should never get very large (mine is 11 kB for 5 functions). I don’t recommend touching the contents of this bucket as it could permanently corrupt your GCF objects.

Some Cloud Functions will also use Cloud Build which dumps artifacts into the us.artifacts.<project-id>.appspot.com bucket. See the The us.artifacts Bucket section below and what can be done to address these objects.

The App Engine Buckets

The staging.<project-id>.apopspot.com bucket is used by the Google App Engine for temporary storage during deployments.

App Engine also creates a bucket that it uses for temporary storage when it deploys new versions of your app. This bucket, named staging.project-id.appspot.com, is for use by App Engine only. Apps can't interact with this bucket.

You can’t get rid of this bucket but you can reduce the number of stored objects by specifying a different bucket at build time with the —bucket flag. Here are the steps to ensure you incur minimal costs from this bucket:

1. Create a new bucket with your desired regional storage (eg. gae_storage) — if desired you can use a different bucket for each app
2. Use the  —bucket flag to specify where build objects should be saved when deploying your app

gcloud app deploy --bucket=gs://gae_storage

3. Delete everything in the staging.<project-id>.apopspot.com directory except for the ae/ folder

Once done the Multi-region staging.<project-id>.apopspot.com bucket will be minimally leveraged and your custom buckets will contain 99% of the objects for each app deployed.

App Engine deployments also leverage us.artifacts.<project-id>.appspot.com bucket. See the The us.artifacts Bucket section below and what can be done to address these objects.

The us.artifacts Bucket

The  us.artifacts.<project-id>.appspot.com bucket is used to store container images generated by the Cloud Build service. The only processes I have observed to generate objects in this bucket are Cloud Functions and App Engine builds. Objects generated by these processes are safe to remove post-deployment as described here.

Once deployment is complete, App Engine no longer needs the container images. Note that they are not automatically deleted, so to avoid reaching your storage quota, you can safely delete any images you don’t need.

The same should apply for Cloud Function artifacts as well.

Although I do not use Firebase to deploy functions I have come across several open tickets online indicating that the approach below might cause issues for you. I might do another article exploring the Firebase issue and possible resolutions.

Do not delete this bucket outright and do not follow the below instruction if you use Firebase to deploy functions!

We cannot remove the bucket altogether but we can follow these steps to minimize space usage.

1. Navigate to the LIFECYCLE tab of the us.artifacts.<project-id>.appspot.com bucket
2. Add a new lifecycle rule deleting objects that have an age greater than X days (I use 7 for mine)
3. Delete all objects in this bucket

Once done you should see your space consumption for this bucket drop significantly. In my case, I was able to free up 85% of the utilized space to less than 300MB.

Conclusion

GCP is a great platform but when it comes to automatic storage of metadata objects and build container images things can get complicated and messy. Through this investigation, I got a chance to learn more about how Cloud Run, App Engine application, and Cloud Functions are managed. I hope that you were able to learn something from this post as well and if not that at least I was able to help you tidy up your GCS environment.

Good luck and happy coding!

Header photo by Pedro da Silva on Unsplash

The Pub/Sub pattern is not something new but with the growing complexity of event systems combined with advances in distributed computing Pub/Sub is growing in popularity. In this article, I will explain the high-level Pub/Sub pattern and try to give you some reasons to include in your projects of varying sizes.

Without Pub/Sub

When building complex event-driven systems that require two or more different components to communicate with each other the traditional and simplest approach is to wire those components directly together. In some cases, this is done using web service APIs, flat-file exchange, or through shared data stores like databases. These approaches work but come with a set of challenges:

1. The modules become coupled together and a change to one component might require updates to any other components that it interacts with
2. New integrations become time-consuming to build and test
3. There is a general lack of scalability as the volume of events and number of integration points grows

Event for smaller hobby projects that involve just 2 or 3 components and very little data I find that this typical direct integration approach is extremely detrimental and discourages me from making iterative enhancements. I find it daunting that to add a new module I need to revisit and reconfigure several components.

The Pub/Sub Pattern

Enter the Pub/Sub pattern. Instead of wiring parts of your infrastructure directly together the communication is done through a set of channels. A module in your infrastructure can either be a publisher to a channel to send events or a subscriber to read events. This presents several benefits and helps address some of the disadvantages outlined above.

1. The modules become completely decoupled from each other and their job when it comes to integration is to simply properly format and publish data, or receive payloads as subscribers and know how to process them
2. Development of new components is simplified as developers don’t need to worry about various integration points and the network layer
3. Testing can be isolated and simplified by subscribing unit test scripts to topics or publishing simulated inputs
4. If the correct Pub/Sub solution is used your system should become infinitely scalable without touching any other components of your infrastructure

Conclusion

The Pub/Sub pattern is becoming the norm for building scalable and maintainable solutions. If you are not using this pattern as part of your infrastructure today I highly recommend at least evaluating it as an option as the benefits of reduced maintenance and future development costs would likely outweigh the effort needed to migrate your solution to this scalable pattern. As I mentioned about I highly recommend this pattern for even smaller hobby projects to improve the solution's future flexibility and to get experience with the Pub/Sub pattern. I use Google's Pub/Sub for most of my personal projects as it comes with 10GB of free data transfer per month which more than covers all my needs.

While designing and implementing solutions, I am often faced with the need to set up recurring batch jobs around data storage and processing. Recently I have been trying to keep my infrastructure as serverless as possible so in this article, I will show you how Google Cloud Platform can be leveraged to run almost any batch job your project might need for free.

Use Cases

For me, this batch pattern is the most useful when it comes to data processing, reconciliation, and cleanup. Here is an example involving data aggregation…

A bucket can be an effective repository for streaming data but if your payloads are small in size and frequent — having a file for every payload can get expensive if you have to do frequent reads. I solve this problem by running a batch job to merge individual payloads into hourly or daily files, allowing for much more cost effective solution.

Or how about database cleanup…

If you have a SQL database containing large timeseries data sets, regular purging is critical for performance. You can squeeze a recurring job into a web application or the ETL system that is loading data into your tables however I solve for this using this serverless batch approach to decouple the solution and simplify maintenance.

Architecture

We will be using 3 GCP services to implement our serverless batch solution. The Cloud Scheduler will trigger our batch events, Pub/Sub will be used to transmit the events to a Cloud Function that will perform the required batch operation.

Pricing

GCP offers a very generous free tier, I have made a simplified cost table below for the three services we will need to schedule and run a batch job. A batch job running every 5 minutes will use up 1 cloud scheduler job, ~9,000 Cloud Function executions, and ~9MB of Pub/Sub throughput.

If you need more than 3 jobs across your projects, you will be charged 10 cents USD per month for every additional job.

Configuration

Pub/Sub

First, let’s configure a Pub/Sub topic as it will be required in our set up of both the scheduler and serverless function. Topics can be configured here in your GCP console.

• As you can see all we have to configure is the topic name -  I am using example-topic

Cloud Scheduler

Next, we configure the Cloud Scheduler as our batch trigger. Head here and create your scheduler job.

• In my example, I am configuring the scheduler to run every 30 minutes but you can set any period desired
• Specify the topic we created in the previous step — I am using example-topic
• Our cloud function will not need anything except the trigger from the scheduler so the payload value is not important so you can put any value —  I am using run

Cloud Function

We are almost done, the last step is to create a Cloud Function that will be triggered when an event is triggered by the scheduler. You can find Cloud Function configuration here.

• Select Cloud Pub/Sub as the trigger type
• Select the topic created in the first step
• Proceed to the code configuration — I will be using the out of the box Node.js function. The function simply logs the contents of the Pub/Sub payload.

The function might take a minute to fully deploy.

Keep in mind that you can use any of the available programming languages in this step.

Testing

To test our configuration we need to head over the Cloud Scheduler list and manually trigger our scheduler using the RUN NOW option.

To make sure our function successfully triggered we can head over to our Cloud Function list, select the function you configured earlier and check out the Logs tab.

You should see log output indicating that your function ran and the payload messaged you configured for the Cloud Scheduler should also be displayed.

Success!

Conclusion

There you have it, in 5 minutes we configured a 100% free solution that you can use to run various types of batch jobs. If you ever find yourself in need of quickly setting up a highly decoupled solution for kicking off or running batch jobs you now have a quick and easy way to get it done using GCP.

Good luck and happy coding!

This post will go over the steps needed to set up an HTTPS certificate for you web application using certbot and an NGINX proxy.

The web application in my case is ghost (node.js content management system) but the below should work for any web application as long as you have the knowledge and capability to expose your web application on ports other then 80 and 443.

Step 1 - set up certbot on your server and generate a certificate

If your web application does not have an HTTPS certificate most browsers will block users and display something like the following message.

This happens because your web application does not have a valid and registered HTTPS certificate. Luckily these can be generated quite easily using free utilities, one of these is certbot. Certbot is a tool you can download on your server and in a few seconds register your domain and generate a valid certificate.

On the certbot site you can choose a type of web server and operating system you are running and the site will give you a set of step by step instructions for generating a certificate. In this example I will use Ubuntu 18.04 as it is the most popular Linux OS. For the server I will use none of the above as this is the simplest and most dynamic form of certificate generation. Here were my instructions as presented by certbot.

I installed certbot as instructed using the below.

sudo apt-get update
sudo apt-get install software-properties-common
sudo add-apt-repository universe
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot

Instead of playing with the webroot of my web application I recommend stopping your web server and running the standalone option. Before proceeding ensure your web application is stopped. You can validate that you have successfully done this using: sudo netstat -ltnp | grep -w '80'. If this returns anything your application is running and as a last resort you can kill it with the kill pid command.

When generating the certificate you will be asked for the domain and subdomains names you are registering. Even if you don't have any subdomains you should still register the www subdomain.  In my case I will be registering theappliedarchitect.com and www.theappliedarchitect.com. You should see the following if the certification was successful.

$ sudo certbot certonly --standalone
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel): theappliedarchitect.com www.theappliedarchitect.com
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for theappliedarchitect.com
http-01 challenge for www.theappliedarchitect.com
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/theappliedarchitect.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/theappliedarchitect.com/privkey.pem
   Your cert will expire on 2020-05-30. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:
   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

If you see the below port 80 is being used by another application, follow the instruction above to identify the culprit and terminate it.

$ sudo certbot certonly --standalone
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel): theappliedarchitect.com www.theappliedarchitect.com
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for theappliedarchitect.com
http-01 challenge for www.theappliedarchitect.com
Cleaning up challenges
Problem binding to port 80: Could not bind to IPv4 or IPv6.

At this point the certificates should be on your server with the location indicated by the certbot output. In my case they are in the /etc/letsencrypt/live/theappliedarchitect.com/ folder.

The next step will cover setting up a proxy to listen on port 443 and 80, provide the correct certificate and direct traffic to the correct web application. You don't need to continue if you are happy setting up generated certificate directly on your web app.

Step 2 - set up NGINX

If you like me have several web applications running on the same server or if you want to standardize your certificate setup process regardless of web application type. Start by installing NGINX (if its not already included in your distribution) sudo apt install nginx

You can verify the install by using a browser to navigate to the domain you just registered and you should see something like the below.

NGINX routing is managed using configuration files. These are usually managed in the /etc/nginx/sites-available directory and deployed as links to the /etc/nginx/sites-enabled folder. You will noticed that there is a default configuration file present by default, we will start by removing the link from the site-enabled folder.

sudo rm /etc/nginx/sites-enabled/default

Next we need to create a new configuration file to direct both HTTP and HTTPS traffic to our web application. I will call my config file blog but you can use any name that is relevant for your purpose. Create the file and link it to the enabled folder.

sudo touch /etc/nginx/sites-available/blog
sudo ln -s /etc/nginx/sites-available/blog /etc/nginx/sites-enabled/blog

These configuration files can get very complicated but we will keep ours bare bones. The file will have two server mappings, one for port 80 (HTTP) and one for port 443 (HTTPS) where we will link our newly generated certificate. Make sure you replace all instances of theappliedarchitect.com with your domain name. Note the SSL certificate file paths, if you moved yours from the defautl folders you might need to update these as well. My blog runs on port 2368 make sure you update the port to reflect the internal port your web application will be running on.

server {
    listen 0.0.0.0:80;
    server_name theappliedarchitect.com;
    access_log /var/log/nginx/theappliedarchitect.com.log;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header HOST $http_host;
        proxy_set_header X-NginX-Proxy true;

        proxy_pass http://127.0.0.1:2368;
        proxy_redirect off;
    }
}



server {
        server_name theappliedarchitect.com;
        listen 443 ssl;

        location / {
                proxy_pass      http://127.0.0.1:2368;
                proxy_set_header    X-Real-IP $remote_addr;
                proxy_set_header    Host      $http_host;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        }

        ssl_certificate     /etc/letsencrypt/live/theappliedarchitect.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/theappliedarchitect.com/privkey.pem;
        ssl on;

}

All you have to do now is restart the NGINX service and it should both provide the certificate and direct traffic to your underlying application.

sudo service nginx restart

One last thing to mention is that the certificates provided by certbot are only valid for 3 months and must be renewed before expiry (the drawback of using a free service). Certbot will send you reminder emails to renew your certificate when the time comes. In a future post I will outline how this renewal can be automated!

In this post I will be going through the process of setting up an Angular front end to connect and utilize some of the TensorFlow models that were set up in previous posts. The model set up and training walk through can be found here and the docker serving walk through here. This post is part of the TensorFlow + Docker MNIST Classifier series.

If you are not familiar with Angular I highly recommend at least going through the official getting started tutorial before implementing any of the code below. Or you can use your own front end instead of Angular.

This is not an Angular tutorial and I will not be going through the code in detail. We will be cloning a project from my git repository and going through some of the key components that are specific to this project. To keep things organized we will be running our Angular application within a docker container.

For reference, here is the final result we are targeting.

The setup

If you are using Docker on windows you might need to share your Drives. This can be done by navigating to Docker settings > Shared Drives and making sure the drives you are working with are checked

We can start by cloning my Angular repository:

$ git clone https://github.com/adidinchuk/angular-mnist-project

The app folder structure should look like this:

├── src
│ ├── app
│ │ ├── components
│ │ │ ├── digits
│ │ │ │ ├── canvas
│ │ │ │ ├── digit-control
│ │ │ │ ├── prediction
│ │ │ │ │.......
│ │ ├── services
│ │ │ ├── digits
│ │ │ │ ├── api
│ │ │.......
│ ├── assets
│ │ ├── css
│ │ ├── js
│ ├── environments
│ │ │.......
├── proxy.config.json
│.......

Some key objects and their responsibilities:

src/app/components/digits/canvas
Canvas object used for several things - primarely for getting user digit input, image scaling and displaying autoencoder results
src/app/components/digits/digit-control
Main component responsible for collecting user input (both noise option and digit)
src/app/components/digits/prediction
Very basic component responsible for displaying the classifier results

src/app/services/digits/api
Services used for communication with the TF serving instance

Let's take a closer look at how we access our TF serving endpoints. Here is what our main service methods look like:

//Autoencoder endpoint to clean data before classification processing
  runAutoencoder(data): Observable<any> {
    return this.http.post(DigitsConfig.API_ENDPOINT_PROXY + DigitsConfig.AUTOENCODER_MODEL + ':' + DigitsConfig.TF_METHOD_NAME, {
      "instances": [data]
    })
  }

  //classification endpoint to classify a 784 vector into a 0-9 digit
  runClassification(data): Observable<any> {
    return this.http.post(DigitsConfig.API_ENDPOINT_PROXY + DigitsConfig.CLASSIFICATION_MODEL + ':' + DigitsConfig.TF_METHOD_NAME, {
      "instances": [data]
    })
  }

And the configuration attributes:

public static API_ENDPOINT_PROXY = '/v1/models/';

public static AUTOENCODER_MODEL = 'autoencoder';
public static CLASSIFICATION_MODEL = 'classifier';

public static TF_METHOD_NAME = 'predict';
public static TF_INPUT_PARAM_NAME = 'instances';

Note that we are using a proxy for the mapping to ensure that any networking changes can be made outside the source code. For this we have a proxy.config.json file in the root directory with the following content:

{
    "/v1/*": {
        "target": "http://SERVING:8501",
        "secure": false,
        "logLevel": "debug",
        "changeOrigin": true
    }
}

Bellow I will be covering exactly what SERVING means and how we force the Angular application to adhere to this proxy. But in the mean time from our components we can leverage these endpoints using something like this:

this.api.runClassification(data).subscribe(
  class_res => {
    this.prediction.digit = this.extractPrediction(class_res.predictions[0]);
  },
  class_err => {
    console.log("Error occured during the classification call.");
})

Fairly straightforward, in fact due to my lacking front end knowledge the majority of my time went towards figuring out how to use canvases correctly to collect user input and display the final result. The TF serving integration was the easy part.  I am certain there are many ways my components could be improved, feel free to clone my repository and go nuts.

Running the docker container

Before building and launching the docker instance let's take a quick look at the Dockerfile, you will notice that the application on start sets up the proxy.config.json file using that we took a look at above using --proxy-config proxy.config.json.

# base image
FROM node:8.15.0

# install chrome for protractor tests
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
RUN apt-get update && apt-get install -yq google-chrome-stable

# set working directory
WORKDIR /app

# add `/app/node_modules/.bin` to $PATH
ENV PATH /app/node_modules/.bin:$PATH

# install and cache app dependencies
COPY package.json /app/package.json
RUN npm install

# add app
COPY . /app

# start app
CMD ng serve --host 0.0.0.0 --proxy-config proxy.config.json

Before we run the container we need to build the image and make sure all dependencies are retrieved. We can do this using:

$ docker build -t angular/mnist .\angular-mnist-project

Depending on your network's performance this could take a few minutes as all required dependencies will be retrieved. Notice that we tagged the image angular/mnist, you can also see this in the output:

Now that we have our image we can run the docker container using:

$ docker run --rm -p 4201:4200 --name angular angular/mnist

Breaking the command down:

--rm
make sure the container is automatically cleaned up on exit
-p 4201:4200
map the docker internal port 4200 to the external port 4201
--name angular
serving give a name to our container for easier identification and termination
angular/mnist
assosiate the image we built above

After running the command if you open http://localhost:4201 in your browser you should see the running application.

However if you try submitting the form you will see that nothing happens and taking a look at the console logs you can see that the TF serving endpoint is timing out.

This is happening because the angular docker container and the TF serving docker container are isolated. While from from your machine you can see both localhost:4201 and localhost:8501, the images are not able to resolve these paths. The solution is to use docker network functionality. We do this by first creating a docker network and then updating our docker start commands to first associate the containers with the network and then associate unique aliases to them. This aliases will allow containers to easily reference each other.  

Create a new docker network

$ docker network create MNIST

Remember to kill and remove any containers that are still active before running

$ docker kill angular
$ docker rm angular
$ docker kill serving
$ docker rm serving

Update and run the Angular docker run scripts

$ docker run --net MNIST --net-alias=ANGULAR --rm -p 4201:4200 --name angular angular/mnist

Update and run the TF Serving docker run scripts (original)

$ set TF_MODEL_DIR=%cd%/serving/mnist
$ docker run --net MNIST --net-alias=SERVING --rm -p 8501:8501 --name serving --mount type=bind,source=%TF_MODEL_DIR%/classifier,target=/models/classifier --mount type=bind,source=%TF_MODEL_DIR%/autoencoder,target=/models/autoencoder --mount type=bind,source=%TF_MODEL_DIR%/config,target=/config tensorflow/serving --model_config_file=/config/models.config

Associating the alias SERVING with --net-alias=SERVING to the TF Serving container allows the Angular application to access the endpoints at SERVING:8501 as both are running on the MNIST network.

With this done the http://localhost:4201 application should now work as expected:

If you check the terminal where the Angular docker container is running (or docker angular logs) you can see that requests are being correctly forwarded to http://SERVING:8501:

That's it! Hope this was helpful to somebody out there, I know I learned a low implementing and documenting this project.

Here is a summary of the components involved in this project:

This post will be covering the process of setting up TensorFlow serving and exposing the two models that were build and trained in the previous post. TensorFlow serving is a system for managing machine learning models and exposing them to consumers via a standardized API. This post is part of the TensorFlow + Docker MNIST Classifier series.

If you are not familiar with docker I highly recommend going through the official getting started tutorial before implementing any of the code below.

For all of my API testing I will be using the postman application. You can use your own testing tool or download postman here.

One of the most practical ways of setting up TensorFlow is via Google's pre-built docker container and this is the approach that will be taken in this post.

Set up the basic docker image

The first step is to ensure we have a docker serving image working correctly on our machine using one of the out of the box testing models. Make sure you have docker installed before running the below scripts in your command line.

If you are using Docker on windows you might need to share your Drives. This can be done by navigating to Docker settings > Shared Drives and making sure the drives you are working with are checked

download the docker image and clone the repository
$ docker pull tensorflow/serving
$ git clone https://github.com/tensorflow/serving

Linux - launching the container

#map the path to the test models 
$ TESTDATA="$(pwd)/serving/tensorflow_serving/servables/tensorflow/testdata" 
$ docker run --rm -p 8501:8501 \ --name serving \  --mount type=bind,source=$TESTDATA/saved_model_half_plus_two_cpu,target=/models/half_plus_two \ -e MODEL_NAME=half_plus_two

Windows - launching the container

#map the path to the test models 
$ set TESTDATA=%cd%/serving/tensorflow_serving/servables/tensorflow/testdata 
$ docker run --rm -p 8501:8501 --name serving --mount type=bind,source=%TESTDATA%/saved_model_half_plus_two_cpu,target=/models/half_plus_two -e MODEL_NAME=half_plus_two

The serving application should not be running in the docker instance and exposed to your network on port 8501. You should see something like this if your container has launched successfully:

Now lets check if the model is up and running and what it looks like. I will be using postman to make some requests and analyze the responses. Let's send a sample request and see if the model works. In this case the endpoint is localhost:8501/v1/models/<model name>:predict, where the <model name> is half_plus_two. In order for the model to serve a prediction it requires for the feature matrix to be provided in the body of the POST request. Here is the payload format in this case: {"instances": <features>} where <features> is an array containing all the sample you are looking to classify. The model we are working with has input and output dimensions of 1 so we will use {"instances": [1.0, 2.0]} as our payload and we expect a result of 2.5 and 3.0 (x / 2 + 2).

Submitting the request using postman we can see that the model is exposed and working exactly as expected!

Before proceeding we want to make sure clean up docker container as we will be redeploying our own models in a few minutes. Run the following:

$ docker kill serving

Deploy a custom model

Next we need to serve a model that does something more useful then the out of the box half_plus_two model. Before proceeding make sure you have a TensorFlow .pb models ready to go. In my previous post I set up an auto encoder and a classifier for processing MNIST images, you can take a look at the post or grab the source code here.

My current working directory currently contains both my serving repository and my TensorFlow project folder and looks like this:

├── tf-mnist-project
│ ├── src
│ │.......
├── serving
│ ├── tensorflow_serving
│ │.......

You might have to adjust my commands to reflect your own folder structure.

Let's create a new serving folder in the serving repository to house our models and a model subdirectory for our first model.

$ mkdir serving\mnist
$ mkdir serving\mnist\autoencoder

You should now see the 2 newly created folder. Now let's copy our first model into the model subdirectory.

$ cp tf-mnist-project\models\autoencoder\production\1 serving\mnist\autoencoder -r

Our folder structure should now look something like this:

├── tf-mnist-project
│ ├── src
│ │.......
├── serving
│ ├── tensorflow_serving
│ ├── mnist
│ │ ├── autoencoder
│ │ │ ├── 1
│ │ │ │ ├── assets
│ │ │ │ ├── variables
│ │ │ │ └── saved_model.pb
│ │.......

Let's try mounting the new model and launching the docker container.

Linux - launching the container

#update the path to our new model 
$ TESTDATA="$(pwd)/serving/mnist" 
$ docker run --rm -p 8501:8501 \ --name serving \  --mount type=bind,source=$TESTDATA/autoencoder,target=/models/autoencoder \ -e MODEL_NAME=autoencoder

Windows - launching the container

#map the path to the new model 
$ set TESTDATA=%cd%/serving/mnist 
$ docker run --rm -p 8501:8501 --name serving --mount type=bind,source="%TESTDATA%"/autoencoder,target=/models/autoencoder -e MODEL_NAME=autoencoder

Lets pause here and make sure we understand what we are asking docker to do with our command, because we are about to run into a problem trying to expose both of our models simultaneously.

--rm
make sure the container is automatically cleaned up on exit
-p 8501:8501
map the docker internal port 8501 to the external port 8501
--name
serving give a name to our container for easier identification and termination
--mount type=bind,source=$TESTDATA/autoencoder,target=/models/autoencoder
mounts the content of the autoencoder folder, this is required for the serving application to locate the correct model
-e MODEL_NAME=autoencoder
pass the environment variable MODEL_NAME to the serving application to help locate the correct model

Hopefully after launching the latest docker container with the above command you see an output without any errors, indicating that our custom model is up and running correctly. Again we will be confirming this with a postman request. This time the POST endpoint localhost:8501/v1/models/autoencoder:predict and the feature vector should have the dimension [1, 784] (28x28 pixels). I made a sample payload of 784 1.0 values, you can grab it bellow.‌  

Here is the sample digit I used.                              

After sending a request via postman we can see that the model is up, running and returning a [1, 784] response as expected!

We can even plot the digits to visualize the result using a function like this:

import matplotlib.pyplot as plt
import numpy as np

def compare_digits(raw, processed):
  image = np.append(raw, processed)
  image = np.array(image, dtype='float')
  pixels = image.reshape((56, 28))
  plt.imshow(pixels, cmap='gray')
  plt.show()

And should see something like this:

Again before moving on we want to make sure clean up docker container:

$ docker kill serving

Deploy multiple custom models simultaneously

Our serving application is able to serve us one model but in a production environment we would typically expect for multiple models to be available at the same time. You will notice in our original docker run command we mounted our model folder and passed -e MODEL_NAME=<model>. Mounting multiple model folders can be done without issue but passing multiple model names cannot be done directly in this request. To bypass this challenge we can store our model information in a configuration file and providing it to the serving application.

Before creating the config file we need to add the folders for the second model and one to store the config file and copy the build classifier model:

$ mkdir serving\mnist\classifier
$ mkdir serving\mnist\config
$ cp tf-mnist-project\models\classifier\production\1 serving\mnist\classifier -r

Now lets create a models.config file in the config directory. This is what it should look like:

model_config_list: { 
  config: {
    name: "classifier",
    base_path: "/models/classifier",
    model_platform: "tensorflow"
  },
  config: {
    name: "autoencoder",
    base_path: "/models/autoencoder",
    model_platform: "tensorflow"
  },
}

The new folder structure should now look like this:

├── tf-mnist-project
│ ├── src
│ │.......
├── serving
│ ├── tensorflow_serving
│ ├── mnist
│ │ ├── autoencoder
│ │ │ ├── 1
│ │ │ │ ├── assets
│ │ │ │ ├── variables
│ │ │ │ └── saved_model.pb
│ │ ├── classifier
│ │ │ ├── 1
│ │ │ │ ├── assets
│ │ │ │ ├── variables
│ │ │ │ └── saved_model.pb
│ │ ├── config
│ │ │ ├── models.config
│ │.......

Lets launch docker again, this time mounting both model folders along with the configuration file.

Linux - launching the container

$ TF_MODEL_DIR="$(pwd)/serving/mnist" 
$ docker run \
--rm -p 8501:8501 --name serving \
--mount type=bind,source=%TF_MODEL_DIR%/classifier,target=/models/classifier \
--mount type=bind,source=%TF_MODEL_DIR%/autoencoder,target=/models/autoencoder \
--mount type=bind,source=%TF_MODEL_DIR%/config,target=/config  tensorflow/serving \
--model_config_file=/config/models.config 

Windows - launching the container

$ set TF_MODEL_DIR=%cd%/serving/mnist 
$ docker run --rm -p 8501:8501 --name serving --mount type=bind,source=%TF_MODEL_DIR%/classifier,target=/models/classifier --mount type=bind,source=%TF_MODEL_DIR%/autoencoder,target=/models/autoencoder --mount type=bind,source=%TF_MODEL_DIR%/config,target=/config tensorflow/serving --model_config_file=/config/models.config

Notice we are mounting 3 folders, one for each model and one that stores our configuration file. We then tell the module to use the model.config from the mounted config folder to figure out how to map our models to the other 2 folders using --model_config_file=/config/models.config. The output should look like the below (take note that there both of our models are now up and running.

We now have two models accessible on:

localhost:8501/v1/models/autoencoder:predict
localhost:8501/v1/models/classifier:predict

If you rerun the autoencoder test from above you will see that the model is still functional, and if you forward the response to the classify endpoint you should see something like this:

As you can see the classifier model is also up and running and has classified our sample 3 digit correctly.

I converted my serving module into a personal repository, you can check it out here (if you clone my repository you will need to change some of the path values as the root folder name will no longer be serving). Now that we have our models up and running the next step is to set up a basic external application to utilize the API endpoints and demonstrate the functionality.  

Here is a summary of the components involved in this project:

This post will be covering the two models that were set up in TensorFlow to process MNIST digit data, how training was conducted and finally how the results were converted into a tangible model to be leveraged down stream. This post is part of the TensorFlow + Docker MNIST Classifier series.

I will not be covering the basics of TensorFlow in these posts. Typically I am not a huge fan of programming literature myself with the massive amount of resources available online, however for learning TensorFlow I highly recommend this e-book for grasping the fundamentals.

The Data Set (MNIST): This is one of the most popular machine learning data sets on the internet at the moment. It consists of tens of thousands of 28 x 28 labeled hand written written digits like the one below.

One of the key success criteria for this project was the use of multiple models in the final solution. The first model will be an auto-encoder to standardize the image data and the second model will classify it.

Features and targets

The features or digits will be passed through the model as a 784 dimensional vectors with each element of the vector representing pixel intensity (white to black) of each pixel in the 28 x 28 image. Scaling was used on the feature data to improve performance converting the value range from [0.0, 255.0] to [0.0, 1.0] by dividing each value by 255.

Data set labels (targets) are a single dimensional vector with values ranging from 0 - 9, representing the 10 potential digit classes. In order to improve model performance and simplicity these were transformed into a 10 dimensional one-hot representation with each dimension representing the probability of the associated digit e.g. [5] -> [0, 0, 0, 0, 0, 1, 0, 0, 0, 0].

The model

One of the goals of this project was to implement a system with 2 models and I chose to use an auto-encoder as my first model and a basic classifier for my second as illustrate below.

Keras is used to simplify development and training, config files are used to store hyperparameters and file paths and I developer a basic helper for loading MNIST image data as I am not using Keras for data loading.

The Auto-encoder: There is a lot of great material on the auto-encoder network online including the wiki entry here. In a nutshell an auto-encoder is an unsupervised symmetrical neural network that compresses the feature vector into significantly fewer dimensions. The network is trained by using features as both the input and output of the network, teaching the filters to compress the features. One of the key uses of the auto-encoder is noise reduction and this is what it will be used for here.

Using Keras we can implement the neural network using the following code.

from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.compat.v1 import flags
from tensorflow.keras import optimizers
import sys, os
import config as conf

#set up and parse custom flags
flags.DEFINE_integer('model_version', conf.version, "Width of the image")
flags.DEFINE_boolean('rebuild', False, "Drop the checkpoint weights and rebuild model     from scratch")
flags.DEFINE_string('lib_folder', conf.lib_folder, "Local library folder")
FLAGS = flags.FLAGS

#mount the library folder
sys.path.append(os.path.abspath(FLAGS.lib_folder))
from data import MNISTProcessor
import visualizer as v

#load data
data_processor = MNISTProcessor(conf.data_path, conf.train_labels, 
                            conf.train_images, '', '')                               
x_data_train, y_data_train = data_processor.load_train(normalize=True).get_training_data()

#initialize the network
input_layer = Input(shape=(784,), name='input')
network = Dense(152, activation='tanh', name='dense_1')(input_layer)
network = Dense(76, activation='tanh', name='dense_2')(network)
network = Dense(38, activation='tanh', name='dense_3')(network)
network = Dense(4, activation='tanh', name='dense_4')(network)
network = Dense(38, activation='tanh', name='dense_5')(network)
network = Dense(76, activation='tanh', name='dense_6')(network)
network = Dense(152, activation='tanh', name='dense_7')(network)
output = Dense(784, activation='tanh', name='output')(network)

autoencoder = Model(inputs=input_layer, outputs=output, name='autoencoder')
autoencoder.compile(optimizer=optimizers.Adadelta(learning_rate=1.0), loss='MSE',     metrics=['accuracy'])

# Create a callback that saves the model's weights
cp_callback = ModelCheckpoint(filepath=conf.checkpoint_path, save_weights_only=True, verbose=1)

#load an existing model to continue training
if(not FLAGS.rebuild):
    try:
        autoencoder.load_weights(conf.checkpoint_path)
    except:
        print('No checkpoint found, building filters from scratch.')

#run the training
autoencoder.fit(x_data_train, x_data_train,
            epochs=conf.epochs,
            batch_size=conf.batch_size,
            shuffle=True,
            callbacks=[cp_callback])

#save the production version of the model
try:
    os.mkdir(conf.final_model_path + '/' + str(FLAGS.model_version))
except OSError:
    print ("Creation of the directory %s failed" % conf.final_model_path + '/' +     str(FLAGS.model_version))

autoencoder.save(conf.final_model_path + '/' + str(FLAGS.model_version),     overwrite=True, save_format='tf') 

autoencoder.summary()   

# run a sample for visualization
clean_images = autoencoder.predict(x_data_train)
v.visualize_autoencoding(x_data_train, clean_images, digits_to_show=10) 

To break down the code a little
lines 10-13 - using tensorflow flags to pull command line argument values
lines 21-23 - process the MNIST data set into features and labels
lines 26-37 - set up the neural network strcture and optimizer
lines 40 - set up callback for saving checkpoints during training
lines 43-47 - load any existing checkpoints
lines 50-54 - train the model
lines 57-62 - save a production version that will be ready for serving
lines 64-68 - display final model strcture and some sample autoencodings

The following function was created to help visualize the auto-encoder result.

import matplotlib.pyplot as plt


def visualize_autoencoding(original_data, decoded_data, digits_to_show=10):
    plt.figure(figsize=(20, 4))
    for i in range(digits_to_show):
        # display original
        sub_plot = plt.subplot(2, digits_to_show, i + 1)
        plt.imshow(original_data[i].reshape(28, 28))
        plt.gray()
        sub_plot.get_xaxis().set_visible(False)
        sub_plot.get_yaxis().set_visible(False)

        # display reconstruction
        sub_plot = plt.subplot(2, digits_to_show, i + 1 + digits_to_show)
        plt.imshow(decoded_data[i].reshape(28, 28))
        plt.gray()
        sub_plot.get_xaxis().set_visible(False)
        sub_plot.get_yaxis().set_visible(False)
    plt.show()

this function can be called like so: visualize_autoencoding(x_data_train, clean_images, digits_to_show=4) from our training program after the training is completed.

After training my error loss was around 0.025 and you can see below what a few sample images looked like after being passed through the trained auto-encoder. The result could be improved but this should be satisfactory for our needs.

The Classifier: The second model will take the 784 dimensional vector output by the auto-encoder and and classifying the data into one of the 10 possible digit values [0, 9]. A simple tanh activated deep neural network will be used.

Keras was used to implement the classifier as well. We first load and process the image data through the auto-encoder before using as the feature input for training of the classifier.

from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.keras import optimizers
from tensorflow.compat.v1 import flags
import tensorflow.keras as Keras
from tensorflow.keras.callbacks import ModelCheckpoint
import sys, os
import config as conf    

#set up and parse custom flags
flags.DEFINE_integer('model_version', conf.version, "Width of the image")
flags.DEFINE_boolean('rebuild', False, "Drop the checkpoint weights and rebuild model from scratch")
flags.DEFINE_string('lib_folder', conf.lib_folder, "Local library folder")
flags.DEFINE_integer('encoder_version', 1, "Autoencoder version to use")
FLAGS = flags.FLAGS

#mount the library folder
sys.path.append(os.path.abspath(FLAGS.lib_folder))
from data import MNISTProcessor

#load data
data_processor = MNISTProcessor(conf.data_path, conf.train_labels, 
                            conf.train_images, '', '')
x_data_train, y_data_train = data_processor.load_train(normalize=True).get_training_data()

# Load the autoencoder model, including its weights and then process images
autoencoder = Keras.models.load_model(conf.autoencoder_model_path + '/' +  str(FLAGS.encoder_version))
clean_images = autoencoder.predict(x_data_train)

#initialize the classification network
input_layer = Input(shape=(784,))
network = Dense(140, activation='tanh', name='dense_1')(input_layer)
network = Dense(80, activation='tanh', name='dense_2')(network)
network = Dense(40, activation='tanh', name='dense_3')(network)
output = Dense(10, activation='tanh', name='dense_4')(network)

classifier = Model(inputs=input_layer, outputs=output, name='classifier')
classifier.compile(optimizer=optimizers.Adadelta(learning_rate=1.0), loss='MSE', metrics=['accuracy'])

# Create a callback that saves the model's weights
cp_callback = ModelCheckpoint(filepath=conf.checkpoint_path, save_weights_only=True, verbose=1)

#load an existing model to continue training
if(not FLAGS.rebuild):
    try:
        classifier.load_weights(conf.checkpoint_path)
    except:
        print('No checkpoint found, building filters from scratch.')

#run the model
classifier.fit(clean_images, y_data_train,
            epochs=conf.epochs,
            batch_size=conf.batch_size,
            shuffle=True,
            callbacks=[cp_callback])

#save the production version of the model
try:
    os.mkdir(conf.final_model_path + '/' + str(FLAGS.model_version))
except OSError:
    print ("Creation of the directory %s failed" % conf.final_model_path + '/' + str(FLAGS.model_version))

classifier.save(conf.final_model_path + '/' + str(FLAGS.model_version), overwrite=True, save_format='tf') 

classifier.summary()     

To break down the code a little
lines 11-15 - using tensorflow flags to pull command line argument values
lines 22-24 - process the MNIST data set into features and labels
lines 27-28 - load the autoencoder model and process the feature data set
lines 31-38 - set up the neural network strcture and optimizer
lines 41 - set up callback for saving checkpoints during training
lines 45-49 - load any existing checkpoints
lines 51-55 - train the model
lines 57-62 - save a production version that will be ready for serving
lines 65 - display final model strcture

Lessons learned

Poor initial model convergence - I wrote my initial code using the stand alone keras library, however due to challenges of saving the models in a servable format I had to switch the the tf.keras library instead. After my switch my models flat out refused to converge during training. After many hours of debugging I discovered that the keras.optimizers.Adadelta optimizer uses a default starting learning rate of 1.0, where as the tf.keras.optimizersAdadelta optimizer initializes with a learning rate of 0.001. Forcing the learning rate addressed this issue for me and you can see this reflected in my code.

For the lazy

My results can be reproduced with the following commands:

#navigate to where you would like to generate the repository
$ git clone https://github.com/adidinchuk/tf-mnist-project
$ cd tf-mnist-project

$ curl http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz --output data/train-images-idx3-ubyte.gz
$ curl http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz --output data/train-labels-idx1-ubyte.gz

#Unzip the data using your prefer compression tool.
#Make sure the file names and location do not change,
#otherwise you will have to make the appropriate changes in the config filesv

#once the data has been extracted train the autoencoder model using
$ py -3.6 src/autoencoder/graph.py --model_version 1

#after the training completes you should see a .pb model file in the models/autoencoder/production folder

#now run the classifier training
$ py -3.6 src/classifier/graph.py  --model_version 1 --encoder_version 1

#after the training completes you should see a .pb model file in the models/classifier/production/#/ folder

You should now see the production models under models/autoencoder/production/1 and models/classifier/production/1 that looks like this:

The entire TensorFlow github repository along with complete instructions on running the model can be found here. Now that we have both the auto-encoder and classifier models generated we can take a look at deploying them via TensorFlow serving, which I will do in my next post.

Here is a summary of the components involved in this project: