Prerequisites

A functional ECS install

If you don't have one, Getting Started with ECS is a great resource to get one set up quickly.

Install the tbnctl command line interface (CLI)

tbnctl is a CLI for interacting with the Turbine Labs public API, and is used throughout this guide to set up tbnproxy. Install tbnctl with these commands (Requires installation of Go, and that $GOPATH/bin is in your $PATH):

$ go get -u github.com/turbinelabs/tbnctl
$ go install github.com/turbinelabs/tbnctl
$ tbnctl login

Use your Houston username and password to login.

Username [somebody@example.com]:
Password:

See the tbnctl Guide for more information.

Get an API Access Token

Create an API Access Token using tbnctl:

$ tbnctl access-tokens add "demo key"
{
  "access_token_key": "<redacted>",
  "description": "demo key",
  "signed_token": "<redacted>",
  "user_key": "<redacted>",
  "created_at": "2017-08-25T22:11:30.907200482Z",
  "checksum": "d60ed8a6-1a40-49a5-5bb1-5bad322d9723"
}

You'll need the value of signed_token later on, so keep it somewhere secure.

Create a Zone

The highest-level unit of organization in the Turbine Labs API is a zone. We'll use the zone "testbed" in this guide, but you can substitute your own if you've already created one. To create the testbed zone, run

$ tbnctl init-zone testbed

You should now be able to see your zone by running

$ tbnctl list zone

Setting up service discovery

Install tbncollect with the following task definition filling in the bracketed variables, including the signed_token obtained with tbnctl:

$ aws ecs \
  register-task-definition \
  --family tbncollect \
  --container-definitions '
{
  "name": "tbncollect",
  "image": "turbinelabs/tbncollect:0.15.1",
  "cpu": 1,
  "memory": 128,
  "memoryReservation": 128,
  "environment": [
    {
      "name": "TBNCOLLECT_CMD",
      "value": "ecs"
    },
    {
      "name": "TBNCOLLECT_API_KEY",
      "value": "<your signed_token>"
    },
    {
      "name": "TBNCOLLECT_API_ZONE_NAME",
      "value": "<your TBN zone name>"
    },
    {
      "name": "TBNCOLLECT_ECS_AWS_ACCESS_KEY_ID",
      "value": "<your AWS access key>"
    },
    {
      "name": "TBNCOLLECT_ECS_AWS_REGION",
      "value": "<your AWS region>"
    },
    {
      "name": "TBNCOLLECT_ECS_AWS_SECRET_ACCESS_KEY",
      "value": "<your AWS secret access key>"
    },
    {
      "name": "TBNCOLLECT_ECS_CLUSTERS",
      "value": "<your cluster name>"
    }
  ]
}
'

With your task definition created, you can proceed to run Create Service from the ECS control panel, or through the CLI:

$ aws ecs \
  create-service \
  --cluster default \
  --service-name tbncollect \
  --task-definition tbncollect:1 \
  --desired-count 1

The all-in-one demo

Configure your tasks and containers

In order for tbncollect to see your all-in-one tasks, you'll need to add a Cluster tag, which is attached to a container definition within a task definition. Clusters are the grouping that tbncollect uses for services, and in the case of ECS, are comprised of one or more tasks. A task can be one, or many containers, but as long as the container includes the same Cluster tag as a Docker label, the containers will be grouped together.

Please note that using the same label for each container in a task will give you multiple instances inside a Cluster in the Turbine Labs service.

The construction of the tag inside of a task definition looks like this for the server:

{
  "dockerLabels": {
    "tbn-cluster": "all-in-one-server:8080"
  }
}

and this for the client:

{
  "dockerLabels": {
    "tbn-cluster": "all-in-one-client:8080"
  }
}

The label key is the "Cluster tag", which will associate the container with a Turbine Labs' Cluster, and the value is Turbine Labs' Cluster name and port. Example task definitions are included later, which include these labels.

Install the all-in-one-server

Next, create the following task definition for the all-in-one-server-blue which returns the color blue to the all-in-one-client.

$ aws ecs \
  register-task-definition \
  --family all-in-one-server-blue \
  --container-definitions '
{
  "name": "all-in-one-server-blue",
  "image": "turbinelabs/all-in-one-server:latest",
  "cpu": 1,
  "memory": 128,
  "memoryReservation": 128,
  "portMappings": [
    {
      "hostPort": 8080,
      "containerPort": 8080,
      "protocol": "tcp"
    }
  ],
  "dockerLabels": {
    "version": "blue",
    "stage": "prod",
    "tbn-cluster": "all-in-one-server:8080"
  },
  "environment": [
    {
      "name": "TBN_COLOR",
      "value": "1B9AE4"
    }
  ]
}
'

Create a service with this task definition on an ECS cluster of your choosing.

$ aws ecs \
  create-service \
  --cluster default \
  --service-name all-in-one-server-blue \
  --task-definition all-in-one-server-blue:1 \
  --desired-count 1

Install the all-in-one-client

The following task definition is for the all-in-one-client, which will show the results of the color returned by the all-in-one-server visually as a blue block.

$ aws ecs \
  register-task-definition \
  --family all-in-one-client \
  --container-definitions '
{
  "name": "all-in-one-client",
  "image": "turbinelabs/all-in-one-client:latest",
  "cpu": 1,
  "memory": 128,
  "memoryReservation": 128,
  "portMappings": [
    {
      "hostPort": 8080,
      "containerPort": 8080,
      "protocol": "tcp"
    }
  ],
  "dockerLabels": {
    "tbn-cluster": "all-in-one-client:8080"
  }
}
'

Create a service with this task definition on your ECS cluster.

$ aws ecs \
  create-service \
  --cluster default \
  --service-name all-in-one-client \
  --task-definition all-in-one-client:1 \
  --desired-count 1

Adding a domain and proxy

Tbnproxy is the container that handles request routing. It serves traffic for a set of domains, which in turn contain release groups and routes. We'll create the domain first.

Go to https://app.turbinelabs.io, and login with your email address and password.

Click "Settings" in the top right portion of the screen, then "Edit Routes".

The screen should indicate that you have no domains. Click "Add One".

type "testbed-domain" in the Name field, then Click "Add Domain"

The screen should now indicate that you have no proxies. Click "Add One".

type "testbed-proxy" in the Name field, and then check the box next to testbed-domain:80. This indicates that the proxy you're adding will serve traffic for testbed-domain. Click "Add Proxy"

Create tbnproxy

Create tbnproxy with this task definition and note the variables you'll need to modify with values for your environment. tbnproxy will be visible to the web, and your customer's traffic. tbnproxy will also need network connectivity to all ECS tasks.

This tbnproxy task definition can be adapted to your needs or environment, including the signed_token obtained with tbnctl:

{
  "name": "tbnproxy",
  "image": "turbinelabs/tbnproxy:0.15.1",
  "cpu": 1,
  "memory": 128,
  "memoryReservation": 128,
  "portMappings": [
    {
      "hostPort": 80,
      "containerPort": 80,
      "protocol": "tcp"
    }
  ],
  "environment": [
    {
      "name": "TBNPROXY_API_KEY",
      "value": "<your signed_token>"
    },
    {
      "name": "TBNPROXY_API_ZONE_NAME",
      "value": "<your zone name>"
    },
    {
      "name": "TBNPROXY_PROXY_NAME",
      "value": "<your proxy name>"
    }
  ]
}

Create a service with this task definition on your ECS cluster.

Create a service to expose tbnproxy

With your task definition created, we can then expose tbnproxy to the internet using an Elastic Load Balancer. First create your ELB using the management console, with no listeners. Now expose tbnproxy to the ELB by running Create Service from the ECS control panel or through the CLI:

$ aws ecs \
  create-service \
  --cluster default \
  --service-name tbnproxy \
  --task-definition tbnproxy:1 \
  --desired-count 1
  --load-balancers <the ELB you created above goes here>

Verifying your deploy

With your ELB running, locate its external IP, and visit it in your browser. You should be able to see blue boxes in a grid, blinking in and out, as they represent responses from the blue version of the all-in-one-server we launched previously.

Demo exercises

Now that you're up and running with Houston on ECS, let's walk through some product use cases.

What's going on here?

The all-in-one client/server provide a UI and a set of services that help visualize changes in the mapping of user requests to backend services. This lets you visualize the impact of Houston on a real deployment without having to involve real customer traffic or load generators.

The application is composed of three sets of blocks, each simulating a user making a request. These are simple users, and they all repeat the same request forever. The services they call return a color. When a user receives a response it paints the box that color, then waits a random amount of time to make another request. While it’s waiting the colors in the box fade. Users are organized into rows based on URL.

You should see pulsating blue boxes for each service, to indicate the initial state of your production services.

Deployed state

Let’s dig deeper into how tbnproxy routes traffic. Traffic is received by a proxy that handles traffic for a given domain. The proxy maps requests to service instances via routes and rules. Routes let you split your domain into manageable segments, for example /bar and /baz. Rules let you map requests to a constrained set of service instances in clusters, for example “by default send traffic to servers labeled stage=prod. Clusters contain sets of service instances, each of which can be labeled with key/value pairs to provide more information to the routing engine.

Your environment should look like the following:

There is a single domain, local.domain that contains two routes. /api handles requests to our demo service instances, and / handles requests for everything else (in this case the demo app). There are two clusters:

  • The all-in-one-server cluster has one instance, labeled as stage=prod,version=blue. The all-in-one-client cluster has a single instance labeled stage=prod.

  • The all-in-one-server cluster has a single instance labeled stage=prod.

Set up an initial route

The rules currently map api traffic to all instances in the cluster. To enable the release workflow we need to constrain routing to a single version at a single stage, so let's configure Houston to route traffic to the blue version.

  1. Make sure you have the 'testbed' zone selected in the top left portion of the screen.
  2. Click the "Settings" menu in the top right portion of the screen, and then select "Edit Routes".
  3. Click the "Select View" menu in the top left portion of the screen, and select the api route.
  4. Change 1 to 'all-in-one-server' to 1 to 'all-in-one-server' stage = prod & version = blue
  5. Click "Save Release Group"

If you look at the all-in-one client you should still see all blue blocks, because we've constrained the routing to only go to servers in the cluster labeled with version=blue.

Deploying a new version

Now we'll deploy a new version of the server that returns green as the color to paint blocks. Use this task definition to create a service for a new server that returns the color green to the all-in-one-client.

$ aws ecs \
  register-task-definition \
  --family all-in-one-server-green \
  --container-definitions '
{
  "name": "all-in-one-server-green",
  "image": "turbinelabs/all-in-one-server:latest",
  "cpu": 1,
  "memory": 128,
  "memoryReservation": 128,
  "portMappings": [
    {
      "hostPort": 8081,
      "containerPort": 8080,
      "protocol": "tcp"
    }
  ],
  "dockerLabels": {
    "version": "green",
    "stage": "prod",
    "tbn-cluster": "all-in-one-server:8081"
  },
  "environment": [
    {
      "name": "TBN_COLOR",
      "value": "83D061"
    }
  ]
}
'

Create a service with this task definition on an ECS cluster of your choosing.

$ aws ecs \
  create-service \
  --cluster default \
  --service-name all-in-one-server-green \
  --task-definition all-in-one-server-green:1 \
  --desired-count 1

Your environment now looks like the following:

The new instance has been added to the all-in-one-server cluster, but no traffic is routed to it. When returning to your client app, you should still see only blue blocks, because we set our routing constraints in the previous step.

Testing before release

Let’s test our green version before we release it to customers. tbnproxy allows you to route to service instances based on headers set in the request. Navigate to app.turbinelabs.io, log in and select the zone you’re working with (testbed by default). Click the "Release Groups" tab below the top-line charts, then click the pencil icon in the "all-in-one-server" row. This will take you to the Release Group editor.

In the "Request-Specific Overrides" section, click "Add an Override". Fill in the values as below:

This tells the proxy to look for a header called X-Tbn-Version. If the proxy finds that header, it uses the value to find servers in the all-in-one-client cluster that have a matching version label. For example, setting X-Tbn-Version: blue on a request would match blue servers, and X-Tbn-Version: green would match green servers.

Click "Save Changes" in the top right. Now click "More..." and then "View Charts" to go back to the chart view.

The demo app converts a X-Tbn-Version query parameter into a header in calls to the backend; if you navigate to http://<your external IP>?X-Tbn-Version=green you should see all green boxes. Meanwhile going to http://<your-client> without that parameter still shows blue.

This technique is extremely powerful. New software was previewed in production without customers being affected. You were able to test the new software on the live site before releasing to customers. In a real world scenario your testers can perform validation, you can load test, and you can demo to stakeholders without running through a complicated multi-environment scenario, even during another release.

Incremental release with Simple Release Workflow

Configuration of Simple Release Workflow

Now we're ready to do an incremental release from blue to green. Right now the default rules for /api send all traffic to blue. Let’s introduce a small percentage of green traffic to customers.

First, we must enable the Simple Release Workflow. Navigate to app.turbinelabs.io, log in and select the zone you’re working with (testbed by default). Click the "Release Groups" tab below the top-line charts, then click the pencil icon in the "all-in-one-server" row. This will take you to the Release Group editor. Scroll down to "Default Behavior"

Click "Manage" to enable Simple Release Management. Choose the label which will vary with different versions of your service (in this case "version"), and the current value (in this case "blue").

Click "Enable Simple Release Workflow". Then click "Save Changes" in the top right of the window. Finally, click "More..." and then "View Charts" to go back to the chart view.

Incremental release

The "all-in-one-server" row should be now marked "RELEASE READY". Click anywhere in the row to expand it, then click "Start Release".

Let's send 25% of traffic to our new green version by moving the slider and clicking "Start Release". The release group should now be marked "RELEASING".

The all in one client should now show a mix of blue and green. You can increment the green percentage as you like. When you get to 100%, the release is complete.

Congratulations! You've safely and incrementally released a new version of your production software. Both blue and green versions are still running; if a problem were found with green, a rollback to blue would be just as easy.

Testing latency and error rates

In order to demo what errors and latency issues may look like in a production environment, we implemented a few parameters that can be set to illustrate these scenarios. By default, each of the demo servers returns a successful (status code 200) response with its color (as a hex string) as the response body.

URL parameters passed to the client web page at can be used to control the mean latency and error rate of each of the different server colors.

An example

The following URL will show an error rate and delayed response for green and blue servers.

http://<your external IP>/?x-blue-delay=25&x-blue-error=.001&x-green-delay=10&x-green-error=.25

This will simulate a bad green release, and a need to rollback to a known good blue release.

Parameter effect

These parameters can be modified in the above example as follows:

  • x-color-delay: sets the mean delay in milliseconds.
  • x-color-error: sets the error rate, describe as a fraction of 1 (e.g., 0.5 causes an error 50% of the time).

The latency and error rates are passed to the demo servers as HTTP headers with the same name and value as the URL parameters described. You can use these parameters to help you visualize the effects of a bad release, or an issue with the code in a new version of your application, which would be cause to step-down the release and return traffic to a known-good version.

Driving synthetic traffic

If you'd like to drive steady traffic to your all-in-one server without keeping a browser window open, you can run the all-in-one-driver image in your kubernetes environment. You can use the template below, filling in the value for ALL_IN_ONE_DRIVER_HOST with the external IP of your ELB. You can also add error rates and latencies for various using environment variables.

$ aws ecs \
  register-task-definition \
  --family all-in-one-driver \
  --container-definitions '
{
  "name": "all-in-one-driver",
  "image": "turbinelabs/all-in-one-driver:0.10.1",
  "cpu": 1,
  "memory": 128,
  "memoryReservation": 128,
  "environment": [
    {
      "name": "ALL_IN_ONE_DRIVER_HOST",
      "value": "<YOUR PUBLIC IP OR HOSTNAME>:80"
    },
    {
      "name": "ALL_IN_ONE_DRIVER_LATENCIES",
      "value": "blue:50ms,green:20ms"
    },
    {
      "name": "ALL_IN_ONE_DRIVER_ERROR_RATES",
      "value": "blue:0.01,green:0.005"
    }
  ]
}
'

With your task definition created, you can proceed to run Create Service from the ECS control panel, or through the CLI:

$ aws ecs \
  create-service \
  --cluster default \
  --service-name all-in-one-driver \
  --task-definition all-in-one-driver:1 \
  --desired-count 1

Wrapping-up

Now that you've seen what Houston can do with our all-in-one examples and ECS , you can try it out with your own services. If you have questions or run into any trouble, please drop us a line, we're here to help.