SpeedUpAmerica - Infrastructure

This document covers the major pieces of SpeedUpAmerica’s (SUA) infrastructure.

The file may have been moved or deleted, or is temporarily unavailable.

https://paper-attachments.dropboxusercontent.com/s_4337AE647D933441D3A2F191D12222E37D0E048F6AA3627081A14F1624B12859_1560048977873_full_architecture.PNG

SpeedUpAmerica architecture- https://projects.invisionapp.com/freehand/document/7os4Lwdbq

These services are used to host SpeedUpAmerica:

AWS Route53 (DNS)

AWS Application Load Balancers (terminates HTTPS and distributes traffic across service instances)

AWS ECS (runs services/tasks on EC2 instances)

AWS RDS (hosts MySQL DBs)

AWS CodePipeline (builds, uploads, and deploys changes)

AWS CodeBuild (builds and initiates nightly tasks)

AWS S3 (holds public and private files in separate buckets)

AWS CloudWatch (logs STDOUT from all tasks/services, builds, uploads, and deploys)

GitHub (hosts code and notifies CodePipeline when to run

Major SpeedUpAmerica code bases:

Frontend+Backend (v1) Ruby on Rails web service

Database Migrator

CodePiplines are setup to receive notifications about changes made by developers and deploy those changes to Test and Production environments. Each environment is a free standing instance of the whole system.

Not shown on the diagram above is a nightly data processing Elastic Container Service Task that imports new M-Lab data, adds Census Tracts to recently collected/imported submissions, and updates the provider statistics table. Also not shown is CloudWatch. Many pieces of the system send logs to CloudWatch. If you’re debugging a problem don’t forget to the look at those logs.

Environments

Production (alpha) - https://speedupamerica.com

Test (gamma) - https://test.speedupamerica.com

Each environment has it’s own VPC, ECS Services & Tasks, scheduled CodeBuild jobs, and RDS instances.

Details on Major Pieces

Nearly all object names in AWS will include alpha or gamma. If an object doesn’t include alpha/gamma it is either alpha or an object shared by both, like data in S3.

Route53

All public DNS record are handled by Route53.

Application Load Balancers

Each ECS Service and Environment has an Application Load Balancer. The ALBs handle TLS/SSL termination, HTTP → HTTPS redirection, and have Target Groups that are managed by ECS. Each ALB has a Target Group that is manged by ECS.

When debugging traffic issues make sure to check the ALBs Target Group and Security Group policies.

ECS

Elastic Container Service runs ECS Tasks using Docker Containers. Each ECS Service will have one or more Tasks receiving traffic from an ALB.

We have a clusters named speedupamerica-alpha(Production) and speedupamerica-gamma(Test). It currently (2019-06-09) each has two t3.medium EC2 instances. Each ECS Cluster also had match short-lived ECS Tasks that are run on-demand to import/update data and run database migrations.

RDS

Currently (2019-06-09) SUA uses a MySQL database to hold boundary data, provider statistics, and submissions (speed test results). Each of these are running a t2.medium instance and have automated snapshots going back 1 week.

CodePipeline & CodeBuild

Using a GitHub webhook and AWS CodePipeline we triggers builds, publish Docker images to Elastic Container Registry, run database migrations, schedule and run nightly data procecess, and deploy to Test and Production. Deploying what is live on Test to Production requires manually approving the deployment in CodePipeline. All builds/projects have a separate projects for Test and Production, take care to keep the project configurations as uniform as possible.

There are a few CodeBuild projects defined that are not part of a pipeline. Currently these tasks are schedule to perform nightly data imports/updates.

S3 Buckets

Currently the infrastructure has two S3 buckets, sua-public and sua-private. The public bucket holds large file we need easy access to, but don’t want to check into our repos. The private bucket holds sensitive files. Currently, the private bucket is only used used by CodeBuild when building Docker Images.

CloudWatch

All objects that can emitting logs should be configured to store those logs in CloudWatch. Currently all CodeBuild projects and ECS Tasks are configured to store STDOUT in CloudWatch logs. These logs are critical for debugging. Keep in mind the logs are in UTC as is most dates in the system, include the databases.