CircleCI + AWS

= Happy Deployment
😃 🚀

Follow Along

v.gd/circleciaws

Russell Heimlich
(like the maneuver)

Lead Developer at Spirited Media

We build software!

We need to get that software to other places

But we don't want to break things

Continuous Integration / Continuous Deployment

  • Automated
  • Predictable
  • Stops if something breaks

Deployment Should Be Boring

Keep Calm Boring is Good
CircleCI

What does CircleCI do?

  • Listens for changes to a GitHub/Bitbucket repo
  • Does stuff we tell it to do

Add a .circleci/config.yml file to the root of your project

Steps in the build process are called jobs

One or more jobs makeup a workflow

Workflows can run sequentially or in parallel
(fan in/out)

Our build process is straightforward

  • Setup the build enviornment
  • Download dependencies
  • Compile static assets (SCSS → CSS, minifying JavaScript files)
  • Lint files to make sure they conform to coding standards

config.yml

version: 2
references:
  # Default container configuration
  docker_config: &docker_config
    docker:
      - image: circleci/php:7.0-node-browsers
Pre-built CircleCI Docker Images
jobs:
  build:
    <<: *docker_config
    steps:
      - checkout
- restore_cache:
    keys:
      - v1-dependency-cache-{{ checksum "yarn.lock" }}
      - v1-dependency-cache
- restore_cache:
    keys:
      - v1-composer-cache-{{ checksum "composer.lock" }}
      - v1-composer-cache
A lock file is a snapshot of the current dependency tree and allows for reproducible builds between machines
- run:
  name: Remove previous Yarn
  command: sudo rm /usr/local/bin/yarn*
- run:
  name: Install Yarn
  command: sudo npm install -g yarn
- run:
  name: Install Grunt CLI
  command: sudo npm install -g grunt-cli
- run:
  name: Install NPM Dependencies
  command: yarn install
- run:
  name: Install Composer Dependencies
  command: composer install -o --no-dev
- save_cache:
  key: v1-dependency-cache-{{ checksum "yarn.lock" }}
  paths: node_modules
- save_cache:
  key: v1-composer-cache-{{ checksum "composer.lock" }}
  paths: vendor
- run:
    name: Building
    command: |
      if [ "${CIRCLE_BRANCH}" == "staging" ]; then
        grunt
      else
        grunt build
      fi
- run:
    name: Maybe Deploying?
    command: |
      if [ "${CIRCLE_BRANCH}" == "staging" ]; then
        .circleci/deploy.sh
      fi
      if [ "${CIRCLE_TAG}" ]; then
        .circleci/deploy.sh
      fi
CircleCI Enviornment Variables
workflows:
    version: 2
    build-and-deploy:
      jobs:
        - build:
            filters:
              branches:
                ignore: master
              tags:
                only: /^v[0-9]+(\.[0-9]+)*/

Deployment

  • 🚀 We want to get our code changes out to servers FAST!!!
  • 🍰 Only download the changes, not every single file
  • 👍 Git is awesome for this

deploy.sh

#!/bin/bash
echo "Starting Deploy..."
git config --global user.name "CircleCI"
git config --global user.email "some-random-email-address@spiritedmedia.com"
# Make a new README.md with build status details
rm README.md
cat > README.md EOF
Build [#$CIRCLE_BUILD_NUM]($CIRCLE_BUILD_URL) by $CIRCLE_USERNAME at $TIMESTAMP
[$CIRCLE_COMPARE_URL]($CIRCLE_COMPARE_URL)
EOF
Build #6608 by kingkool68 at 2019-02-07 04:00 AM UTC
spiritedmedia/spiritedmedia/compare/fc0e1c7f394a...52af362adfde
See this gist for details about parsing a Git log
# This should be ignored in .gitignore-build
# but let's try and remove it just to be safe
rm -rf node_modules/

# Find all .git/ directories and remove them.
# If we commit directories with .git in them then they are
# treated like sub-modules and screw that.
find . | grep -w ".git" | xargs rm -rf

# Remove all .gitignore files in
# the wp-content/ and vendor/ directories
find wp-content/ vendor/ -name ".gitignore" | xargs rm

rm .gitignore
mv .gitignore-build .gitignore
git clone git@github.com:spiritedmedia/spiritedmedia-build.git tmp/
mv tmp/.git .
rm -rf tmp/
# If no branch is set then assume master
if [ ! $CIRCLE_BRANCH  ]; then
  CIRCLE_BRANCH="master"
fi

# Switch branches... maybe?
if [ ! `git branch --list $CIRCLE_BRANCH` ]; then
  # Branch doesn't exist. Create it and check it out.
  # See http://stackoverflow.com/a/21151276
  git checkout -b $CIRCLE_BRANCH
else
  git checkout $CIRCLE_BRANCH
fi
# Add everything and commit
git add -A
git commit -m "Build #$CIRCLE_BUILD_NUM by $CIRCLE_USERNAME on $TIMESTAMP"
git push origin $CIRCLE_BRANCH --force

echo "Code changes pushed!"

🎉 Hooray!

🤔 Now how do we get this code to servers?

AWS CodePipeline is the glue between GitHub and AWS CodeDeploy

AWS CodeDeploy talks to our servers and tells them to update

(Requires the AWS CodeDeploy Agent to be installed)

CodeDeploy is also configured by a YAML file, appspec.yml.

version: 0.0
os: linux
files:
hooks:
  BeforeInstall:
    - location: bin/codedeploy.sh
      timeout: 600
      runas: root

codedeploy.sh

#!/bin/bash
# The appsec.yml file prohibits calling arbitrary scripts.
# This stub shell script downloads and executes
# the shell script specified in the
# EC2 instance User Data field.
# The user data shell script calls the deploy script
# baked in to the AMI running on the server.

# Fetch the user data script associated with this
# type of instance and save it into a temporary shell script
sudo curl http://169.254.169.254/latest/user-data > temp.sh

# Execute the shell script to run the update
sudo bash temp.sh

# Clean up
sudo rm temp.sh

“You can specify user data to configure an instance or run a configuration script during launch.”
🆒

deploy-production.sh

#!/bin/bash
# Shell script to update the app with
# the latest changes from GitHub (ex. when called form AWS CodeDeploy)
# Should be placed in /var/www/spiritedmedia.com/scripts/
# and run as root

cd /var/www/spiritedmedia.com/htdocs/

# Force git pull
git fetch --all
git reset --hard origin/master

# Reset file ownership
chown -R www-data:www-data /var/www/spiritedmedia.com/htdocs/

Tips & Tricks

Make a separate GitHub user and use that for managing credentials to different servers.

(GitHub calls these "Machine Users")
Debug CircleCI builds via SSH
Setup deployment notifications (like via Slack) using AWS SNS
Purge AWS CodePipeline artifacts in S3 with a lifecycle rule

💰 Pricing

CircleCI

  • 1st container = 1000 build minutes free per month
  • $50/month for each additional container, unlimited build minutes
  • Open source = 4 free containers

AWS CodeBuild

  • $0.005 per build minute (smallest tier)
  • First 100 minutes per month are free

📈 If your monthly build time is…

  • Less than 168.333 hours → AWS CodeBuild
  • Greater than 168.333 hours → CircleCI

CircleCI
vs
AWS CodeBuild

Pros of CircleCI

  • Awesome debugging experience
  • Much easier to set-up paralell jobs
  • Pricing if you need 1000 build minutes or less per month

Cons of CircleCI

  • Expensive once you need to move past the free tier

Pros of AWS CodeBuild

  • Pay-for-what-you-use pricing
  • Cost if you need more than 1000 build minutes per month
  • Run as many concurrent builds as you need

Cons of AWS CodeBuild

  • Breaking up one job into multiple concurrent parts is difficult
  • More work to set up (you need CodePipeline to trigger builds for example)

Further Reading

In Conclusion

Continuous Integration / Continuous Deployment is like playing dominos with YAML files and shell scripts.

But it's so awesome when it all works! 🚀

Thank You!

Tweet me! @kingkool68

Questions?