I encountered a lot of issues installing Spinnaker (1.6.1 and 1.7.x) on Ubuntu in Amazon Web Services. This post is the collection of steps I took to get a demo server with a working Spinnaker and Jenkins installation. I am going to use GitHub and the GitHub Branch Source Plugin from CloudBees in order to trigger my build/bakes. My target flow for the demo is:
This implementation is opinionated and very insecure, but it gives you an easy 10-15 minute commit through deploy demonstration to really open discussions about Spinnaker as a deployment platform for your project or organization.
Spinnaker-demo is a sample repo that includes a Jenkinsfile (with some master branch specific work), includes a few simple test files, and builds a Debian package that is designed to work on a vanilla Ubuntu server. Spinnaker-demo is really two html files and a bunch of test and packaging logic. This makes it easy for any developers or interested parties to make simple changes to an html file and watch their changes go live.
You can clone this repo or generate your own, but I think it is important that the demo application be simple so that version changes are apparent and updates are not a source of complication.
We need to create a few AWS objects the Spinnaker needs and/or expects in order to function properly. For simplicity and consistency, this Spinnaker instance will run and deploy in us-west-2. This is mostly a reworking of the setup instructions with a few additions and some subtle naming changes.
This is more than Spinnaker requires, but the goal is to get Spinnaker up and running with the least hassle.
Login to AWS and proceed to add a new IAM user.
Create VPC
Add a Tag to your subnet indicating the purpose with Key:immutable_metadata
and Value:{"purpose": "spin demo"}
When Spinnaker launches an EC2 instance, it will assign this role. For a light demonstration this role may not be important, and may not even get used, but Spinnaker expects to have a Role available for assignment.
When Spinnaker launches an EC2 instance, it will associate this key pair. Like the EC2 Role, this may not be necessary for a demonstration, but Spinnaker will expect to assign a key pair to instances it launches.
For whatever reason, the concept of “managing/managed” feels overly complicated to me. I am using a single account, and I’ll name with the concept of primary and assumed. The important parts here are our two EC2 roles: spinnaker (itself/primary) and spinnaker-assumed. Spinnaker needs a policy that allows the AssumeRole action, so that it can assume the spinnaker-assumed role and launch ec2 instances there. Spinnaker-assumed needs a policy that allows the PassRole action, so that it can pass the spinnaker-launched IAM EC2 role to newly launched instances. In addition, spinnaker-assumed needs to have a trust relationship with spinnaker in order to allow it to use AssumeRole successfully.
Replacing <YOUR_ACCOUNT_ID>
with your AWS account id, select the JSON tab and enter:
{
"Version": "2012-10-17",
"Statement": [{
"Action": "sts:AssumeRole",
"Resource": [
"arn:aws:iam::<YOUR_ACCOUNT_ID>:role/spinnaker-assumed"
],
"Effect": "Allow"
}]
}
Review the policy and provide the name ‘spinnakerAssumeRolePolicy’
Create the policy
Replacing
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [ "ec2:*" ],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/spinnaker-launched"
}]
}
Review the policy and provide the name ‘spinnakerPassRolePolicy’
Create the policy
The spinnaker-assumed role must ‘trust’ the spinnaker user.
Edit trust relationship and change it to (replaceing spinnaker-user-arn with your arn):
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": "<spinnaker-user-arn>"
},
"Action": "sts:AssumeRole"
}]
}
Update the Trust relationship
I had success with an r4.large; an m5.xlarge or maybe m5.large should also work well. In the interest of not troubleshooting memory issues I would use something with at least 8GB of RAM.
Once the instance is up, use your desired DNS solution to point some domain at the public IP of the EC2 instance. Be sure to remove this DNS record when you remove the demo server!
Connect to your server via ssh and install jenkins and some “build server” dependencies. I am installing from jenkins-ci.org and also installing the dependencies to support an s3 apt repository (deb-s3). I am moving Jenkins to port 8888 because Spinnaker and Jenkins will both try to use 8080.
wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install openjdk-8-jdk jenkins software-properties-common
sudo sed -i 's/=8080/=8888/g' /etc/default/jenkins
sudo service jenkins restart
sudo apt-add-repository ppa:brightbox/ruby-ng
sudo apt-get update
sudo apt-get install build-essential ruby2.2 ruby2.2-dev zlib1g-dev liblzma-dev
sudo gem install bundler dev-s3
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
I am installing a completely generic Jenkins and I am going to leave it protected only by the security group. Again, this is NOT suitable for production.
http://<spinnaker-instance-uri>:8888
and enter the initialAdminPassword we retreived from the server.The spinnaker-demo project is setup to be fairly generic and functional for simple pipeline demos or to get to a working starting point. Clone that or create a comparable repository to link with Jenkins (via GitHub Branch Source Plugin).
Once you have a repository, proceed to your GitHub access tokens and generate a new token for spinnaker-demo with admin:repo_hook, read:org, and repo:status scopes. Copy that token, and use it to configure your organization in Jenkins.
github-<username>
(or whatever you prefer)github-<username>
to add your token for this job.<the-github-access-token>
and leave all other fields blank./********
credentials option
The scan should succeed and likely kick off a master branch test that will fail as it has no changes.
Proceed to the repository settings/hooks page and add a Webhook pointing at http://<your uri>:8888/github-webhook/
. All the other options can remain at their default.
In order to highlight an enforced control, enable master branch protection and require a successful check before merging to master. If you are using an organization or have multiple collaborators, consider requiring a review as well.
Jenkins will kick of the job that builds our Debian package and attempt to use deb-s3 to store it in an s3 bucket. We must add the bucket name, access key id, and secret access key values to jenkins as Global or domain scoped credentials. I am using Global, because (one more time) this is not suitable for production use. Create 3 Global Credentials in Jenkins of type “Secret text”. Jenkinsfile expects credential ids: jenkins-aws-s3-spinnaker-demo-bucket, jenkins-aws-s3-spinnaker-demo-access-key-id, and jenkins-aws-s3-spinnaker-demo-secret-access-key to be available. Set those accurately, and the deb-s3 upload should successfully post our Debian packages.
At this point, you can checkout a non-master branch, make an update, push the branch to GitHub, and create a PR to validate a few demonstrative concepts:
Halyard manages configuration of Spinnaker, and while you can configure things manually or try to use configuration management I’ve only encountered frustration when trying to do Halyard’s work in another way. My implementations use Halyard and then throw a backup somewhere that we can fetch on launch. It’s not classy, but it works.
I will update the hosts file here, you’ll need to set YOUR_CONFIGURED_URI (from the DNS update above)
export YOUR_CONFIGURED_URI=<your uri>
sudo sed -i "s/localhost/localhost $(hostname) $YOUR_CONFIGURED_URI/" /etc/hosts
curl -O https://raw.githubusercontent.com/spinnaker/halyard/master/install/debian/InstallHalyard.sh
chmod 0750 InstallHalyard.sh
sudo ./InstallHalyard.sh
My Halyard version installed at 1.1.0.
Use hal version list
to show available spinnaker releases and Halyard version compatibility. I will deploy spinnaker at version 1.7.3. I am setting explicit access key id and secret access key values for AWS services instead of using an EC2 instance Role. I experienced what seemed like front50 and/or clouddriver issues when trying to rely on an instance role and have not had time to further troubleshoot. Explicitly setting the keys is either more reliable or a happy coincidence with other updates I made.
I found the Halyard command reference helpful when dealing with these configurations.
One more time, these commands include a few concerning implementations from a security perspective, and this configuration does not include any authentication/authorization mechanisms. Do NOT open the spinnaker security group to unknown IPs or users; Spinnaker has poweruser access to your AWS account.
export SPINNAKER_ACCESS_KEY_ID=<spinnaker user access key id>
export SPINNAKER_SECRET_ACCESS_KEY=<spinnaker user secret access key>
export SPINNAKER_AWS_ACCOUNT_DISPLAY_NAME=<what you want your account to be called in spinnaker, e.g. 'my-aws-account'>
export SPINNAKER_AWS_ACCOUNT_ID=<your aws account id>
export SPINNAKER_URI=<your spinnaker uri>
hal config deploy edit --type localdebian
hal config version edit --version 1.7.3
echo $SPINNAKER_SECRET_ACCESS_KEY | hal config storage s3 edit --region us-west-2 --access-key-id $SPINNAKER_ACCESS_KEY_ID --secret-access-key
hal config storage edit --type s3
echo $SPINNAKER_SECRET_ACCESS_KEY | hal config provider aws edit --access-key-id $SPINNAKER_ACCESS_KEY_ID --secret-access-key
hal config provider aws account add $SPINNAKER_AWS_ACCOUNT_DISPLAY_NAME --account-id $SPINNAKER_AWS_ACCOUNT_ID --regions us-west-2 --assume-role role/spinnaker-assumed
hal config provider aws enable
echo "host: 0.0.0.0" | tee ~/.hal/default/service-settings/gate.yml ~/.hal/default/service-settings/deck.yml
hal config security ui edit --override-base-url http://$SPINNAKER_URI:9000
hal config security api edit --override-base-url http://$SPINNAKER_URI:8084
hal config ci jenkins master add local-jenkins --address http://$SPINNAKER_URI:8888
hal config ci jenkins enable
sudo hal deploy apply
On the first launch you can not flush the infrastructure caches, but if you are making updates and restarting spinnaker, I highly recommend using sudo hal deploy apply --flush-infrastructure-caches
to be sure your cached data is not causing confusion. After launch, give the various services time to start. Nothing should be using much CPU, and you should be able to confirm that ports 9000, 8084, and 8888 are not bound to 127.0.0.1 (netstat -lnt
). Spinnaker logs all route to /var/log/syslog on 16.04 LTS with Spinnaker 1.7.3 by default, hopefully there’s not a lot to troubleshoot, but that’s a good place to start if need be.
Fire up http://$SPINNAKER_URI:9000 and bask in all its DevOpsy radiance.
I create an application with an ELB and a Security Group, and also a deploy pipeline. Our pipeline will be a simple Bake + Deploy, that is triggered by a successful test on our repository’s master branch (which should be the only time a deploy artifact is created).
<some email>
Create two security groups: one for the ELB and one for the EC2 instances.
Note that Spinnaker expects security groups to be present in AWS, even though there is a create Security Group button. Without security groups present in your vpc, the UI will throw errors.
HTTP 80
and Internal HTTP 80
HTTP 80 /health
(assuming you are using the spinnaker-demo repository)
I will create a simple Bake and Deploy pipeline, and we’ll use some custom parameters a trigger our pipeline based on the spinnaker-demo master branch test.
aws_subnet_id
subnet-<id for demo.internal.us-west-2a subnet>
repository
http://<your apt repo bucket>.s3-website-us-west-2.amazonaws.com trusty main
Deploy a few revisions, reconfigure, and demonstrate the Rollback/Enable/Disable features within the Red/Black deploy. A demo like this was a key first step to gaining support for a more fully featured proof of concept.
I tried to be overly opinionated in the interest of getting a functional demo in a reliable way. If you are seeing issues check security groups, hostnames, CORS, and caching. If nothing obvious shows up, check the Spinnakerteam Slack, lots of helpful people are usually around.
Spinnaker Hello Deployment Codelab
Armory Application Deployment Pipeline