Jenkins runner with Terraform

How to use Terraform to provision an ASG of Jenkins runners

Requirements

  • Hosted in AWS, managed by infrastracture as a code.

  • Have option to have different type of runners

  • Scale up and down the runners independently

  • Runners must have installed docker && docker-compose (1.7.1)

Solution

We will manage/provision our AWS resources with Terraform

We will use for the runners labels and for each of the label we provision an Autoscale group (ASG). We do it like that because we can have for example different runners with higher CPU or higher RAM.

One example is to have runners for Selenium testing and runners for unit testing.

==The final module can be found here AWS Jenkins module==

Pre Requirements

You have a working master Jenkins server. You have created credentials with public / private keys for the nodes that will act as slaves, also you have a user account in jenkins that can use the Jenkins remote api through cli.

All the above are out of this scope, and can we written in a follow up post.

How to use this module

Because this is a terraform module you can just reference this into your TF files like this

module "jenkins-runner" {
    source = "bitbucket.org/alexsapran/aws-jenkins-runner"
    ...variables...
}

What is does is just provision an Ubuntu machine (you can alter your prefer version of AMI but this was tested with 16.04 LTS amd64 hvm:ebs-ssd 20160627 ) then downloads the Jenkins cli jar client from the master jenkins and issue an add node command with the selected authenticated user.

Output variables

  • The id of the autoscale group output "autoscalegroup_id" { value = "${aws_autoscaling_group.jenkins.id}" }
  • The generated name of the autoscale group output autoscalegroup_name { value = "${aws_autoscaling_group.jenkins.name}" }

Input variables

  • We define a name for this new ASG runner, this will also be part of the labels variable name { default = "tf" }
  • Define the VPC subnets in which we will add the runners variable vpc_subnets { default = "" }
  • Define the availability zone

    # Define the vpc zones
    variable vpc_zones {
    default = ""
    }
    
  • Define some autoscaling variables for the ASG minimum size, max size and desired capacity. For this we can have an ASG with min 0 and alter to 1 in a predefined 9-5 time window.

    variable min_size {
    default = "1"
    }
    variable max_size {
    default = "2"
    }
    variable capacity {
    default = "1"
    }
    
  • We select what type of instances we want the ASG to provision

    variable instance_type {
    default = "t2.small"
    }
    
  • From which AMI, remember default is 16.04 LTS amd64 hvm:ebs-ssd 20160627

    variable image_id {
    default = "ami-26e70c49"
    }
    
  • If we want the runners to be assigned public IP address, if for example the runners produce an full environment then this should be true but if they just run test that we will not have a public IP to see the runners outcome then we can set to false.

    variable public_ip_address {
    default = "true"
    }
    
  • What security groups we want to assign to the runner machines

    variable security_groups {
    default = ""
    }
    
  • What type of EBS backend root volume we want to have.

    variable volume_size {
    default = "100"
    }
    variable volume_type {
    default = "gp2"
    }
    variable volume_delete {
    default = "true"
    }
    
  • How many executor this runner should have, this is related to the type of the instance, for example bigger instances can host more executors.

    variable worker_executor {
    default = "2"
    }
    
  • The Jenkins credentials that we created in the master

    variable jenkins_credential_id {
    default = "jenkins-runner"
    }
    
  • The Jenkins master url, this should be something like http://jenkins.mycompany.ltd

    variable jenkins_master {
    default = ""
    }
    
  • A Jenkins user that has permission for the remote cli

    variable jenkins_user {
    default = ""
    }
    variable jenkins_pass {
    default = ""
    }
    
  • The master public ssh key that the master will use in order to access the node.

    variable master_ssh_public_key {
    default = ""
    }
    
  • The AWS keypair in order to allow access to the machine

    variable key_name {
    default = ""
    }
    

Example with different ASG’s per label

module "jenkins-utci" {
    source = "bitbucket.org/alexsapran/aws-jenkins-runner"

    name                  = "unit"
    jenkins_label         = "unit,ci"
    vpc_subnets           = "..."
    min_size              = 1
    max_size              = 2
    capacity              = 1
    instance_type         = "t2.medium"
    # Use the default ami from eu-central-1a
    # image_id            = "ami-26e70c49"
    security_groups       = "${aws_security_group.jenkins.id}, ${aws_security_group.some_other.id}"

    jenkins_credential_id = "jenkins-runner"
    jenkins_master        = "http://master-jenkins.mycompany.ltd"
    jenkins_user          = "jenkins-user"
    jenkins_pass          = "awsome_password"
    master_ssh_public_key = "${var.ssh_key}"
}

module "jenkins-qa" {
    source = "bitbucket.org/alexsapran/aws-jenkins-runner"

    name                  = "qa"
    jenkins_label         = "selenium,it"
    vpc_subnets           = "..."
    min_size              = 1
    max_size              = 2
    capacity              = 1
    instance_type         = "t2.medium"
    # Use the default ami from eu-central-1a
    # image_id            = "ami-26e70c49"
    security_groups       = "${aws_security_group.jenkins.id}, ${aws_security_group.some_other.id}"

    jenkins_credential_id = "jenkins-runner"
    jenkins_master        = "http://master-jenkins.mycompany.ltd"
    jenkins_user          = "jenkins-user"
    jenkins_pass          = "awsome_password"
    master_ssh_public_key = "${var.ssh_key}"
}

Alexandros Sapranidis

Software engineer, keen on wearing many hat, current Senior Software Engineer @Elastic cloud

Athens, Greece http://sapranidis.gr