Tuesday, February 24, 2015

Vagrant CoreOS EC2 and VirtualBox

Okay in a previous post I had been looking at Vagrant. I have a Vagrant file I liked to bring up ubuntu on either VirtualBox or AWS. But my target isn't really ubuntu (well not on the metal anyway.) I was looking to install CoreOS and then I will layer ubuntu on top of it. So how to get Vagrant to provision a CoreOS box. So lets see what Dr. Google says. https://coreos.com/docs/running-coreos/platforms/vagrant/

Hokay looks like they want me to get a Vagrant file.


git clone https://github.com/coreos/coreos-vagrant.git
cd coreos-vagrant
Let's take a peek at what we have.


$ ls -la
total 80
drwxr-xr-x  11 jahnke  staff   374 Feb 23 20:25 ./
drwxr-xr-x   3 jahnke  staff   102 Feb 23 20:25 ../
drwxr-xr-x  13 jahnke  staff   442 Feb 24 20:27 .git/
-rw-r--r--   1 jahnke  staff   117 Feb 23 20:25 .gitattributes
-rw-r--r--   1 jahnke  staff    35 Feb 23 20:25 .gitignore
-rw-r--r--   1 jahnke  staff  2448 Feb 23 20:25 CONTRIBUTING.md
-rw-r--r--   1 jahnke  staff   104 Feb 23 20:25 MAINTAINERS
-rw-r--r--   1 jahnke  staff  4240 Feb 23 20:25 README.md
-rw-r--r--   1 jahnke  staff  4355 Feb 23 20:25 Vagrantfile
-rw-r--r--   1 jahnke  staff  2353 Feb 23 20:25 config.rb.sample
-rw-r--r--   1 jahnke  staff   722 Feb 23 20:25 user-data.sample

Interesting... There is a Vagrantfile and 2 'other' files. So lets peek in the Vagrant file first. Whoa, it is 135 lines long. Holy crap let me dig into this.


require 'fileutils'

Vagrant.require_version ">= 1.6.0"

CLOUD_CONFIG_PATH = File.join(File.dirname(__FILE__), "user-data")
CONFIG = File.join(File.dirname(__FILE__), "config.rb")

# Defaults for config options defined in CONFIG
$num_instances = 1
$instance_name_prefix = "core"
$update_channel = "alpha"
$enable_serial_logging = false
$share_home = false
$vm_gui = false
$vm_memory = 1024
$vm_cpus = 1
$shared_folders = {}

This first bit defines the two 'config' files we saw. As well as sets the default vars. These values can be overridden in 'config.rb' file. So I only need one Vagrant file that can be reused by changing the config.rb file. I mean the Vagrant file exists so I can have a Vagrant file per install why this extra flexibility is needed is beyond me. But what do I know? what I will do is change the user-data and config.rb to 'vagrant-user-data' and 'vagrant-config.rb' so that it is more obvious why these files are hanging out in my app dir.

Okay next bit.


# Attempt to apply the deprecated environment variable NUM_INSTANCES to
# $num_instances while allowing config.rb to override it
if ENV["NUM_INSTANCES"].to_i > 0 && ENV["NUM_INSTANCES"]
  $num_instances = ENV["NUM_INSTANCES"].to_i
end

if File.exist?(CONFIG)
  require CONFIG
end

# Use old vb_xxx config variables when set
def vm_gui
  $vb_gui.nil? ? $vm_gui : $vb_gui
end

def vm_memory
  $vb_memory.nil? ? $vm_memory : $vb_memory
end

def vm_cpus
  $vb_cpus.nil? ? $vm_cpus : $vb_cpus
end

It looks like at some point in the past there was another way to do this. So some people might be setting NUM_INSTANCES as a shell var. And other configs might have vb_ (for virtual box) vars. The only bit of this that I will keep is thing that loads in the CONFIG file. The rest isn't needed (although we have to make a small change later in the script to accommodate the vm_ vars.


Vagrant.configure("2") do |config|
  # always use Vagrants insecure key
  config.ssh.insert_key = false

  config.vm.box = "coreos-%s" % $update_channel
  config.vm.box_version = ">= 308.0.1"
  config.vm.box_url = "http://%s.release.core-os.net/amd64-usr/current/coreos_production_vagrant.json" % $update_channel

  ["vmware_fusion", "vmware_workstation"].each do |vmware|
    config.vm.provider vmware do |v, override|
      override.vm.box_url = "http://%s.release.core-os.net/amd64-usr/current/coreos_production_vagrant_vmware_fusion.json" % $update_channel
    end
  end

  config.vm.provider :virtualbox do |v|
    # On VirtualBox, we don't have guest additions or a functional vboxsf
    # in CoreOS, so tell Vagrant that so it can be smarter.
    v.check_guest_additions = false
    v.functional_vboxsf     = false
  end

  # plugin conflict
  if Vagrant.has_plugin?("vagrant-vbguest") then
    config.vbguest.auto_update = false
  end
This is looking pretty normal for a Vagrant file (well the ones I have seen at any rate.) We are setting the box name and some other info that will be used to grab the correct version of the box. Then some specific settings for both vmware as well as virtual box. And finally disable a conflicting plugin. Since I don't run vmware on my box, and I looked at the vaguest plugin and it seemed kind of out of date and not at all interesting to me I won't be using it either. Both of these segments can go.

This next bit was a little interesting.


  (1..$num_instances).each do |i|
    config.vm.define vm_name = "%s-%02d" % [$instance_name_prefix, i] do |config|
      config.vm.hostname = vm_name
 
What appears to be happening here is the Vagrant file is making different config sets for each machine. In the config.rb file you can define how many you want to start. And this is creating a config for each and naming it by adding a 2 digit number to the end of the name. Sweet...


      if $enable_serial_logging
        logdir = File.join(File.dirname(__FILE__), "log")
        FileUtils.mkdir_p(logdir)

        serialFile = File.join(logdir, "%s-serial.txt" % vm_name)
        FileUtils.touch(serialFile)

        ["vmware_fusion", "vmware_workstation"].each do |vmware|
          config.vm.provider vmware do |v, override|
            v.vmx["serial0.present"] = "TRUE"
            v.vmx["serial0.fileType"] = "file"
            v.vmx["serial0.fileName"] = serialFile
            v.vmx["serial0.tryNoRxLoss"] = "FALSE"
          end
        end

        config.vm.provider :virtualbox do |vb, override|
          vb.customize ["modifyvm", :id, "--uart1", "0x3F8", "4"]
          vb.customize ["modifyvm", :id, "--uartmode1", serialFile]
        end
      end

This bit turns on the serial logging for the VM. Once again I don't use VM Ware, I can't use it for AWS and it looks like it is expensive for virtual box. I could probably do without any of it. But I will leave just the non VM ware bits in place.


      if $expose_docker_tcp
        config.vm.network "forwarded_port", guest: 2375, host: ($expose_docker_tcp + i - 1), auto_correct: true
      end

This isn't useful for AWS either, but it will work for VirtualBox.


      ["vmware_fusion", "vmware_workstation"].each do |vmware|
        config.vm.provider vmware do |v|
          v.gui = vm_gui
          v.vmx['memsize'] = vm_memory
          v.vmx['numvcpus'] = vm_cpus
        end
      end

      config.vm.provider :virtualbox do |vb|
        vb.gui = vm_gui
        vb.memory = vm_memory
        vb.cpus = vm_cpus
      end

So remember the bits up top I was thinking about getting rid of? They will have to be fixed here. Adding a $ in front of vm_ will fix them. And get rid of the vmware bits.


      ip = "172.17.8.#{i+100}"
      config.vm.network :private_network, ip: ip
Since we are starting more than one box, this will increment the VirtualBox ip addresses, AWS will of course ignore this totally.

Finally


      # Uncomment below to enable NFS for sharing the host machine into the coreos-vagrant VM.
      #config.vm.synced_folder ".", "/home/core/share", id: "core", :nfs => true, :mount_options => ['nolock,vers=3,udp']
      $shared_folders.each_with_index do |(host_folder, guest_folder), index|
        config.vm.synced_folder host_folder.to_s, guest_folder.to_s, id: "core-share%02d" % index, nfs: true, mount_options: ['nolock,vers=3,udp']
      end

      if $share_home
        config.vm.synced_folder ENV['HOME'], ENV['HOME'], id: "home", :nfs => true, :mount_options => ['nolock,vers=3,udp']
      end

      if File.exist?(CLOUD_CONFIG_PATH)
        config.vm.provision :file, :source => "#{CLOUD_CONFIG_PATH}", :destination => "/tmp/vagrantfile-user-data"
        config.vm.provision :shell, :inline => "mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/", :privileged => true
      end

    end
  end
end
The rest of this is to deal with sharing the file systems between VirtualBox and the host system. I don't actually use this right now. But I will leave it in place incase I find a compelling reason to use it.

If you remember earlier I needed to set some values to make AWS work correctly. As it turns out the config bits are a little bit tricky if I set the config.vm.box in one place it won't be able to be reset in a different place. And both the VirtualBox and AWS use different boxes. So some reading up on how the Ruby do iterator was working led me to the following answer.


  config.vm.provider :virtualbox do |v, override|
    override.vm.box = "coreos-%s" % $update_channel
    override.vm.box_version = ">= 308.0.1"
    override.vm.box_url = "http://%s.release.core-os.net/amd64-usr/current/coreos_production_vagrant.json" % $update_channel
    # On VirtualBox, we don't have guest additions or a functional vboxsf
    # in CoreOS, so tell Vagrant that so it can be smarter.
    v.check_guest_additions = false
    v.functional_vboxsf     = false
  end

  config.vm.provider :aws do |aws, override|
      aws.access_key_id = ENV['AWS_ACCESS_KEY_ID']
      aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
      aws.keypair_name = "jahnke-gsg-keypair"
      aws.ami = "ami-8097d4e8"
      aws.region = "us-east-1"
      aws.instance_type = "t1.micro"

      override.vm.box = "dummy"
      override.ssh.username = "core"
      override.ssh.private_key_path = "/Users/jahnke/ec2-keys/jahnke-gsg-keypair.pem"
  end
This way neither config sets it directly they set via the override. And so there won't be a conflict. I statically chose the stable coreOS I need to add a conditional here to choose the right AMI based on the update_channel var. And instead of using ubuntu as the base identity CoreOS uses core. This small addition now lets me start with either VirtualBox or AWS.


require 'fileutils'

Vagrant.require_version ">= 1.6.0"

CLOUD_CONFIG_PATH = File.join(File.dirname(__FILE__), "vagrant-user-data")
CONFIG = File.join(File.dirname(__FILE__), "vagrant-config.rb")

# Defaults for config options defined in CONFIG
$num_instances = 1
$instance_name_prefix = "core"
$update_channel = "alpha"
$enable_serial_logging = false
$share_home = false
$vm_gui = false
$vm_memory = 1024
$vm_cpus = 1
$shared_folders = {}

if File.exist?(CONFIG)
  require CONFIG
end

Vagrant.configure("2") do |config|
  # always use Vagrants insecure key
  config.ssh.insert_key = false

  config.vm.provider :virtualbox do |v, override|
    override.vm.box = "coreos-%s" % $update_channel
    override.vm.box_version = ">= 308.0.1"
    override.vm.box_url = "http://%s.release.core-os.net/amd64-usr/current/coreos_production_vagrant.json" % $update_channel
    # On VirtualBox, we don't have guest additions or a functional vboxsf
    # in CoreOS, so tell Vagrant that so it can be smarter.
    v.check_guest_additions = false
    v.functional_vboxsf     = false
  end

  config.vm.provider :aws do |aws, override|
      aws.access_key_id = ENV['AWS_ACCESS_KEY_ID']
      aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
      aws.keypair_name = "jahnke-gsg-keypair"
      aws.ami = "ami-8097d4e8"
      aws.region = "us-east-1"
      aws.instance_type = "t1.micro"

      override.vm.box = "dummy"
      override.ssh.username = "core"
      override.ssh.private_key_path = "/Users/jahnke/ec2-keys/jahnke-gsg-keypair.pem"
  end

  (1..$num_instances).each do |i|
    config.vm.define vm_name = "%s-%02d" % [$instance_name_prefix, i] do |config|
      config.vm.hostname = vm_name

      if $enable_serial_logging
        logdir = File.join(File.dirname(__FILE__), "log")
        FileUtils.mkdir_p(logdir)

        serialFile = File.join(logdir, "%s-serial.txt" % vm_name)
        FileUtils.touch(serialFile)

        config.vm.provider :virtualbox do |vb, override|
          vb.customize ["modifyvm", :id, "--uart1", "0x3F8", "4"]
          vb.customize ["modifyvm", :id, "--uartmode1", serialFile]
        end
      end

      if $expose_docker_tcp
        config.vm.network "forwarded_port", guest: 2375, host: ($expose_docker_tcp + i - 1), auto_correct: true
      end

      ip = "172.17.8.#{i+100}"
      config.vm.network :private_network, ip: ip

      config.vm.provider :virtualbox do |vb|
        vb.gui = $vm_gui
        vb.memory = $vm_memory
        vb.cpus = $vm_cpus
      end

      # Uncomment below to enable NFS for sharing the host machine into the coreos-vagrant VM.
      #config.vm.synced_folder ".", "/home/core/share", id: "core", :nfs => true, :mount_options => ['nolock,vers=3,udp']
      $shared_folders.each_with_index do |(host_folder, guest_folder), index|
        config.vm.synced_folder host_folder.to_s, guest_folder.to_s, id: "core-share%02d" % index, nfs: true, mount_options: ['nolock,vers=3,udp']
      end

      if $share_home
        config.vm.synced_folder ENV['HOME'], ENV['HOME'], id: "home", :nfs => true, :mount_options => ['nolock,vers=3,udp']
      end

      if File.exist?(CLOUD_CONFIG_PATH)
        config.vm.provision :file, :source => "#{CLOUD_CONFIG_PATH}", :destination => "/tmp/vagrantfile-user-data"
        config.vm.provision :shell, :inline => "mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/", :privileged => true
      end

    end
  end
end

No comments:

Post a Comment