964284 - Get webmaker-suite vagrant instance working with AWS provider

Status: RESOLVED FIXED

(Webmaker Graveyard :: DevOps, defect)

Description

[JP Schneider :jp]

We have a vagrant instance setup for webmaker-suite already, but up until now, we have only focused on allowing individuals to use this vagrant instance on their local server.

One of our other use cases deals with people wanting to show off branches to other people, many of who may be remote and unable to walk over to a desk or screenshare.

The new versions of vagrant allow for AWS as a potential provider, so if we can get that working, it will fit nicely into our workflow and fill in this use case gap.

Hama has a personal free-tier account, and has volunteered to work on this!  Thanks!!

It is likely the deliverable here will be a new Vagrantfile we add to the webmaker-suite/vagrant folder, called Vagrantfile-AWS, as well as relevant documentation added to the readme.

Comment 1

[Dmitry Hamabama Yastremskiy]

Vagrant can work with AWS using amazon-aws plugin (https://github.com/mitchellh/vagrant-aws). This instruction explains a simple setup of vagrant-aws connection:

1. Assuming that you are already have Vagrant installed on your machine, you will need to install the amazon-aws plugin by issuing a command: vagrant plugin install vagrant-aws
2. The next step is to configure Vagrantfile to access AWS. A template of Vagrantfile is attached. To configure the file you will need:
 - to specify a name for the box (the box you will create after you have vagrantfile configured, e.g. aws.box). 
 - assuming that you have an account at AWS, you need to find out the access_key_id and secret_access_key. Those are available by https://portal.aws.amazon.com/gp/aws/securityCredentials? 
 - to specify a type of instance you would like to run ( under Free Tier account for free available only t1.micro )
 - to specify a region and availability_zone for an AMI you are going to use. (e.g. ami-76bfdf46 (Ubuntu 12.04) resides at region US-WEST-2 and availability_zone US-WEST-2c. Can be found under your AWS control panel.
 - since amazon-aws plugin has no implementation of port forwarding yet, you need to create a security group with all the necessary ports opened and specify the name of the group in the vagrantfile.
- and last thing is a path to your keypair xxx.pem .
3. grab the metadata.json file attached and put it to the vagrant folder along with vagrantfile.
4. After you have configured Vagrantfile, you need to create a box. Since you are using AWS, box will not contain a Linux distro, but configuration settings. To create a box you need to issue the command " tar cvzf aws.box ./metadata.json ./Vagrantfile ".
5. to add the box to the list of boxes by issuing command "vagrant box add name_of_box name.box" (e.g. vagrant box aws aws.box) 
6. to run an instance issue " vagrant up --provider=aws" and then to ssh to it "vagrant ssh". To terminate the instance "vagrant destroy".

The extended version of instruction can be found at https://github.com/mitchellh/vagrant-aws

Comment 2

[Dmitry Hamabama Yastremskiy]

Attachment: Vagrantfile template to work with AWS

Comment 3

[Dmitry Hamabama Yastremskiy]

Attachment: metadata.json comes along with vagrantfile to create a box

Comment 4

[JP Schneider :jp]

I was not able to create a proper box through the instructions above.

When I tar the files, then run this bit:
5. to add the box to the list of boxes by issuing command "vagrant box add name_of_box name.box" (e.g. vagrant box aws aws.box) 

ocalmac-11:26:30-aws# vagrant box add aws aws.box
Downloading or copying the box...
Extracting box...e: 0/s, Estimated time remaining: --:--:--)
/Applications/Vagrant/embedded/lib/ruby/1.9.1/json/common.rb:148:in `parse': 746: unexpected token at '{ (JSON::ParserError)
provider: "aws"
}
'
	from /Applications/Vagrant/embedded/lib/ruby/1.9.1/json/common.rb:148:in `parse'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box.rb:50:in `initialize'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box_collection.rb:128:in `new'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box_collection.rb:128:in `block (3 levels) in add'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box_collection.rb:388:in `with_temp_dir'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box_collection.rb:125:in `block (2 levels) in add'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box_collection.rb:388:in `with_temp_dir'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box_collection.rb:109:in `block in add'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box_collection.rb:375:in `block in with_collection_lock'
	from <internal:prelude>:10:in `synchronize'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box_collection.rb:374:in `with_collection_lock'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/box_collection.rb:77:in `add'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/action/builtin/box_add.rb:62:in `call'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/action/warden.rb:34:in `call'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/action/builder.rb:116{
:in `call'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/action/runner.rb:61:in `block in run'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/util/busy.rb:19:in `busy'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/action/runner.rb:61:in `run'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/plugins/commands/box/command/add.rb:37:in `execute'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/plugins/commands/box/command/root.rb:47:in `execute'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/cli.rb:38:in `execute'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/lib/vagrant/environment.rb:478:in `cli'
	from /Applications/Vagrant/embedded/gems/gems/vagrant-1.3.5/bin/vagrant:96:in `<top (required)>'
	from /Applications/Vagrant/bin/../embedded/gems/bin/vagrant:23:in `load'
	from /Applications/Vagrant/bin/../embedded/gems/bin/vagrant:23:in `<main>'


The way I may have found around it came from https://github.com/mitchellh/vagrant-aws, in the quickstart section.

I ran:
$ vagrant box add aws https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
I am now able to vagrant up!

localmac-11:34:29-aws# vagrant up --provider=aws
Bringing machine 'default' up with 'aws' provider...
WARNING: Nokogiri was built against LibXML version 2.8.0, but has dynamically loaded 2.9.1
[fog][WARNING] Unable to load the 'unf' gem. Your AWS strings may not be properly encoded.
[default] Warning! The AWS provider doesn't support any of the Vagrant
high-level network configurations (`config.vm.network`). They
will be silently ignored.
[default] Launching an instance with the following settings...
[default]  -- Type: m1.small
[default]  -- AMI: ami-8f2718e6
[default]  -- Region: us-east-1
[default]  -- Availability Zone: us-east-1c
[default]  -- Keypair: mofo-vagrant-webmakersuite
[default]  -- Security Groups: ["webmakersuite-vagrant-ec2-sg"]
[default]  -- Block Device Mapping: []
[default]  -- Terminate On Shutdown: false
[default]  -- Monitoring: false
[default]  -- EBS optimized: false
[default] Waiting for instance to become "ready"...
[default] Waiting for SSH to become available...
[default] Machine is booted and ready for use!
[default] Rsyncing folder: /Users/John/vagrant/aws/ => /vagrant

Comment 5

[JP Schneider :jp]

We found the problem with the box build, it was quoting around provider in the metadata.json file.

I'll am creating a webmaker-suite AMI, which I will use in a new part of the README and update in Vagrantfile.

We've just merged something that might break sudo, but we'll see.
https://github.com/mozilla/webmaker-suite/pull/16

I'm rebuilding both localdev and aws-provider boxes now.

Comment 6

[JP Schneider :jp]

Excellent work Hama!
I have used your method, and created an AMI with webmaker-suite fully ready to go.  I added the files (including the new Vagrantfile with updated AMI) and instructions here: 

https://github.com/mozilla/webmaker-suite/tree/master/vagrant/aws