1036158 - MSISDN-Gateway unit tests fail on Ubuntu/RHEL - missing fake_dynamo

Status: VERIFIED FIXED

(Cloud Services Graveyard :: MobileID, defect, P1)

Description

[James Bonacci [:jbonacci]]

Standard git clone and install of msisdn-gateway on Ubuntu 14.04.
I start up redis-server and then run this
"npm test"
which fails immediately:

> mozilla-msisdn-gateway@0.4.0-DEV test /home/ubuntu/msisdn-gateway
> make test

/bin/sh: 1: fake_dynamo: not found
make: *** [cover-mocha] Error 127
npm ERR! Test failed.  See above for more details.
npm ERR! not ok code 0

A scrollback of the install step on Ubuntu does not show any errors or unexpected warnings.

In package.json, I do see "dynamo-client": "1.1.5", under the list of dependencies.

A scrollback of the install step on RHEL shows additional errors for bigint.cc:
https://jbonacci.pastebin.mozilla.org/5535169


also, FYI - I have to use "sudo" in order to install this repo on any OS...

Comment 1

[Rémy Hubscher (:natim)]

Yes this is because we want to test the dynamo backend using fake_dynamo.
You can install it by hand before running test or you can use make install to do it.

fake_dynamo is a system dependency needed in order to test the dynamodb backend.

    gem install fake_dynamo

You should be able to install it without root access.

Comment 2

[James Bonacci [:jbonacci]]

This will have to wait.
Any call to gem install fake_dynamo tries to access this URL:
https://api.rubygems.org/api/v1/dependencies?gems=activesupport,json,sinatra

which is currently throwing 503s!

Comment 3

[James Bonacci [:jbonacci]]

$ sudo gem install fake_dynamo
ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
    bad response Service Unavailable 503 (https://api.rubygems.org/api/v1/dependencies?gems=activesupport,json,sinatra)

"Application Error
An error occurred in the application and your page could not be served. Please try again in a few moments.
If you are the application owner, check your logs for details."

Fun.

Comment 4

[James Bonacci [:jbonacci]]

So, no, this trick does not work.
I am unable to get fake-dynamo installed on Ubuntu and RHEL

Ubuntu:
$ gem install fake_dynamo
Fetching: rack-1.5.2.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /usr/lib/ruby/gems/1.9.1 directory.

$ sudo gem install fake_dynamo
Fetching: rack-1.5.2.gem (100%)
Successfully installed rack-1.5.2
Fetching: tilt-1.4.1.gem (100%)
Successfully installed tilt-1.4.1
Fetching: rack-protection-1.5.3.gem (100%)
Successfully installed rack-protection-1.5.3
Fetching: sinatra-1.4.5.gem (100%)
Successfully installed sinatra-1.4.5
Fetching: i18n-0.6.11.gem (100%)
Successfully installed i18n-0.6.11
Fetching: json-1.8.1.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing fake_dynamo:
	ERROR: Failed to build gem native extension.

    /usr/bin/ruby1.9.1 extconf.rb
/usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- mkmf (LoadError)
	from /usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require'
	from extconf.rb:1:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /usr/lib/ruby/gems/1.9.1/gems/json-1.8.1 for inspection.
Results logged to /usr/lib/ruby/gems/1.9.1/extensions/x86_64-linux/1.9.1/json-1.8.1/gem_make.out

Comment 5

[James Bonacci [:jbonacci]]

Same error as above for RHEL 6.5.


So, I am limited to running msisdn-gateway unit tests on Mac only right now...

Comment 6

[Rémy Hubscher (:natim)]

Attachment: Link to github PR

Comment 7

[Rémy Hubscher (:natim)]

In order to install fake_dynamo you will need to install ruby-dev package as well.

Comment 8

[Rémy Hubscher (:natim)]

https://github.com/mozilla-services/msisdn-gateway/commit/0c478b4e3c1f618ce5ad5a2bba800a8186dbb0e2

Comment 9

[James Bonacci [:jbonacci]]

Interesting fix. I will look at this on Monday.

Comment 10

[James Bonacci [:jbonacci]]

Grabbed the latest master with the vary same result:
$ make test
which: no fake_dynamo in (/home/ec2-user/.gem/ruby/*/bin)
which: no fake_dynamo in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ec2/tools/bin:/opt/aws/bin:/home/ec2-user/bin)
/bin/sh: --db: command not found
make: *** [cover-mocha] Error 127

Comment 11

[James Bonacci [:jbonacci]]

Slightly different for Ubuntu:

$ npm test

> mozilla-msisdn-gateway@0.5.0-DEV test /home/ubuntu/msisdn-gateway
> make test

/bin/sh: 1: --db: not found
make: *** [cover-mocha] Error 127
npm ERR! Test failed.  See above for more details.
npm ERR! not ok code 0

Comment 12

[Rémy Hubscher (:natim)]

Where did you install fake_dynamo?

Comment 13

[James Bonacci [:jbonacci]]

I did not install it manually. It appeared to me that it was now part of the build process? Is that not the case? Perhaps I misinterpreted the console log after the error happened?...

If this is a mandatory manual step, then let's document it clearly with instructions:
Where to get it
How to install it 
Where to install it

Comment 14

[Rémy Hubscher (:natim)]

make install should install it if not existing.

Comment 15

[James Bonacci [:jbonacci]]

On Ubuntu, a quick system-wide search for fake_dynamo shows this:
./home/ubuntu/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/fake_dynamo-XXXX.gemspec
where XXX means various versions

Also: /usr/bin/gem


On RHEL 6.5:
./home/ec2-user/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/fake_dynamo-0.2.5.gemspec
and
./root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/fake_dynamo-XXX.gemspec

Also: /usr/bin/gem


On RHEL 6.4:
./root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/fake_dynamo-0.2.5.gemspec

Also: /usr/bin/gem


So where is "make install" looking? locally (in user's msisdn-gateway repo) or at system-level or both?

Comment 16

[Rémy Hubscher (:natim)]

You should have either fake_dynamo in your PATH or in `~/.gem/ruby/*/bin/fake_dynamo`

Comment 17

[Rémy Hubscher (:natim)]

What does `gem list fake_dynamo` tells you?

Comment 18

[James Bonacci [:jbonacci]]

This is starting to sound like a chicken and egg problem!
Assuming fake_dynamo does not already exist, that means we would need 'make install' to update the PATH or let the user know to do so.
If fake_dynamo already exists (from a previous install of msisdn-gateway or from some other source), then 'make install' would still need to update the PATH or let the user know to do so.

As a minimum, we need to add more steps here: https://github.com/mozilla-services/msisdn-gateway#how-to-install, similar to what we had to do for redis-server. We certainly need to let the user know to have gem and ruby-dev installed before running 'make install' (unless we want to add those to 'make install' as well).


But, it sounds like there is a bigger problem in that either
1. 'make install' is not installing fake_dynamo at all or not correctly
2. 'make install' is installing fake_dynamo but not updating the steps
3. 'make install' finds fake_dynamo but does not use it 

From / or $home, 'gem list fake_dynamo' returns the following on all linux flavors:
*** LOCAL GEMS ***

(empty list)

Comment 19

[Rémy Hubscher (:natim)]

and what is the return code of gem list fake_dynamo?

    echo $? should be different from 0

Comment 20

[James Bonacci [:jbonacci]]

On all 3 systems:
$ gem list fake_dynamo
$ echo $?
returns 0

Comment 21

[Rémy Hubscher (:natim)]

Ok so there is the problem. When we ask if fake_dynamo is installed it tells yes so that it is not reinstalled.

Comment 22

[Rémy Hubscher (:natim)]

Can you try again with my fix: https://github.com/mozilla-services/msisdn-gateway/commit/802c122430cbb8b740d6bb6b4b7b757440af6952

It is last master.

Comment 23

[Tarek Ziadé (:tarek)]

Are we good here ?

Comment 24

[James Bonacci [:jbonacci]]

Not sure yet. Will try this today.
- all those pesky deployments delaying my testing ;-)

Comment 25

[James Bonacci [:jbonacci]]

No not fixed.

Basically, I can not install the this repo on Mac unless I use "sudo".
I should not have to do that.

...etc...
npm ERR! Error: EACCES, open '/Users/James/.npm/i18n-abide/0.0.20/package.tgz'
npm ERR!  { [Error: EACCES, open '/Users/James/.npm/i18n-abide/0.0.20/package.tgz']
npm ERR!   errno: 3,
npm ERR!   code: 'EACCES',
npm ERR!   path: '/Users/James/.npm/i18n-abide/0.0.20/package.tgz',
npm ERR!   parent: 'mozilla-msisdn-gateway' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.
...etc...


On Linux, I don't even get that far.
This installer assumes that ruby and gem are already installed.
I have found no good way to install these as a linux user, only as root.
But if I install them as root, then I can't install this repo.




Here is how it looks:
$ make install
which: no fake_dynamo in (/home/ec2-user/.gem/ruby/*/bin)
which: no fake_dynamo in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ec2/tools/bin:/opt/aws/bin:/home/ec2-user/bin)
Installing fake_dynamo...
WARNING:  You don't have /home/ec2-user/.gem/ruby/1.8/bin in your PATH,
	  gem executables will not run.
Building native extensions.  This could take a while...
ERROR:  Error installing fake_dynamo:
	ERROR: Failed to build gem native extension.

    /usr/bin/ruby extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /home/ec2-user/.gem/ruby/1.8/gems/json-1.8.1 for inspection.
Results logged to /home/ec2-user/.gem/ruby/1.8/extensions/x86_64-linux/1.8/json-1.8.1/gem_make.out
make: *** [install] Error 1


So, now I add the path as user ec2-user:
$ export PATH=$PATH:/home/ec2-user/.gem/ruby/1.8/bin
and try again
$ make install
which: no fake_dynamo in (/home/ec2-user/.gem/ruby/*/bin)
which: no fake_dynamo in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ec2/tools/bin:/opt/aws/bin:/home/ec2-user/bin:/home/ec2-user/.gem/ruby/1.8/bin)
Installing fake_dynamo...
Building native extensions.  This could take a while...
ERROR:  Error installing fake_dynamo:
	ERROR: Failed to build gem native extension.

    /usr/bin/ruby extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /home/ec2-user/.gem/ruby/1.8/gems/json-1.8.1 for inspection.
Results logged to /home/ec2-user/.gem/ruby/1.8/extensions/x86_64-linux/1.8/json-1.8.1/gem_make.out
make: *** [install] Error 1


$ which ruby
/usr/bin/ruby

$ cd /
$ sudo find . -name ruby.h -print
returns nothing

even though ruby is installed on all these AWS instances



Now, if I turn this around and 
1. Set my path (as local user - ubuntu or ec2-user)
2. Run "sudo make install"
3. I get the path error again, because although the path was set as the user (default for these AWS instances), I am running make as 'root', which of course, does not have that local path set.

I have not found any success after installing gem and ruby locally.

The biggest blocker for me on this repo is the requirement to use "sudo" to install everything.

I can try this on a Linux VM where I have root access, but basically, I am stuck...

Comment 26

[James Bonacci [:jbonacci]]

Forgot to add that gem and ruby are installed on all AWS instances:
/usr/bin/gem
/usr/bin/ruby


I have the rubygems code in each of the user folders.
I installed as follows:
$ cd rubygems*
$ ruby setup.rb

Anyway, then I back up one level and delete the .gem directory
Then
$ cd msisdn-gateway
$ npm install (instead of 'make install')
This gets me a lot farther along. So either npm/make or the deletion of .gem directory is helping.

On rhel linux, I appear to get a complete build. But:
start up redis-server
run npm test
get the same error again!
which: no fake_dynamo in (/home/ec2-user/.gem/ruby/*/bin)
which: no fake_dynamo in (/usr/lib/node_modules/npm/bin/node-gyp-bin:/home/ec2-user/msisdn-gateway/node_modules/.bin:/usr/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ec2/tools/bin:/opt/aws/bin:/home/ec2-user/bin)
/bin/sh: --db: command not found
make: *** [cover-mocha] Error 127
npm ERR! weird error 2
npm ERR! not ok code 0



On Ubuntu, I still get this:
...etc...
npm ERR! git fetch -a origin (git://github.com/mozilla-services/express-logging.git) error: cannot open FETCH_HEAD: Permission denied
npm http 304 https://registry.npmjs.org/raven/0.6.2
npm ERR! Error: Command failed: error: cannot open FETCH_HEAD: Permission denied
npm ERR! 
npm ERR! 
npm ERR!     at ChildProcess.exithandler (child_process.js:637:15)
npm ERR!     at ChildProcess.EventEmitter.emit (events.js:98:17)
npm ERR!     at maybeClose (child_process.js:735:16)
npm ERR!     at Socket.<anonymous> (child_process.js:948:11)
npm ERR!     at Socket.EventEmitter.emit (events.js:95:17)
npm ERR!     at Pipe.close (net.js:466:12)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <[e-mail]>

npm ERR! System Linux 3.13.0-24-generic
npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install"
npm ERR! cwd /home/ubuntu/msisdn-gateway
npm ERR! node -v v0.10.22
npm ERR! npm -v 1.3.14
npm ERR! code 1
...etc...

$ sudo npm install
This completes the build.
And, right back to square one with Ubuntu also:

$ make test
/bin/sh: 1: --db: not found
make: *** [cover-mocha] Error 127

$ npm test

> mozilla-msisdn-gateway@0.5.0-DEV test /home/ubuntu/msisdn-gateway
> make test

/bin/sh: 1: --db: not found
make: *** [cover-mocha] Error 127
npm ERR! Test failed.  See above for more details.
npm ERR! not ok code 0

Comment 27

[James Bonacci [:jbonacci]]

Testing with my SciLinux 6.4 VM...
Definitely requires an install of Ruby, which is required to install Gem
$ su
$ yum -y install ruby
$ yum -y install rubygems

$ cd msisdn-gateway
$ make install

shows this:

which: no fake_dynamo in (/home/mozilla/.gem/ruby/*/bin)
which: no fake_dynamo in (/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/mozilla/bin)
Installing fake_dynamo...
WARNING:  You don't have /home/mozilla/.gem/ruby/1.8/bin in your PATH,
	  gem executables will not run.
Building native extensions.  This could take a while...
ERROR:  Error installing fake_dynamo:
	ERROR: Failed to build gem native extension.

/usr/bin/ruby extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/ruby.h

Gem files will remain installed in /home/mozilla/.gem/ruby/1.8/gems/json-1.8.1 for inspection.
Results logged to /home/mozilla/.gem/ruby/1.8/gems/json-1.8.1/ext/json/ext/generator/gem_make.out
make: *** [install] Error 1


So, I set the path then
$ make install

which, of course, gets me right back here

which: no fake_dynamo in (/home/mozilla/.gem/ruby/*/bin)
which: no fake_dynamo in (/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/mozilla/bin:/home/mozilla/.gem/ruby/1.8/bin)
Installing fake_dynamo...
Building native extensions.  This could take a while...
ERROR:  Error installing fake_dynamo:
	ERROR: Failed to build gem native extension.

/usr/bin/ruby extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/ruby.h


Gem files will remain installed in /home/mozilla/.gem/ruby/1.8/gems/json-1.8.1 for inspection.
Results logged to /home/mozilla/.gem/ruby/1.8/gems/json-1.8.1/ext/json/ext/generator/gem_make.out
make: *** [install] Error 1


So, it's not AWS specifically.

Comment 28

[Rémy Hubscher (:natim)]

As you can see this is related to your ruby installation.
As explained in Comment 7: https://bugzilla.mozilla.org/show_bug.cgi?id=1036158#c7

You must install ruby-dev package in order to be able to build gem native extension.

Comment 29

[James Bonacci [:jbonacci]]

Aha! ok.
That's next. I expect that will clean this all up and I can close out the bug.

Comment 30

[Rémy Hubscher (:natim)]

Well fake_dynamo is not part of the msisdn project, it is just a mock for the AWS DynamoDB service.
I suppose we shouldn't bother to much about it because it is a Developer thing to be able to run tests.

Comment 31

[James Bonacci [:jbonacci]]

Yep. That works for Ubuntu. I can now 'sudo make install' and 'make test' the repo (same as for Mac).
On rhel, it's a little more complicated. A 'sudo yum install ruby' gets me version 1.8.7. But we need a newer version:
ERROR:  Error installing fake_dynamo:
	activesupport requires Ruby version >= 1.9.3.
make: *** [install] Error 1

Working on it...

Comment 32

[James Bonacci [:jbonacci]]

After all that work on a fairly clean RHEL 6.5, I still see this:

$ sudo make install
which: no fake_dynamo in (/root/.gem/ruby/*/bin)
which: no fake_dynamo in (/sbin:/bin:/usr/sbin:/usr/bin)
Installing fake_dynamo...
/usr/lib/ruby/1.9.1/yaml.rb:84:in `<top (required)>':
It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
ERROR:  Loading command: install (LoadError)
	cannot load such file -- zlib
ERROR:  While executing gem ... (NoMethodError)
    undefined method `invoke_with_build_args' for nil:NilClass
make: *** [install] Error 1

Comment 33

[James Bonacci [:jbonacci]]

Also, for RHEL 6.5, according to the system, I do have the latest libyaml...


On a special, clean version of RHEL 6.4 (SciLinux), I see this after installing ruby 1.9.3, ruby-devel, and gem:
$ sudo make install
which: no fake_dynamo in (/sbin:/bin:/usr/sbin:/usr/bin)

but then it builds successfully.

Then, of course, this happens:
$ make test
which: no fake_dynamo in (/home/ec2-user/.gem/ruby/*/bin)
which: no fake_dynamo in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ec2/tools/bin:/opt/aws/bin:/home/ec2-user/bin)
/bin/sh: --db: command not found
make: *** [cover-mocha] Error 127


So, I pop up a level and run this:
sudo gem install fake_dynamo
which gives me a warning similar to RHEL 6.5:
/usr/lib/ruby/1.9.1/yaml.rb:84:in `<top (required)>':
It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
Fetching: fake_dynamo-0.2.5.gem (100%)
Successfully installed fake_dynamo-0.2.5
Installing ri documentation for fake_dynamo-0.2.5
1 gem installed

but it does install

back to msisdn-gateway:
sudo make clean
sudo make install
make test

This also fails:
which: no fake_dynamo in (/home/ec2-user/.gem/ruby/*/bin)
/usr/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:313:in `to_specs': Could not find 'sinatra' (>= 0) among 7 total gem(s) (Gem::LoadError)
Checked in 'GEM_PATH=/home/ec2-user/.gem/ruby/1.9.1:/usr/lib/ruby/gems/1.9.1', execute `gem env` for more information
	from /usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1313:in `block in activate_dependencies'
	from /usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1302:in `each'
	from /usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1302:in `activate_dependencies'
	from /usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1284:in `activate'
	from /usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_gem.rb:60:in `block in gem'
	from <internal:prelude>:10:in `synchronize'
	from /usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_gem.rb:59:in `gem'
	from /usr/bin/fake_dynamo:22:in `<main>'
make: *** [cover-mocha] Error 1


$ gem env
/usr/lib/ruby/1.9.1/yaml.rb:84:in `<top (required)>':
It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
RubyGems Environment:
  - RUBYGEMS VERSION: 2.4.1
  - RUBY VERSION: 1.9.3 (2014-05-14 patchlevel 547) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/lib/ruby/gems/1.9.1
  - RUBY EXECUTABLE: /usr/bin/ruby
  - EXECUTABLE DIRECTORY: /usr/bin
  - SPEC CACHE DIRECTORY: /home/ec2-user/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /usr/etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /usr/lib/ruby/gems/1.9.1
     - /home/ec2-user/.gem/ruby/1.9.1
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/bin
     - /bin
     - /usr/bin
     - /usr/local/sbin
     - /usr/sbin
     - /sbin
     - /opt/ec2/tools/bin
     - /opt/aws/bin
     - /home/ec2-user/bin



SUMMARY:
I can get msisdn-gateway install and tests to work on Mac if I install with 'sudo'.
I can get msisdn-gateway install and tests to work on Ubuntu if I first install ruby AND ruby-dev AND gem then install msisdn-gateway with 'sudo'.
I can not get the correct sequence on RHEL to install and test.
This would be helpful but not required.

All this needs to be documented. I will open a proper ticket for that.