999609 - Upgrade NodeJS for developer.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Comment 1

[David Walsh :davidwalsh]

We currently have an ancient NodeJS install on developer.mozilla.org, as well as developer.allizom.org (staging).  We'd like that upgraded so that we may use/install a few NodeJS packages to improve site performance.  Current version is v0.10.26 so as close as we can get to that will be good.  CC'ing Les Orchard to add detail as needed.

Comment 2

[Les Orchard [:lorchard]]

Paired with this, we'll need to upgrade KumaScript with bugfixes that work with node v0.10.x

    https://github.com/mozilla/kumascript/commit/6429d5c7bdc9780408b600fe6712c5a36dddfc2c

Upgrading the version of KumaScript involves updating a git submodule reference - that's something an MDN dev can handle. 

But, this kumascript upgrade in turn requires a new version of a compiled node module, named fibers - we have v0.6.5 and will need v1.0.1

On my laptop, I'd do it with `npm install -g fibers==1.0.1`. But, I don't think that's acceptable for our dev/stage/prod servers. However, I don't know how we got that node fibers initially module compiled & installed on production. Need some IT help in understanding how that's done & getting it done.

Comment 3

[Les Orchard [:lorchard]]

(In reply to David Walsh :davidwalsh from comment #1)
> We currently have an ancient NodeJS install on developer.mozilla.org, as
> well as developer.allizom.org (staging).

Oh yeah, and don't forget developer-dev.allizom.org

Comment 4

[Chris Turra [:cturra]]

looks like nodejs is version pinned to 0.6.15-1.el6. i will update that, started in dev, to the latest available from epel, which is 0.10.26-1.el6. stay tuned for an update.

Comment 5

[David Walsh :davidwalsh]

Thank you Chris!

Comment 6

[Chris Turra [:cturra]]

developer-dev has been updated to latest for testing. please let me know when you're happy with it and i can bump the version in stage next.

 [[e-mail]3 ~]$ node --version
 v0.10.26

Comment 7

[Les Orchard [:lorchard]]

(In reply to Chris Turra [:cturra] from comment #6)
> developer-dev has been updated to latest for testing. please let me know
> when you're happy with it and i can bump the version in stage next.
> 
>  [[e-mail]3 ~]$ node --version
>  v0.10.26

KumaScript will be broken on -dev when it next restarts. Before moving onto staging, we need to get the fibers module upgraded, too. But, I don't know how we got it installed in the first place.

Comment 8

[David Walsh :davidwalsh]

Could you check out Les' comment, Chris?

Comment 9

[Chris Turra [:cturra]]

i have packaged the latest fibers module and installed it in dev, but kumascript is failing to startup with the following error:

module.js:340
    throw err;
          ^
Error: Cannot find module 'fibers'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/data/www/developer-dev.allizom.org/kuma/kumascript/lib/kumascript/templates.js:13:14)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:31


fibers was unpackaged in /usr/lib/node_nodules/ which appears consistent with production.

[[e-mail]3 fibers]# ls /usr/lib/node_modules/fibers/
bin  binding.gyp  build.js  fibers.js  future.js  LICENSE  package.json  quick-test.js  README.md  src  test  test.js


:lorchard - do you know if this should this be installed in a location other than /usr/lib/node_modules/ ?

[[e-mail]3 fibers]# yum list installed | grep fibers
nodejs-fibers.x86_64              1.0.1-1                  @mozilla

Comment 10

[Chris Turra [:cturra]]

a quick follow up, i can see that supervisord set the NODE_PATH environment variable to the following in the kumascript config file:

 [[e-mail]3 /]# cat /etc/supervisord.conf.d/nodejs-kumascript.conf
 [program:nodejs-kumascript]
 command=/usr/bin/nodejs kumascript/run.js
 directory=/data/www/developer-dev.allizom.org/kuma
 user=apache
 numprocs=1
 stdout_logfile=/var/log/nodejs-kumascript-supervisor.log
 stderr_logfile=/var/log/nodejs-kumascript-supervisor.log
 autostart=true
 autorestart=true
 startsecs=10
 environment=NODE_PATH=/usr/lib/node_modules
 ...


when i manually test this manually, it appears to work correctly.

 [[e-mail]3 /]# export NODE_PATH=/usr/lib/node_modules

 [[e-mail]3 /]# node
 > var Fibers = require('fibers');
 undefined
 > var Future = require('fibers/future');
 undefined
 >
 (^C again to quit)
 >

Comment 11

[Les Orchard [:lorchard]]

(In reply to Chris Turra [:cturra] from comment #9)
> i have packaged the latest fibers module and installed it in dev, but
> kumascript is failing to startup with the following error:

That's expected. Per comment #2, KumaScript needs to be upgraded to this commit:

    https://github.com/mozilla/kumascript/commit/6429d5c7bdc9780408b600fe6712c5a36dddfc2c

It would be great if we could get that upgraded temporarily & manually on -dev to see if it works. 

The more permanent option is when we update the commit in the kuma git submodule. But, once we do that, we have to halt all pushes to stage & prod until we have all of this working.

Comment 12

[Les Orchard [:lorchard]]

:cturra, can you manually update the kumascript checkout in the kuma directory, if only to test out the changes?

Also, it looks like the permissions on the fibers installation are readable by root only, so I can't load it in my node REPL:

[[e-mail]3 kumascript]$ ls -al /usr/lib/node_modules/fibers/
total 92
drwxr-xr-x  5 root root  4096 Apr 22 14:44 .
drwxr-xr-x  3 root root  4096 Feb 19 20:11 ..
drwxr-xr-x 18 root root  4096 Apr 22 14:44 bin
-rw-r-----  1 root root  1255 Apr 22 14:18 binding.gyp
-rwxr-x---  1 root root  2235 Apr 22 14:18 build.js
-rw-r-----  1 root root   526 Apr 22 14:18 fibers.js
-rw-r-----  1 root root  8215 Apr 22 14:18 future.js
-rw-r-----  1 root root  1054 Apr 22 14:18 LICENSE
-rw-r-----  1 root root    21 Apr 22 14:18 .npmignore
-rw-r-----  1 root root 14897 Apr 22 14:18 package.json
-rw-r-----  1 root root   151 Apr 22 14:18 quick-test.js
-rw-r-----  1 root root 13407 Apr 22 14:18 README.md
drwxr-xr-x  3 root root  4096 Apr 22 14:44 src
drwxr-xr-x  2 root root  4096 Apr 22 14:44 test
-rwxr-x---  1 root root  1048 Apr 22 14:18 test.js

Comment 13

[Chris Turra [:cturra]]

(In reply to Les Orchard [:lorchard] from comment #12)
> :cturra, can you manually update the kumascript checkout in the kuma
> directory, if only to test out the changes?

updated in dev manually:

 [[e-mail]3 kuma]# git submodule status
 +6429d5c7bdc9780408b600fe6712c5a36dddfc2c kumascript (remotes/origin/HEAD)
 f079d3624d39a48b14172a0a2e53f6c8307927f7 vendor (remotes/origin/768498-more-access-control-40-gf079d36)


after the kumascript update, i am still seeing the same error starting kumascript with supervisorctl:

 module.js:340
    throw err;
          ^
 Error: Cannot find module 'fibers'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/data/www/developer-dev.allizom.org/kuma/kumascript/lib/kumascript/templates.js:12:13)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)


> Also, it looks like the permissions on the fibers installation are readable
> by root only, so I can't load it in my node REPL:

these permissions match fibers in production.

 [[e-mail]3 ~]# ls -l /usr/lib/node_modules/fibers/
 total 48
 -rw-r--r-- 1 root root   543 Feb  2  2012 fibers.js
 -rw-r--r-- 1 root root  8057 Mar  8  2012 future.js
 -rw-r--r-- 1 root root  1054 Aug 13  2011 LICENSE
 -rw-r--r-- 1 root root   397 Apr 26  2012 Makefile
 -rw-r--r-- 1 root root  1088 May  3  2012 package.json
 -rw-r--r-- 1 root root 11535 Feb  2  2012 README.md
 drwxr-xr-x 3 root root  4096 May  4  2012 src
 drwxr-xr-x 2 root root  4096 May  4  2012 test
 -rwxr-xr-x 1 root root   174 Mar 26  2012 test.sh

Comment 14

[Les Orchard [:lorchard]]

(In reply to Chris Turra [:cturra] from comment #13) 

> these permissions match fibers in production.

No, they don't match :) On dev, there's no world-readable bit for fibers.js and friends.

On prod, "-rw-r--r--"
On dev,  "-rw-r-----"

[[e-mail]3 ~]$ wc -l /usr/lib/node_modules/fibers/fibers.js
wc: /usr/lib/node_modules/fibers/fibers.js: Permission denied

[[e-mail]3 ~]$ wc -l /usr/lib/node_modules/fibers/fibers.js
13 /usr/lib/node_modules/fibers/fibers.js

Comment 15

[Chris Turra [:cturra]]

(In reply to Les Orchard [:lorchard] from comment #14)
> (In reply to Chris Turra [:cturra] from comment #13) 
> 
> > these permissions match fibers in production.
> 
> No, they don't match :) On dev, there's no world-readable bit for fibers.js
> and friends.
> 
> On prod, "-rw-r--r--"
> On dev,  "-rw-r-----"

gah! sorry dude, i totally missed that :(

after fiddling with the rpm spec file, it looks like i have managed to get the package to have correct permissions! this is why we have a development environment ;)

 [[e-mail]3 node_modules]# /usr/bin/supervisorctl restart nodejs-kumascript
 nodejs-kumascript: ERROR (not running)
 nodejs-kumascript: started

Comment 16

[Chris Turra [:cturra]]

how is dev looking with these updates?

Comment 17

[David Walsh :davidwalsh]

Hmmm; I'm getting a Permission Denied when I try to hit the "New Template" screen on dev.  

https://developer-dev.allizom.org/en-US/docs/new?slug=Template:

Les, what would be the best way to testing this?  Can we have Chris run the test suite on developer-dev (i.e. can you provide the command for that?)?

Comment 18

[Les Orchard [:lorchard]]

(In reply to David Walsh :davidwalsh from comment #17)
> Hmmm; I'm getting a Permission Denied when I try to hit the "New Template"
> screen on dev.  

How did you sign in? Looks like you have 2 accounts:

* [e-mail] - admin, can author kumascript templates

* [e-mail] - not admin, cannot author kumascript templates

> Les, what would be the best way to testing this?  Can we have Chris run the
> test suite on developer-dev (i.e. can you provide the command for that?)?

I was able to do a git checkout of kumascript into my home directory on developer-dev. All the tests passed when I ran the suite.

But, the real installation of kumascript on dev is broken again. The regular updates from master branch clobbered the manual upgrade :cturra did.

It gets tricky for the next steps:

We have to commit a change to kuma making the kumascript upgrade permanent. That should fix things on developer-dev.

But then, we'll have to hold all pushes and upgrade node & fibers on staging. After that, we need to immediately push to staging. Then, verify that kumascript still works.
 
Repeat for prod: Upgrade node & fibers. Push new kumascript. Verify.

So I think that means we need to have a time where we're all-hands-on-deck for the upgrades. Maybe we want to wait for Monday to do this, rather than doing it on a Friday? Is there a time that works for everyone?

Comment 19

[Chris Turra [:cturra]]

^ +1 to this. we definitely don't want to do this on a friday. lets schedule a time monday to work through this together as a group.

Comment 20

[David Walsh :davidwalsh]

I'm all for Monday, as my entire day is open!

Comment 21

[Chris Turra [:cturra]]

upgrades complete in stage and prod. woohoo! \o/