987383 - make test-perf producing invalid JSON

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect, P1)

Description

[Jonathan Griffin (:jgriffin)]

Attachment: perf.json

make test-perf is currently producing invalid JSON, and as a result, no data is getting submitted to datazilla.

I'm attaching the latest perf.json it produces.

In this case the error is an extra comma after the first dict.  I'm guessing this is an artifact of generating the JSON from within a make target.  Is there any way we can have the JSON created by the test harness instead?

Comment 1

[Hubert Figuiere [:hub]]

*sigh* once again.

Comment 2

[Hubert Figuiere [:hub]]

Can we have a link to the jobs that fail?

Thanks

Comment 3

[Jonathan Griffin (:jgriffin)]

http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Perf/job/b2g.hamachi.mozilla-central.master.mozperftest/1459/

Comment 4

[Hubert Figuiere [:hub]]

Oh, it changed hostname. Thanks.

Comment 5

[Hubert Figuiere [:hub]]

Seems that I can't connect to this either. Not sure what VPN to use since I was told the MV VPN as no longer functional.

Comment 6

[Hubert Figuiere [:hub]]

Basically a single test doesn't output anything at the beginning. This cause the double commas.

Comment 7

[Jonathan Griffin (:jgriffin)]

(In reply to Hubert Figuiere [:hub] from comment #5)
> Seems that I can't connect to this either. Not sure what VPN to use since I
> was told the MV VPN as no longer functional.

You have to use the new VPN, openvpn.scl3.mozilla.com.

Comment 8

[Jonathan Griffin (:jgriffin)]

See also: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=30769829

Comment 9

[Jonathan Griffin (:jgriffin)]

Attachment: Full Jenkins log

Comment 10

[Hubert Figuiere [:hub]]

(In reply to Jonathan Griffin (:jgriffin) from comment #8)
> See also: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=30769829

I was looking on Intranet. Yet again.

But then yeah that's the one I'm using now and it doesn't work. I have requested more ACL for the less open stuff. That's sad. Less open....

Comment 11

[Hubert Figuiere [:hub]]

13:59:08 node_modules installed.
13:59:08 make[1]: Leaving directory `/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia'
13:59:08 Warning: Native modules not compiled.  XOR performance will be degraded.
13:59:08 Warning: Native modules not compiled.  UTF-8 validation disabled.
13:59:08 Warning: Native modules not compiled.  XOR performance will be degraded.
13:59:08 Warning: Native modules not compiled.  UTF-8 validation disabled.
13:59:09 Warning: Native modules not compiled.  XOR performance will be degraded.
13:59:09 Warning: Native modules not compiled.  UTF-8 validation disabled.
13:59:09 Warning: Native modules not compiled.  XOR performance will be degraded.
13:59:09 Warning: Native modules not compiled.  UTF-8 validation disabled.
13:59:17 
13:59:17 /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/shared/test/integration/profile_builder.js:115
13:59:17         throw err;
13:59:17               ^
13:59:17 Error: Command failed: make: *** [preferences] Segmentation fault (core dumped)
13:59:17 
13:59:17     at ChildProcess.exithandler (child_process.js:637:15)
13:59:17     at ChildProcess.EventEmitter.emit (events.js:98:17)
13:59:17     at maybeClose (child_process.js:743:16)
13:59:17     at Process.ChildProcess._handle.onexit (child_process.js:810:5)

There, you have it. Btw I don't run the profile builder explicitly, so I have no idea what is causing this. But this would explain why there was no output.

Comment 12

[Hubert Figuiere [:hub]]

So, yeah. Apparently |make| crashes. I don't know why, but that's quite interesting. I should be able to actually recover from that.

Comment 13

[Hubert Figuiere [:hub]]

Is there a way for you to investigate as to why make is actually crashing? It works fine here....

Comment 14

[Jonathan Griffin (:jgriffin)]

Could this be a node version issue?  Dave, can you tell us what version of node is on the test slave?

Comment 15

[Hubert Figuiere [:hub]]

The process that crashes is really |make| and not |node|. That |make| is called from Node.js

Comment 16

[Jonathan Griffin (:jgriffin)]

Do you have any suggestions for how we could debug this, then?  We could probably get you ssh access to the slave, so you can manually run make, if that would help.

Comment 17

[Dave Hunt [:davehunt] [he/him] ⌚BST]

$ node -v
v0.10.22
$ npm -v
1.3.14

Comment 18

[Hubert Figuiere [:hub]]

(In reply to Jonathan Griffin (:jgriffin) from comment #16)
> Do you have any suggestions for how we could debug this, then?  We could
> probably get you ssh access to the slave, so you can manually run make, if
> that would help.

I guess getting ssh access to one of the slaves would help. I'd be able to see more.

Comment 19

[Hubert Figuiere [:hub]]

The good news: I could reproduce. The bad news, it is on Mike Lien (Taipei) machine.

Looks like Ubuntu is causing us trouble here.

Comment 20

[Jonathan Griffin (:jgriffin)]

Stephen, can you tell hub what he needs to do to get ssh access to the Jenkins slave that is running the hamachi mozperftest jobs?

Comment 21

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Jonathan Griffin (:jgriffin) from comment #20)
> Stephen, can you tell hub what he needs to do to get ssh access to the
> Jenkins slave that is running the hamachi mozperftest jobs?

Sure; either 1) ping :jabba on #it and/or 2) file a bug here: https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=Mozilla%20VPN:%20ACL%20requests, requesting access to http://selenium.qa.mtv2.mozilla.com:8080/ and its QA VLAN (you'll need VPN, I think).  Raymond (:retornam) can help with the Web QA-specific part after that's done.

Comment 22

[Hubert Figuiere [:hub]]

(In reply to Hubert Figuiere [:hub] from comment #19)
> The good news: I could reproduce. The bad news, it is on Mike Lien (Taipei)
> machine.
> 
> Looks like Ubuntu is causing us trouble here.

His config is:

Ubuntu:         12.04
make 3.81-8.1ubuntu1.1
x86_64


I'll look at setting up a VM with that configuration and see what is going on.

Comment 23

[Hubert Figuiere [:hub]]

(In reply to Stephen Donner [:stephend] from comment #21)
> (In reply to Jonathan Griffin (:jgriffin) from comment #20)
> > Stephen, can you tell hub what he needs to do to get ssh access to the
> > Jenkins slave that is running the hamachi mozperftest jobs?
> 
> Sure; either 1) ping :jabba on #it and/or 2) file a bug here:
> https://bugzilla.mozilla.org/enter_bug.
> cgi?product=Infrastructure%20%26%20Operations&component=Mozilla%20VPN:
> %20ACL%20requests, requesting access to
> http://selenium.qa.mtv2.mozilla.com:8080/ and its QA VLAN (you'll need VPN,
> I think).  Raymond (:retornam) can help with the Web QA-specific part after
> that's done.

I have gotten access to that machine with VPN.


Raymond, how can I ssh to the slave machine - I can provide a public key if you need that? Thanks.

Comment 24

[Dave Hunt [:davehunt] [he/him] ⌚BST]

There are currently 4 potential nodes this could run on. I've sent :hub an e-mail with the details for each.

Comment 25

[Hubert Figuiere [:hub]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/18430

This is the simple fix of reporting an error. This doesn't address the crash that causes it, nor does it hide - I haven't been able to reproduce or figure out here.

Was that, the JSON will stay valid.

Comment 26

[Julien Wajsberg [:julienw]]

Comment on attachment 8408431 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/18430

r=me

What happens then when there are unexpected failures? Is it being reported?

Comment 27

[Hubert Figuiere [:hub]]

(In reply to Julien Wajsberg [:julienw] from comment #26)

> What happens then when there are unexpected failures? Is it being reported?

stderr is output on stderr. So we are fine. In the present case it is already reported, but the stdout being empty caused a ',' to be output causing the invalid JSON.

Comment 28

[Hubert Figuiere [:hub]]

https://github.com/mozilla-b2g/gaia/commit/4d8da43bff102ef8fe5d14814ad20cf36cfd7bb6

Comment 29

[Hubert Figuiere [:hub]]

Will file a follow up for the segfault.

Comment 30

[Hubert Figuiere [:hub]]

See bug 998317

Comment 31

[Hubert Figuiere [:hub]]

There is another issue with JSON validation. *sigh*

Comment 32

[Hubert Figuiere [:hub]]

The JSON validation is caused by an attempt to fix bug 962918 where they output things on stdout... *sigh*. I need to validate the fix.

Comment 33

[Julien Wajsberg [:julienw]]

(In reply to Hubert Figuiere [:hub] from comment #27)
> (In reply to Julien Wajsberg [:julienw] from comment #26)
> 
> > What happens then when there are unexpected failures? Is it being reported?
> 
> stderr is output on stderr. So we are fine. In the present case it is
> already reported, but the stdout being empty caused a ',' to be output
> causing the invalid JSON.

I meant, monitored and reported by someone as a bug ;)

Comment 34

[Hubert Figuiere [:hub]]

(In reply to Julien Wajsberg [:julienw] from comment #33)

> I meant, monitored and reported by someone as a bug ;)


We have alerts that are being setup to report when there is no data. So we'll track these. Not sure exactly the bug number, but given that I actually output an error instead of nothing is already showing the intent of doing so :-)

Comment 35

[Hubert Figuiere [:hub]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/18525

Kevin, if you don't mind reviewing this. Thanks.

Comment 36

[Hubert Figuiere [:hub]]

Removing blocker.

Comment 37

[Kevin Grandon :kgrandon]

Comment on attachment 8410259 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/18525

Looks good, thanks.

Comment 38

[Kevin Grandon :kgrandon]

Landed: https://github.com/mozilla-b2g/gaia/commit/b3d380d24ba78bf9cfd8ebb10884d0d283203b72

Comment 39

[Hubert Figuiere [:hub]]

Thanks Kevin !

Comment 40

[Hubert Figuiere [:hub]]

v1.3t uplift
https://github.com/mozilla-b2g/gaia/commit/30f14d53cbd529bb9404acede7f7eeeecb867134
https://github.com/mozilla-b2g/gaia/commit/f2aad8924908aae9a1f28c2493068fb42a679217