Closed Bug 987383 Opened 7 years ago Closed 7 years ago

make test-perf producing invalid JSON

Categories

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect, P1)

defect

Tracking

(b2g-v1.3T fixed)

RESOLVED FIXED
1.4 S6 (25apr)
Tracking Status
b2g-v1.3T --- fixed

People

(Reporter: jgriffin, Assigned: hub)

Details

(Keywords: perf, Whiteboard: [c=automation p=2 s= u=])

Attachments

(4 files)

Attached file perf.json
make test-perf is currently producing invalid JSON, and as a result, no data is getting submitted to datazilla.

I'm attaching the latest perf.json it produces.

In this case the error is an extra comma after the first dict.  I'm guessing this is an artifact of generating the JSON from within a make target.  Is there any way we can have the JSON created by the test harness instead?
Assignee: nobody → hub
Priority: -- → P1
Whiteboard: [c=automation p=2 s= u=]
*sigh* once again.
Can we have a link to the jobs that fail?

Thanks
Flags: needinfo?(jgriffin)
Oh, it changed hostname. Thanks.
Seems that I can't connect to this either. Not sure what VPN to use since I was told the MV VPN as no longer functional.
Basically a single test doesn't output anything at the beginning. This cause the double commas.
(In reply to Hubert Figuiere [:hub] from comment #5)
> Seems that I can't connect to this either. Not sure what VPN to use since I
> was told the MV VPN as no longer functional.

You have to use the new VPN, openvpn.scl3.mozilla.com.
Attached file Full Jenkins log
Status: NEW → ASSIGNED
Keywords: perf
(In reply to Jonathan Griffin (:jgriffin) from comment #8)
> See also: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=30769829

I was looking on Intranet. Yet again.

But then yeah that's the one I'm using now and it doesn't work. I have requested more ACL for the less open stuff. That's sad. Less open....
13:59:08 node_modules installed.
13:59:08 make[1]: Leaving directory `/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia'
13:59:08 Warning: Native modules not compiled.  XOR performance will be degraded.
13:59:08 Warning: Native modules not compiled.  UTF-8 validation disabled.
13:59:08 Warning: Native modules not compiled.  XOR performance will be degraded.
13:59:08 Warning: Native modules not compiled.  UTF-8 validation disabled.
13:59:09 Warning: Native modules not compiled.  XOR performance will be degraded.
13:59:09 Warning: Native modules not compiled.  UTF-8 validation disabled.
13:59:09 Warning: Native modules not compiled.  XOR performance will be degraded.
13:59:09 Warning: Native modules not compiled.  UTF-8 validation disabled.
13:59:17 
13:59:17 /var/jenkins/workspace/b2g.hamachi.mozilla-central.master.mozperftest/gaia/shared/test/integration/profile_builder.js:115
13:59:17         throw err;
13:59:17               ^
13:59:17 Error: Command failed: make: *** [preferences] Segmentation fault (core dumped)
13:59:17 
13:59:17     at ChildProcess.exithandler (child_process.js:637:15)
13:59:17     at ChildProcess.EventEmitter.emit (events.js:98:17)
13:59:17     at maybeClose (child_process.js:743:16)
13:59:17     at Process.ChildProcess._handle.onexit (child_process.js:810:5)

There, you have it. Btw I don't run the profile builder explicitly, so I have no idea what is causing this. But this would explain why there was no output.
So, yeah. Apparently |make| crashes. I don't know why, but that's quite interesting. I should be able to actually recover from that.
Is there a way for you to investigate as to why make is actually crashing? It works fine here....
Flags: needinfo?(jgriffin)
Could this be a node version issue?  Dave, can you tell us what version of node is on the test slave?
Flags: needinfo?(jgriffin) → needinfo?(dave.hunt)
The process that crashes is really |make| and not |node|. That |make| is called from Node.js
Do you have any suggestions for how we could debug this, then?  We could probably get you ssh access to the slave, so you can manually run make, if that would help.
$ node -v
v0.10.22
$ npm -v
1.3.14
Flags: needinfo?(dave.hunt)
(In reply to Jonathan Griffin (:jgriffin) from comment #16)
> Do you have any suggestions for how we could debug this, then?  We could
> probably get you ssh access to the slave, so you can manually run make, if
> that would help.

I guess getting ssh access to one of the slaves would help. I'd be able to see more.
Flags: needinfo?(jgriffin)
The good news: I could reproduce. The bad news, it is on Mike Lien (Taipei) machine.

Looks like Ubuntu is causing us trouble here.
Stephen, can you tell hub what he needs to do to get ssh access to the Jenkins slave that is running the hamachi mozperftest jobs?
Flags: needinfo?(jgriffin) → needinfo?(stephen.donner)
(In reply to Jonathan Griffin (:jgriffin) from comment #20)
> Stephen, can you tell hub what he needs to do to get ssh access to the
> Jenkins slave that is running the hamachi mozperftest jobs?

Sure; either 1) ping :jabba on #it and/or 2) file a bug here: https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=Mozilla%20VPN:%20ACL%20requests, requesting access to http://selenium.qa.mtv2.mozilla.com:8080/ and its QA VLAN (you'll need VPN, I think).  Raymond (:retornam) can help with the Web QA-specific part after that's done.
Flags: needinfo?(stephen.donner)
(In reply to Hubert Figuiere [:hub] from comment #19)
> The good news: I could reproduce. The bad news, it is on Mike Lien (Taipei)
> machine.
> 
> Looks like Ubuntu is causing us trouble here.

His config is:

Ubuntu:         12.04
make 3.81-8.1ubuntu1.1
x86_64


I'll look at setting up a VM with that configuration and see what is going on.
(In reply to Stephen Donner [:stephend] from comment #21)
> (In reply to Jonathan Griffin (:jgriffin) from comment #20)
> > Stephen, can you tell hub what he needs to do to get ssh access to the
> > Jenkins slave that is running the hamachi mozperftest jobs?
> 
> Sure; either 1) ping :jabba on #it and/or 2) file a bug here:
> https://bugzilla.mozilla.org/enter_bug.
> cgi?product=Infrastructure%20%26%20Operations&component=Mozilla%20VPN:
> %20ACL%20requests, requesting access to
> http://selenium.qa.mtv2.mozilla.com:8080/ and its QA VLAN (you'll need VPN,
> I think).  Raymond (:retornam) can help with the Web QA-specific part after
> that's done.

I have gotten access to that machine with VPN.


Raymond, how can I ssh to the slave machine - I can provide a public key if you need that? Thanks.
Flags: needinfo?(mozbugs.retornam)
There are currently 4 potential nodes this could run on. I've sent :hub an e-mail with the details for each.
Flags: needinfo?(mozbugs.retornam)
Blocks: 980993
This is the simple fix of reporting an error. This doesn't address the crash that causes it, nor does it hide - I haven't been able to reproduce or figure out here.

Was that, the JSON will stay valid.
Attachment #8408431 - Flags: review?(felash)
Comment on attachment 8408431 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/18430

r=me

What happens then when there are unexpected failures? Is it being reported?
Attachment #8408431 - Flags: review?(felash) → review+
(In reply to Julien Wajsberg [:julienw] from comment #26)

> What happens then when there are unexpected failures? Is it being reported?

stderr is output on stderr. So we are fine. In the present case it is already reported, but the stdout being empty caused a ',' to be output causing the invalid JSON.
https://github.com/mozilla-b2g/gaia/commit/4d8da43bff102ef8fe5d14814ad20cf36cfd7bb6
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Will file a follow up for the segfault.
See bug 998317
There is another issue with JSON validation. *sigh*
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Depends on: 962918
The JSON validation is caused by an attempt to fix bug 962918 where they output things on stdout... *sigh*. I need to validate the fix.
No longer blocks: 980993
(In reply to Hubert Figuiere [:hub] from comment #27)
> (In reply to Julien Wajsberg [:julienw] from comment #26)
> 
> > What happens then when there are unexpected failures? Is it being reported?
> 
> stderr is output on stderr. So we are fine. In the present case it is
> already reported, but the stdout being empty caused a ',' to be output
> causing the invalid JSON.

I meant, monitored and reported by someone as a bug ;)
(In reply to Julien Wajsberg [:julienw] from comment #33)

> I meant, monitored and reported by someone as a bug ;)


We have alerts that are being setup to report when there is no data. So we'll track these. Not sure exactly the bug number, but given that I actually output an error instead of nothing is already showing the intent of doing so :-)
Kevin, if you don't mind reviewing this. Thanks.
Attachment #8410259 - Flags: review?(kgrandon)
Removing blocker.
No longer depends on: 962918
Comment on attachment 8410259 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/18525

Looks good, thanks.
Attachment #8410259 - Flags: review?(kgrandon) → review+
Landed: https://github.com/mozilla-b2g/gaia/commit/b3d380d24ba78bf9cfd8ebb10884d0d283203b72
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
Thanks Kevin !
Target Milestone: --- → 1.4 S6 (25apr)
You need to log in before you can comment on or make changes to this bug.