Closed Bug 1015382 Opened 8 years ago Closed 8 years ago

We should use Jenkins to manage the Android Eideticker nodes/phones

Categories

(Testing Graveyard :: Eideticker, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wlach, Assigned: wlach)

References

Details

Attachments

(1 file, 1 obsolete file)

Using jenkins has proven to be a big help with managing the B2G Eideticker nodes. In particular:

1. It provides a web interface to the logs from the test, which makes failures easy to track down (I set up something similar for the Android Eideticker nodes using cron2rss, but it's not nearly as good).
2. It was easy to hook it up the tinderbox directory so we can be notified when new builds appear (and schedule new builds automatically).
3. It automatically checks out and configures a working eideticker instance using the latest version in git for each run, eliminating the need to update things manually when there are new changes we want to use.

(and probably other advantages I can't remember right now)

Now that we have a Jenkins VM in the datacenter, we can manage multiple worker nodes simultaneously, making it vastly easier to scale eideticker up to handling more devices.

It would be good to port the Android Eideticker over to this setup, so it can enjoy the same advantages (and we can hopefully spend more time writing code for Android, and less time investigating failures). In most respects, Android Eideticker works pretty much the same as B2G Eideticker, only some minor configuration is different. We can probably use the existing eideticker-ci configuration (https://github.com/mozilla/eideticker-ci) as a starting point at the very least... and quite possibly we could make it work for both (though we'll probably want to have one jenkins instance running for both of Android and B2G to reduce confusion).
Dave: Do you have any opinion on whether we can just add Eideticker for Android as a new set of download/test jobs to the existing eideticker-ci setup? Or should we fork the project? As mentioned, I think we should run two seperate jenkins instances regardless.
Flags: needinfo?(dave.hunt)
I actually think that we can have a single project to serve both Firefox Mobile and Firefox OS needs. It's possible in Jenkins to configure multiple 'views', and with a sensible job naming strategy it should be easy to keep the two products distinct. Jenkins isn't too much trouble to maintain, but a single instance would be half as much work. I've raised bug 1016297 for setting up the single instance for B2G, and once that's done we can experiment with adding Firefox Mobile jobs.
Depends on: 1016297
Flags: needinfo?(dave.hunt)
I decided to take this on, since it'll make a bunch of other stuff easier.
Assignee: nobody → wlachance
Attached patch Android work (obsolete) — Splinter Review
Ok, so I got this working!

I'm not proposing this goes in as-is (there are some changes here I put in just to get things working locally), but I wanted to get some feedback on the overall approach.
Attachment #8458997 - Flags: feedback?(dave.hunt)
Comment on attachment 8458997 [details] [diff] [review]
Android work

Review of attachment 8458997 [details] [diff] [review]:
-----------------------------------------------------------------

Looks great! I've added a few comments, but they're probably things you're already aware of, and generally minor.

::: jenkins-master/buildstep-config-files.xml
@@ +10,2 @@
>          <content>#!/bin/bash -ex
> +dm push orng /mnt/sdcard/orng

This won't work across Android/B2G devices because of the varying path to the temporary storage. We could use a PATH argument, which could be set on the Jenkins slave instances.

@@ +34,5 @@
> +        <name>Install Eideticker dependencies</name>
> +        <comment>Install Eideticker dependencies</comment>
> +        <content>#!/bin/bash -ex
> +./bootstrap.sh
> +pip install -e /home/wlach/src/mozilla-central-hg/testing/mozbase/mozdevice</content>

Would we want to add this as a dependency within bootstrap.sh?

@@ +165,5 @@
> +    <entry>
> +      <string>org.jenkinsci.plugins.managedscripts.ScriptConfig1404941328553</string>
> +      <org.jenkinsci.plugins.managedscripts.ScriptConfig>
> +        <id>org.jenkinsci.plugins.managedscripts.ScriptConfig1404941328553</id>
> +        <name>Download fennec</name>

Nit: s/fennec/Fennec

@@ +167,5 @@
> +      <org.jenkinsci.plugins.managedscripts.ScriptConfig>
> +        <id>org.jenkinsci.plugins.managedscripts.ScriptConfig1404941328553</id>
> +        <name>Download fennec</name>
> +        <comment>Download the fennec .apk from specified location</comment>
> +        <content>  wget -r -l1 -nd -np -A.en-US.android-arm.apk $1

Nit: Leading whitespace

@@ +203,5 @@
> +        <content>#!/bin/bash -ex
> +dm push ntpclient /mnt/sdcard/ntpclient
> +dm shell --root cp /mnt/sdcard/ntpclient /data/local/ntpclient
> +dm shell --root chmod 700 /data/local/ntpclient
> +dm shell rm /mnt/sdcard/ntpclient</content>

As above, regarding path to storage location.

::: jenkins-master/config.xml
@@ +122,5 @@
> +      <jobNames>
> +        <comparator class="hudson.util.CaseInsensitiveComparator"/>
> +        <string>android.galaxy-nexus.mozilla-central.tests</string>
> +        <string>android.mozilla-central.download</string>
> +      </jobNames>

Rather than list the jobs, you can use a regex to construct the list dynamically.

@@ +162,4 @@
>            <string>NDK_PATH</string>
> +          <string>/home/wlach/opt/android-ndk-r9b/</string>
> +          <string>TEST_DEVICE</string>
> +          <string>10.242.25.20</string>

These values will all need to be removed before landing.

@@ +181,4 @@
>            <string>DASHBOARD_LOCAL_PATH</string>
>            <string>${WORKSPACE}/dashboard/</string>
>            <string>DASHBOARD_REMOTE_PATH</string>
> +          <string>www</string>

This will need to be different for B2G and Android until the dashboards are combined. Perhaps the best way to do this would be to explicitly override DASHBOARD_REMOTE_PATH in the B2G jobs.
Attachment #8458997 - Flags: feedback?(dave.hunt) → feedback+
So this is pretty close to landing, pending me fixing/addressing the issues pointed out above. We may as well start things off on the right foot by running results against the branches that would be most useful to developers.

Blassey: Which branches would you like us to fetch/test builds from? mozilla-inbound? fx-team? Which tests would you like us to run? (if we run a subset of the tests we were running before, we could generate more results)
Flags: needinfo?(blassey.bugs)
as discussed in this week's testing meeting we are going to run tests on mozilla-inbound, fx-team and mozilla-cental with the current roster of tests. We can adjust later if the frequency of test runs doesn't meet our needs.
Flags: needinfo?(blassey.bugs)
Depends on: 1046380
(In reply to Dave Hunt (:davehunt) from comment #5)
> Comment on attachment 8458997 [details] [diff] [review]
> Android work
> 
> Review of attachment 8458997 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks great! I've added a few comments, but they're probably things you're
> already aware of, and generally minor.
> 
> ::: jenkins-master/buildstep-config-files.xml
> @@ +10,2 @@
> >          <content>#!/bin/bash -ex
> > +dm push orng /mnt/sdcard/orng
> 
> This won't work across Android/B2G devices because of the varying path to
> the temporary storage. We could use a PATH argument, which could be set on
> the Jenkins slave instances.

Perhaps the easiest thing to do is add a "dm deviceroot" command which we can use to get a temporary storage area that's guaranteed to be writable dynamically. Filed bug 1046380 for that.

> @@ +34,5 @@
> > +        <name>Install Eideticker dependencies</name>
> > +        <comment>Install Eideticker dependencies</comment>
> > +        <content>#!/bin/bash -ex
> > +./bootstrap.sh
> > +pip install -e /home/wlach/src/mozilla-central-hg/testing/mozbase/mozdevice</content>
> 
> Would we want to add this as a dependency within bootstrap.sh?

This was a temporary workaround which is no longer needed now that we have mozdevice 0.38. I took it out.

> @@ +165,5 @@
> > +    <entry>
> > +      <string>org.jenkinsci.plugins.managedscripts.ScriptConfig1404941328553</string>
> > +      <org.jenkinsci.plugins.managedscripts.ScriptConfig>
> > +        <id>org.jenkinsci.plugins.managedscripts.ScriptConfig1404941328553</id>
> > +        <name>Download fennec</name>
> 
> Nit: s/fennec/Fennec

Fixed

> @@ +167,5 @@
> > +      <org.jenkinsci.plugins.managedscripts.ScriptConfig>
> > +        <id>org.jenkinsci.plugins.managedscripts.ScriptConfig1404941328553</id>
> > +        <name>Download fennec</name>
> > +        <comment>Download the fennec .apk from specified location</comment>
> > +        <content>  wget -r -l1 -nd -np -A.en-US.android-arm.apk $1
> 
> Nit: Leading whitespace

Fixed

> ::: jenkins-master/config.xml
> @@ +122,5 @@
> > +      <jobNames>
> > +        <comparator class="hudson.util.CaseInsensitiveComparator"/>
> > +        <string>android.galaxy-nexus.mozilla-central.tests</string>
> > +        <string>android.mozilla-central.download</string>
> > +      </jobNames>
> 
> Rather than list the jobs, you can use a regex to construct the list
> dynamically.

Oh cool, fixed.

> @@ +162,4 @@
> >            <string>NDK_PATH</string>
> > +          <string>/home/wlach/opt/android-ndk-r9b/</string>
> > +          <string>TEST_DEVICE</string>
> > +          <string>10.242.25.20</string>
> 
> These values will all need to be removed before landing.

Yup, was planning to. :)

> @@ +181,4 @@
> >            <string>DASHBOARD_LOCAL_PATH</string>
> >            <string>${WORKSPACE}/dashboard/</string>
> >            <string>DASHBOARD_REMOTE_PATH</string>
> > +          <string>www</string>
> 
> This will need to be different for B2G and Android until the dashboards are
> combined. Perhaps the best way to do this would be to explicitly override
> DASHBOARD_REMOTE_PATH in the B2G jobs.

Makes sense. Will fix.
Ok, so this is actually running semi-reliably now. 

http://eideticker.mozilla.org/#/samsung-gn/mozilla-central/startup-abouthome-dirty/timetostableframe/7

Almost ready to commit these changes, pending 1050552.

However, we're not even running the full complement of tests yet and already runs are taking about 40 minutes to complete. We don't really have enough capacity to add inbound and fxteam across all the tests. 

There's one other node I could bring into operation quite easily, which might provide us with the necessary capacity. But I guess the question then becomes whether we should bring up an lg-g2x or another galaxy-nexus. More devices or more branches?
Flags: needinfo?(blassey.bugs)
What's the question? Adding more devices sounds like a good idea.
Flags: needinfo?(blassey.bugs)
Attached patch Android patch v2Splinter Review
I plan on landing bug 1050552 before this one, which will change some of the logic here somewhat, but I think the changes should be relatively minor (just the addition of DASHBOARD_ID, DASHBOARD_NAME environment variables). Does this look good otherwise? I think I addressed all the comments you had previously.
Attachment #8458997 - Attachment is obsolete: true
Attachment #8472753 - Flags: review?(dave.hunt)
Comment on attachment 8472753 [details] [diff] [review]
Android patch v2

Review of attachment 8472753 [details] [diff] [review]:
-----------------------------------------------------------------

With the comments address, this looks great!

::: jenkins-master/buildstep-config-files.xml
@@ +10,2 @@
>          <content>#!/bin/bash -ex
> +# hack to get around broke dm cli until bug 1046380 fixed

This is now fixed, can we remove the hack?

::: jenkins-master/config.xml
@@ +106,5 @@
> +              <string>Galaxy Nexus</string>
> +              <string>DM_TRANS</string>
> +              <string>sut</string>
> +              <string>TEST_DEVICE</string>
> +              <string>10.242.24.48</string>

I don't think we should include an actual IP address in the repo.

@@ +185,2 @@
>            <string>ADB_PATH</string>
> +          <string>/home/wlach/opt/android-sdk-linux/platform-tools/adb</string>

We shouldn't include this path in the repo.

@@ +187,2 @@
>            <string>NDK_PATH</string>
> +          <string>/home/wlach/opt/android-ndk-r9b/</string>

Same here.

@@ +210,4 @@
>            <string>DASHBOARD_URL</string>
> +          <string>http://eideticker.mozilla.org/</string>
> +          <string>DASHBOARD_USER</string>
> +          <string>eideticker</string>

Are we thinking it's safe to include these now we're going to have a merged dashboard? If so, it's fine with me.

::: jenkins-master/jenkins.model.JenkinsLocationConfiguration.xml
@@ +1,4 @@
>  <?xml version='1.0' encoding='UTF-8'?>
>  <jenkins.model.JenkinsLocationConfiguration>
>    <adminAddress>address not configured yet &lt;nobody@nowhere&gt;</adminAddress>
> +  <jenkinsUrl>http://10.242.24.50:8080/</jenkinsUrl>

Let's not include this change.

::: jenkins-master/jobs/android.galaxy-nexus.mozilla-central.tests/config.xml
@@ +136,5 @@
> +        <propertiesContent>VENV=${WORKSPACE}/.env
> +PATH=${VENV}/bin:${ADB_PATH}:${PATH}
> +CAPTURE_DEVICE=decklink
> +DEVICE_TYPE=android
> +DM_TRANS=sut

As we're setting this on the job, we can probably remove it from the node configuration.
Attachment #8472753 - Flags: review?(dave.hunt) → review+
Pushed with some changes to make it work, now running on the jenkins master!

https://github.com/mozilla/eideticker-ci/commit/df01d613ecda9a9cbd6c0c37f274a8bb4dbf344c

I'll file some followups to enable additional jobs.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.