Closed Bug 740829 Opened 12 years ago Closed 12 years ago

Nightly builds report for android doesn't have any data

Categories

(Socorro :: Webapp, task)

Product:
Socorro
Component:
Webapp
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: jrmuizel, Assigned: rhelmer)

References

Details

Attachments

(1 file)

qa - verified
106.01 KB, image/png
Details 
Josh can you take a look at this? From a quick look at update_nightly_builds() SP I don't see why it shouldn't be doing FennecAndroid.

I don't see any other reports having data on crash-stats other than Firefox, actually.
Assignee: nobody → josh
Status: NEW → ASSIGNED
The problem is that the cron job for nightly_builds hasn't been run since March 22.  

So, assigning back to you.
Assignee: josh → rhelmer
(In reply to [:jberkus] Josh Berkus from comment #2)
> The problem is that the cron job for nightly_builds hasn't been run since
> March 22.  
> 
> So, assigning back to you.

Yep, looks like we didn't enable the cronjob at the time that we switched over to using the matview. I think we should do a point release for this today.
Target Milestone: --- → 3
Blocks: 740975
(In reply to Robert Helmer [:rhelmer] from comment #3)
> (In reply to [:jberkus] Josh Berkus from comment #2)
> > The problem is that the cron job for nightly_builds hasn't been run since
> > March 22.  
> > 
> > So, assigning back to you.
> 
> Yep, looks like we didn't enable the cronjob at the time that we switched
> over to using the matview. I think we should do a point release for this
> today.

OK pushed the fix to prod in bug 740975, back to you to backfill the matview :P
Assignee: rhelmer → josh
Backfilled.

Once cache times out, we can verify it.
Umm, this still seems to not work, and I'm getting requests for this, as people want to see what crashes "are still happening" in current nightly builds.

Can we please look into this ASAP?
Target Milestone: 3 → 6
Has the cronjob been fixed?
Assignee: josh → rhelmer
(In reply to [:jberkus] Josh Berkus from comment #7)
> Has the cronjob been fixed?

Yes, was pushed in bug 740975. I'll take a look.
(In reply to Robert Helmer [:rhelmer] from comment #8)
> (In reply to [:jberkus] Josh Berkus from comment #7)
> > Has the cronjob been fixed?
> 
> Yes, was pushed in bug 740975. I'll take a look.

Yes job has been running, or at least logging successfully, since bug 740975 was pushed:

2012-04-11 02:19:14,629 INFO - Running update_nightly_builds

The SP it's calling is "update_nightly_builds", Josh can you make sure that's set up ok in the DB?
Assignee: rhelmer → josh
Ok, cron job confirmed working, and there's plenty of data in the matview for FennecAndroid 14.0a1.  So we should be seeing a report here.

So this is apparently a middleware or UI bug.  Needs to be reassigned.
Rob, can you puzzle this one out?
Assignee: josh → rhelmer
(In reply to Laura Thomson :laura from comment #11)
> Rob, can you puzzle this one out?

Sure, taking a look now - Firefox, Thunderbird and SeaMonkey all seem to work find and be up to date, nothing for Fennec and FennecAndroid.
OK here's the problem, in this middleware service:

https://github.com/mozilla/socorro/blob/master/socorro/external/postgresql/products_builds.py#L98

The SQL that gets generated is:
"""
/* socorro.external.postgresql.builds.Builds.builds */ 
SELECT  product_name as product,
         version,
         platform,
         build_id as buildid,
         build_type,
         beta_number,
         repository,
         build_date(build_id) as date 
FROM releases_raw 
WHERE product_name = E'FennecAndroid'
             
AND build_date(build_id) >=
    timestamp with time zone '2012-04-05'
AND repository IN ('mozilla-central', 'mozilla-1.9.2',
                   'comm-central', 'comm-1.9.2',
                   'comm-central-trunk')
ORDER BY build_date(build_id) DESC, product_name ASC, version ASC,
"""

However the product_name in releases_raw comes from FTP, so they look like:
seamonkey, firefox, mobile, thunderbird

Josh, should this be using product_info or some other table instead? I know we already have logic in the SPs to determine which "mobile" builds are Fennec versus FennecAndroid.
Robert,

Yes, it should be using product_info.
Or, if product_info doesn't have enough data, it should be using product_versions.
hmmm.

You know, let me write the query for this, since nobody on the development team has ever dealt with product_version_builds before.
Here's my suggested query:

SELECT DISTINCT product_versions.product_name,
  product_versions.version_string,
  product_version_builds.platform,
  product_version_builds.build_id,
  product_versions.build_type,
  product_versions.beta_number,
  build_date(product_version_builds.build_id) as build_date,
  product_versions.version_sort
FROM product_versions JOIN product_version_builds
  USING ( product_version_id )
WHERE product_name = E'FennecAndroid'
  AND build_date(build_id) >= DATE '$build_date'
ORDER BY build_date DESC, product_name ASC, version_sort ASC;

This loses the repository column, though, since we're not tracking that.
(In reply to [:jberkus] Josh Berkus from comment #17)
> Here's my suggested query:
> 
> SELECT DISTINCT product_versions.product_name,
>   product_versions.version_string,
>   product_version_builds.platform,
>   product_version_builds.build_id,
>   product_versions.build_type,
>   product_versions.beta_number,
>   build_date(product_version_builds.build_id) as build_date,
>   product_versions.version_sort
> FROM product_versions JOIN product_version_builds
>   USING ( product_version_id )
> WHERE product_name = E'FennecAndroid'
>   AND build_date(build_id) >= DATE '$build_date'
> ORDER BY build_date DESC, product_name ASC, version_sort ASC;
> 
> This loses the repository column, though, since we're not tracking that.

Thanks, I think that's fine as we don't seem to be surfacing it in https://crash-stats.mozilla.com/products/Firefox/builds

I can work up a patch for this
BTW I think we should take this for Socorro 6 (April 18) even though it missed freeze.
(In reply to Robert Helmer [:rhelmer] from comment #20)
> BTW I think we should take this for Socorro 6 (April 18) even though it
> missed freeze.

Yes, please.
OK so there are two potential issues with using the product_versions table for this, instead of the raw table:

1) it's only updated one per day (currently it is once-per-hour)
2) we can't restrict the repository without making some deeper changes which require more testing

I am assuming that #1 won't be a big problem (but throwing it out here now in case it is), but #2 makes it pretty unusable in it's current state.

So, since the product_builds service already hardcodes mozilla-specific config (repositories), I am temporarily adding hardcoding of Fennec*->mobile mappings as well.

For Socorro 7 (which will be released the week after 6), we can move all this stuff to the DB side and make it configurable so it won't break external installs.

This particular report is already broken for external installs, which is why I am OK adding more hardcoding for 6 :)
(In reply to Robert Helmer [:rhelmer] from comment #22)
> 1) it's only updated one per day (currently it is once-per-hour)

Actually, that could be a problem, as we'd like to see the current nightly for a day on that report.

> 2) we can't restrict the repository without making some deeper changes which
> require more testing

I have no idea what we need this repository thing for in this report.


But as I guess you're doing it differently now anyhow, I guess both are not affecting thhis solution now anyhow. :)
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #24)
> (In reply to Robert Helmer [:rhelmer] from comment #22)
> > 1) it's only updated one per day (currently it is once-per-hour)
> 
> Actually, that could be a problem, as we'd like to see the current nightly
> for a day on that report.


This is good to know, thanks. We might need to change strategy in this case, let's discuss in irc.


> > 2) we can't restrict the repository without making some deeper changes which
> > require more testing
> 
> I have no idea what we need this repository thing for in this report.

The FTP scraper picks up release metadata for everything posted, so we have been restricting it to just the major repositories (currently things like 'mozilla-central' are embedded in the source code, and should be configurable.)

Otherwise builds for all kinds of other repositories show up (dev branches like "oak", ESR, etc - basically all the different types of dir in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ - repository is really not the 100% correct name in many cases, it's more like a "conceptual branch" :) )
 
> But as I guess you're doing it differently now anyhow, I guess both are not
> affecting thhis solution now anyhow. :)

I think the solution for 6 will work just fine from a user PoV, this should be configurable (both to ease maintenance for us, and for non-mozilla installs.)
(In reply to Robert Helmer [:rhelmer] from comment #25)
> This is good to know, thanks. We might need to change strategy in this case,
> let's discuss in irc.

Sure. I won't be around on IRC before Monday, probably, it's already Saturday and weekend around here. ;-)

> Otherwise builds for all kinds of other repositories show up (dev branches
> like "oak", ESR, etc - basically all the different types of dir in
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ - repository is
> really not the 100% correct name in many cases, it's more like a "conceptual
> branch" :) )

Ah, it's "the part before the platfom in the FTP subdirectory name", which often corresponds to a repo. IIRC, releng calls it "branch".

> I think the solution for 6 will work just fine from a user PoV, this should
> be configurable (both to ease maintenance for us, and for non-mozilla
> installs.)

OK, good, that's what I care most about. :)
Commit pushed to stage at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/e2c036a569e87ef462b1fe9c3f10ab873425279d
bug 740829 - temporary manual override for Fennec and FennecAndroid
Merged into master: https://github.com/mozilla/socorro/commit/6146b43bd139e301ee9888dcc30db7db674cd1cc

Also merged into our stage branch (see previous comment).
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
> 1) it's only updated one per day (currently it is once-per-hour)

From the database side, there's no particular obstacle to running update_products() more often.   Actually, I was under the impression that we were already running it 4 times a day.  No?

> So, since the product_builds service already hardcodes mozilla-specific
> config (repositories), I am temporarily adding hardcoding of Fennec*->mobile
> mappings as well.

good luck on the whole FennecXUL vs. FennecAndroid thing.  My suggestion is that you just label them all "Fennec" and don't worry about Android vs. XUL.  The if-then logic for that is annoying.

> For Socorro 7 (which will be released the week after 6), we can move all
> this stuff to the DB side and make it configurable so it won't break
> external installs.

You open a bug for that yet?
Attached image qa - verified — 
QA verified on stage - the nightly builds table appears to be correctly populated now for FennecAndroid - https://crash-stats.allizom.org/products/FennecAndroid/builds

All other products also contain nightly builds data except for Camino which is expected.
Status: RESOLVED → VERIFIED
OS: Mac OS X → All
Hardware: x86 → All
Commits pushed to processor2012 at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/1d9eec10d50f35891b9d3d5382d093ae94a170cb
bug 740829 - temporary manual override for Fennec and FennecAndroid

https://github.com/mozilla/socorro/commit/6146b43bd139e301ee9888dcc30db7db674cd1cc
Merge pull request #500 from rhelmer/bug740829-android-nightly-report
Hmmmm.

Since we never opened a bug for the database work, I've completely forgotten what we were going to change to fix this.  I think I also needed a list of repositories, which I never got.

Help?
(In reply to [:jberkus] Josh Berkus from comment #32)
> Hmmmm.
> 
> Since we never opened a bug for the database work, I've completely forgotten
> what we were going to change to fix this.  I think I also needed a list of
> repositories, which I never got.

Filed bug 748194

Just to clarify, this follow-up is not urgent - it's just a cleaner way of doing this (we shouldn't be building reports right on top of the raw tables, and it should be configurable and not require code changes to adjust).
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: