411354 - Add ability to search by build ID

Status: VERIFIED FIXED

(Socorro :: General, task, P1)

Description

[Samuel Sidler (old account; do not CC)]

Reported by ispiked, Jun 29, 2007

Need to be able to search based on a given build ID. Probably need to
expland this to do ranges of build IDs, not just a single one.

TODO: For some reason this doesn't work in the case where a product is not
selected.

--

Comment 1 by bsmedberg, Jul 03, 2007

(No comment was entered for this change.)

Labels: -Type-Defect Type-Enhancement Milestone-3

--

Comment 2 by morgamic, Nov 02, 2007

Bumping this to M2.  This is important to gain information per build about MTBF -
Mean Time Between Failure.  It gives us insight on whether the build is improving overall or if its getting worse over time.  That trend helps us gauge the robustness of the product.

We are exploring other points of entry for this data, but so far this is the best.

Labels: -Milestone-3 Milestone-2

--

Comment 3 by bsmedberg, Nov 02, 2007

Yeah, this has been brought up as important for the topcrash reports: limiting the query by build-date (buildid) gives much more accurate results.

It's easy enough to add to the search form.

--

Comment 4 by morgamic, Nov 29, 2007

Working on this.

Comment 1

[Samuel Sidler (old account; do not CC)]

This is pretty important now that we're about to release an RC2 of Firefox 3. We need a way to tell the difference between RC1 and RC2 on the top crash page, specifically. I can fake it if I can search by build ID though. That'd be more than good enough for now...

Comment 2

[Justin Gallardo]

Committed support for this in revision 391. Once implemented, we should be able to access the page by visiting http://crash-stats.mozilla.org/topcrasher/byversion/Firefox/3.0/<buildId>.

Comment 3

[Samuel Sidler (old account; do not CC)]

What's the timeframe for getting this live?

Comment 4

[Michael Morgan [:morgamic]]

Justin did you back out the aggregate stuff so its on a branch?  If so we can stage this today.

Comment 5

[Justin Gallardo]

I did back out the aggregate stuff last week. Should be ready to go.

Comment 6

[Michael Morgan [:morgamic]]

We need to add this to the top crasher listing as well for test builds.

Comment 7

[Michael Morgan [:morgamic]]

From our meeting on Friday this would be a basic text field on the search page where the build could be manually entered.

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to comment #7)
> From our meeting on Friday this would be a basic text field on the search page
> where the build could be manually entered.

as a range?  suggested in comment 0 - which would be very helpful 
eg.  >= 20081118

Comment 11

[Wayne Mery (:wsmwk)]

Is there further roadmap information on making this possible?   

The most recent accounting I find is TBD of https://wiki.mozilla.org/Breakpad/2009Q3RoadMap  It would be useful today of course, but this will become much more important as Thunderbird goes into shorter development cycles - https://wiki.mozilla.org/User:Dmose/Tb_Post-3.0_Scratchpad

Comment 12

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The lack of this feature makes tracking topcrashes in nightlies (trunk or branch) nearly impossible; it's why I don't even bother, and only look at betas.

Comment 13

[Austin King [:ozten]]

Attachment: First attempt to add search by build to query form

Patch adds a build_id field to the query form, updates the rules for a "too general" search to allow for either a prod/vers OR a build number, and adds build clause to search SQL.

Comment 14

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Comment on attachment 420218 [details] [diff] [review]
First attempt to add search by build to query form

Code looks fine.  r+

Comment 15

[K Lars Lohn [:lars] [:klohn]]

warning: build is not an indexed column in the reports partitions.  In order for this to work reasonably in production, each partition will have to get this index.

Comment 16

[Austin King [:ozten]]

(In reply to comment #15)
Great catch. I'll run the SQL by you to create the index.

Comment 17

[Austin King [:ozten]]

Sending        webapp-php/application/libraries/MY_SearchReportHelper.php
Sending        webapp-php/application/models/common.php
Sending        webapp-php/application/views/common/query_form.php
Sending        webapp-php/css/screen.css
Transmitting file data ....
Committed revision 1675.

Comment 18

[Austin King [:ozten]]

Attachment: First stab at Updates to schema.py

Adds index creation to schema.py

Comment 19

[Austin King [:ozten]]

Attachment: A series of index creations for existing reports partitions

A series of index creation SQL statements for existing partitions.

Comment 20

[Austin King [:ozten]]

Here are the affected queries, with 2 permutations for analyzing the queries:
http://pastebin.mozilla.org/695130

Comment 21

[Frank Griswold [:griswolf] [:fgriswold]]

Attachment: Drop and create some indexes on existing reports partitions

After some examination of 
  EXPLAIN (one of several queries involving the build table) ;
Decided two indexes are useful: One is new, one is an extension of an old index. 
Also found one redundant index.

Indexes cannot be ALTERED to have new columns, so both the redundant index and the extended one are dropped, then the new and extended ones are built.

Note that the first few partitions are not useful: Two are too small to matter, and the one for 2009-02-02 is both too large, and does not actually contain (only) data with date_processed that it should. Those three partitions are not indexed.

Comment 22

[Frank Griswold [:griswolf] [:fgriswold]]

Comment on attachment 420355 [details] [diff] [review]
First stab at Updates to schema.py 

Superseded by changed requirements. New code checked in: r1684

Comment 23

[Austin King [:ozten]]

Comment on attachment 420451 [details]
Drop and create some indexes on existing reports partitions

Sounds good.

Comment 24

[Frank Griswold [:griswolf] [:fgriswold]]

Attachment: Drop and create some indexes on existing reports partitions

Drat. That should teach me to test a file that hasn't been written to disk, and then upload that file, still in its bad state. Sigh. But it probably won't.

Comment 25

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Is this supposed to be working yet?  I see a build ID form in the query field, but I can't get it to return any results, e.g.:
http://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A3.7a1pre&date=&range_value=1&range_unit=weeks&query_search=signature&query_type=exact&query=&build_id=2010011000&do_query=1
is empty.

Comment 26

[Wayne Mery (:wsmwk)]

buildid from one of my crashes pulled up a fine list. 20091220050304

2010011000 is a tad short?

Comment 27

[Austin King [:ozten]]

(In reply to comment #25)
Yes, this is deployed and should be ready.

Comment 28

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Is it possible to search with the initial substring of the build ID so we can search for all crashes for a specific day?

Comment 29

[Austin King [:ozten]]

(In reply to comment #28)
Yes, reopening and setting to next release.

Comment 30

[Laura Thomson :laura]

Should tackle this modification with the other search bugs

Comment 31

[Laura Thomson :laura]

What is needed to close this: search on buildid prefix.  

Will our planned ES implementation Just Fix This?

Comment 32

[[DEACTIVATED] Adrian Gaudebert]

The current implementation of the new search API does search by build id. Searching into a range shouldn't be difficult to add, I just need to know if this build id is an integer, and we would have to change the ES mapping of this field so the range function is possible. 

If the build id is not an integer, the range would not be possible. We could search into the build date though, or use date range. 

This is only using ES, idk how hard it would be to implement that with the Postgres version.

Comment 33

[Laura Thomson :laura]

Staying in 2.0, or going to 2.1?  I don't mind if it's not supported in the PG version.

Comment 34

[[DEACTIVATED] Adrian Gaudebert]

Searching for one build id is ok, searching for ranges should be pushed to 2.1.

Comment 35

[[DEACTIVATED] Adrian Gaudebert]

Searching by a range of build ids is pushed to 2.1.

Comment 36

[[DEACTIVATED] Adrian Gaudebert]

First part of this problem is solved (being able to search for one or several build ids). Was part of Socorro 2.0. 

Second part of this bug (search by range of build ids) will be covered by bug 677434.

Comment 37

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Post-fix screenshot (prod)

Comment 38

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED: https://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A7.0&range_value=1&range_unit=weeks&date=09%2F29%2F2011+21%3A55%3A10&query_search=signature&query_type=contains&query=&reason=&build_id=20110922153450&process_type=any&hang_type=any&do_query=1