411357 - Store Bugzilla or Issue-tracking IDs in database

Status: RESOLVED FIXED

(Socorro :: General, task, P2)

Description

[Samuel Sidler (old account; do not CC)]

Reported by morgamic, Jul 10, 2007

Right now there is no way to store bug information related to a crash
signature or group of crash signatures in the database.  There needs to be
a way to store this info so we can integrate our reporting system with
other applications.  

Another feature we could have in the future could be a notification system
to tell users when their issue has been resolved (or filed) so they can
optionally track it.

--

Comment 1 by bsmedberg, Jul 10, 2007

See
http://groups.google.com/group/mozilla.dev.quality/browse_frm/thread/109c6342ebcd9732/2327b70f05f4c299?lnk=gst&q=mapping+crash+reports&rnum=1#2327b70f05f4c299
for the newsgroup discussion.

My current thought is a simple mapping table:

bugsignatures
  signature     '@NS_InitXPCOM3'
  bugid         123456
  bugstatus     'FIXED'

This table would be pulled from bugzilla data on a cronjob. Right now I don't think it's necessary to automatically file bugs on crashes, although we might want to provide an easy way to manually file the bug (e.g. a link to enter_bug.cgi with relevant data pre-filled).

--

Comment 2 by bsmedberg, Jul 10, 2007

oh and the primary key would be signature/bugid... you can have multiple bugs for a single signature, and a single bug can be for multiple signatures (e.g.
platform-specific signatures that have a common cause)

--

Comment 3 by morgamic, Jul 10, 2007

(No comment was entered for this change.)

Labels: -Priority-High Priority-Critical

--

Comment 4 by morgamic, Jul 10, 2007

(No comment was entered for this change.)

Labels: -Priority-Critical Priority-Medium

--

Comment 5 by morgamic, Jul 10, 2007

(No comment was entered for this change.)

Labels: Hackathon

--

Comment 6 by morgamic, Jul 12, 2007

Simple table structure attached.

Some constraints we talked about:
* Only grab 2-3 bugs per status
* There are 5 statuses so that turns into 10-15 bugs retrieved total
* Only retrieve the top 100 signatures by product

Working on standalone script for using xml from bz to get ids.

--

Comment 7 by bsmedberg, Jul 12, 2007

Are those technical constraints? Displaying only one/a few bugs for crashes is one of the very common complaints about the current reports.

Could we just run a cronjob on bugzilla that queries all bugs and searches for a
whiteboard pattern, e.g. [@.*] and spews it out matches in XML?

--

Comment 8 by morgamic, Jul 12, 2007

Sure, those were ideas Jay had for limiting which bugs to pull out (since all might be excessive and generate a lot of load).  If you'd rather handle it via whiteboard or another field, that'd work too.

--

Comment 9 by bsmedberg, Jul 13, 2007

I was hoping a full tablescan on bugzilla twice a day wouldn't be too much load, but I don't really know anything about it. My preference, if it's technically feasible, is to have more data.

Talkback uses the whiteboard or the summary currently?

Comment 1

[Samuel Sidler (old account; do not CC)]

fwiw, I'd like to see the following layout when this is done:

  Rank   Signature   #   Win   Mac   Lin   Open Bugs   Closed Bugs   Query

The Query link will query all bugzilla (all open/closed, all products/components, comments, summaries, etc) for the signature.

Also, I think we should just update those bugzilla numbers twice a day or so (I'd be okay with three or four times even). Anything more than that would probably be too much.

If you're planning on only displaying a couple or three bugs per status (open or closed), choose the most recently modified ones. Or maybe most recently filed is better? I'm not sure, honestly. Maybe for RESOLVED, the most recently resolved ones are better and for OPEN the most recently opened ones are better, preferring NEW over UNCO.

For the bug numbers themselves, you should pull from the summary since that's where QA has been putting the information in for the form of: crash [@ Stack Signature]. But you probably don't need to search for the first "crash" part. That'll allow me to put "topcrash" sometimes instead.

I also see no reason we shouldn't check bugzilla for the top 500 or 1000 crashes.

I'll file a follow up about making it easier to file right out of the Socorro UI, but that's not priority right now.

I'm marking this bug as a P2 since it will really help QA instead of requiring us to query to see if a topcrash is filed.

Comment 2

[Samuel Sidler (old account; do not CC)]

(In reply to comment #1)
> I'll file a follow up about making it easier to file right out of the Socorro
> UI, but that's not priority right now.

Filed as bug 411448.

Comment 3

[Michael Morgan [:morgamic]]

Dave, we talked about this briefly in #bmo -- but could you reiterate what a good cron schedule would be for spamming bugzilla?  And how far apart should pings be?  Also, does it make sense to query bugzilla externally or should be use a shadow db to just query directly?  Any advice would be appreciated.  :)  Thanks.

Comment 4

[Dave Miller [:justdave]]

As long as each query you run completes before you run the next one and you give a couple seconds between them you'll probably be fine.  Querying externally is probably the safest from a security standpoint...  if you query the DB directly you have to jump through hoops to tell if a bug is a security bug or not.  Linking a crash to a security bug just screams "hey, this crash is dangerous" and gives people an idea where to look to try to exploit it.  Querying externally lets you retrieve just public bugs without having to jump through all kinds of hoops to tell if they're public or not.  If this ends up being a performance issue, we can always put another copy of Bugzilla running on another slave database somewhere.

Comment 5

[K Lars Lohn [:lars] [:klohn]]

need to develop a discovery program that can discover associations between
signatures and bugzilla bug ids.  This will probably require a many-to-many
relationship between signatures and bugs - may need signatures table

Comment 6

[(not currently active) Ted Mielczarek]

Might be an easy win here by letting people manually associate bug numbers with entries in the topcrash list. QA is doing this manually on wiki.mo right now:
http://wiki.mozilla.org/QA/Topcrashes

An automated solution would be awesome as well, but if you can get the DB backend together, and let them store that data in it, it'd be halfway there.

Comment 7

[Tony Mechelynck [:tonymec]]

(In reply to comment #6)
> Might be an easy win here by letting people manually associate bug numbers with
> entries in the topcrash list. QA is doing this manually on wiki.mo right now:
> http://wiki.mozilla.org/QA/Topcrashes
> 
> An automated solution would be awesome as well, but if you can get the DB
> backend together, and let them store that data in it, it'd be halfway there.

One more reason for this is that (IIUC) crash reports wouldn't be processed unless someone asks for them (or maybe, there is nothing else to do); and why ask for my latest report if it isn't to search for a bug about it, and write one up if I can't find any?

-- Of course, the cron job would have to treat different resolutions in various ways: follow RESO DUPL recursively, drop (temporarily) RESO INVA, RESO INCO and RESO WONT, maybe move RESO FIXE to a different list, reenter REOP into the list (undoing what had been done before for the same bug), ...

Comment 8

[K Lars Lohn [:lars] [:klohn]]

I really want to see an automated way for Socorro to discover relationships between Bug IDs and signatures.

Is there a uniform location in bugzilla where a crash signature might appear?  Would I find it only embedded in text in the comments section?  Does Bugzilla offer an RSS feed of comments to bugs?

I'm envisioning a system with a script run by cron that pulls an RSS feed of recent comments from Bugzilla.  It looks for strings like "sig:aCrashSignature" embedded in the comment.  On finding such a string, it locates the signature within the Socorro database and associates it with the Bug ID from the RSS entry.  All the developer has to do is mention a bug signature in a comment in a standard format to enable Socorro to discover the relationship.

I fear, though, that Bugzilla doesn't offer such an RSS feed...

Comment 9

[Tony Mechelynck [:tonymec]]

In reply to comment #8:
You might look in the comments for a string such as bp-37b8e690-5b1c-4eb7-935f-790120081115 which Bugzilla makes into a link to a Breakpad report. Other than that... nothing that I know of.

Comment 10

[timeless]

bugzilla searches can be subscribed to as rss feeds.

crash signatures should be in the summary.

they should be of the form [@ foo::bar] or [@ foo::bar - pie::yay] there are some minor variations and the exact amount of whitespace is somewhat variable.

note that the annoying mac devs insist on trying to force people to stick crash reports into attachments. they're hopefully being ignored. if they succeed, a stack trace could be stored as an attachment in text/rtf (i.e. not plain text, really rtf w/ rtf meta bits all over). for now, just ignore them :). summaries should have signatures.

Comment 11

[(not currently active) Ted Mielczarek]

I guess this got fixed, since bug 464934 was fixed. :)