661941 - (sisyphus-webapp) sisyphus web app

Status: RESOLVED FIXED

(Testing Graveyard :: Sisyphus, defect)

Description

[Mark Côté [:mcote]]

A web UI for sisyphus.

Comment 1

[Mark Côté [:mcote]]

Attachment: Very early worker admin view

Here's a very first crack at a UI.

Some notes:
- At the moment, this is a static site that fetches JSON data via AJAX calls.  The contents of sisyphus/python/sisyphus/webapp/html should be served at one level lower than the django (so if django serves the bughunter app as http://<server>/bughunter/, the workers page should be http://<server>/workers.html).  A little futzing with the webserver might be needed to get this right, since django wants to serve one level below /bughunter/ too--but not for any good reason, since that index is currently broken anyway.

 - Alternatively, we could go with a templated version that is filled in with the initial data, and all subsequent updates would use AJAX calls.  There are various tradeoffs here.

- The data is refreshed every 5 seconds.  If you're spending a lot of time examining the page and want it to stop refreshing, you can always use the firebug console to call "app.stopRefreshing()" (I can hook a proper UI up to this if desired).

- The site uses jQuery, jQuery-UI, the jQuery datatables plugin, and backbone.js.

- The old site should continue to work as before.

Comment 2

[Mark Côté [:mcote]]

Attachment: Very early worker admin view

New patch with better colours and a few improvements.

Comment 3

[Bob Clary [:bc] (inactive)]

https://bugzilla.mozilla.org/show_bug.cgi?id=661941 config changes

Comment 4

[Mark Côté [:mcote]]

Attachment: Worker admin view

This has the worker list and worker summary views now.  The header will change after I add a log-in and the rest of the views (crashes etc).  Lots still to come.

Comment 5

[Bob Clary [:bc] (inactive)]

Comment on attachment 539358 [details] [diff] [review]
Worker admin view


>diff --git a/python/sisyphus/webapp/bughunter/views.py b/python/sisyphus/webapp/bughunter/views.py
>+@json_view
>+def worker_summary_api(request):
>+    worker_types = ['builder', 'crashtest', 'unittest']
>+    worker_data  = {}
>+
>+    worker_rows = models.Worker.objects.all()

worker_types, worker_data and worker_rows not used in this function.

Applies well and runs on qp-bughunter after nginx/bughunter restarts.

Comment 6

[Mark Côté [:mcote]]

I removed worker_summary_api() altogether, since it isn't used at the moment.  I'll clean up and rename the admin() function when we decide to remove the old UI.

I also changed the worker-list row text colour to red if the machine's state is zombie or dead.

Pushed as http://hg.mozilla.org/automation/sisyphus/rev/91aa900252d3

Comment 7

[Mark Côté [:mcote]]

Attachment: Update state-highlighting on refresh

Fixes the problem with the states not highlighting when the page refreshes.

Comment 8

[Mark Côté [:mcote]]

Pushed as http://hg.mozilla.org/automation/sisyphus/rev/5911446be41a

Comment 9

[Mark Côté [:mcote]]

Attachment: Worker details/logs view

This provides a worker details/log view, accessible through hyperlinks in the hostname column of the workers table.

There are a few little things I would like to clean up, but most are internal, so you may as well take a look right now and comment on the UI design.

I also enabled site-wide caching with a 10-second timeout.  This is done to mitigate db accesses if several people load the site at roughly the same time.  Not entirely sure if it's totally necessary, though, so we can take it out if the load doesn't seem bad.

Comment 10

[Mark Côté [:mcote]]

Attachment: Logs views, take 2

Provides a combined logs view in addition to per-worker logs. Various other tweaks.

Comment 11

[Mark Côté [:mcote]]

Attachment: Logs views, take 3

This fixes a few things:

- datetime picker is removed. Instead they are now regular text-entry boxes with the HTML 5 pattern element to restrict the entry to <YYYY>-<MM>-<DD>[T<HH>:<MM>[:<SS>]] -- in other words, you can enter just a date, or a date plus hour and minute, or a date plus hour, minute, and second. Invalid dates/times disable form submission.

- default sorting on the workers table is by ascending hostname

- sorting preference on both the workers and worker-summary tables is preserved via localStorage.

Replaces previous patch.

Comment 12

[Mark Côté [:mcote]]

Patch pushed as http://hg.mozilla.org/automation/sisyphus/rev/520c36cc5acf along with tiny change to make default sort order ascending on worker-summary page.

Also, forgot to note that this patch requires an external library: just run "easy_install wadofstuff-django-serializers".  This provides a superior JSON deserializer for Django objects.

Comment 14

[Mark Côté [:mcote]]

Attachment: Add parameters to #crashes/date. Fixed log clearing.

UI for #crashes/date params, including new-crashes-only. Fixed log clearing.

Comment 15

[Mark Côté [:mcote]]

Pushed as http://hg.mozilla.org/automation/sisyphus/rev/6943f6805539.

Noted problem with refreshing with no parameter changes; fix coming separately.

Comment 16

[Mark Côté [:mcote]]

Attachment: Fix refresh.

Fixed refresh button when parameters are unchanged.

Comment 17

[Bob Clary [:bc] (inactive)]

Comment on attachment 555554 [details] [diff] [review]
Fix refresh.

fixes the refresh issue.

Comment 18

[Mark Côté [:mcote]]

Pushed as http://hg.mozilla.org/automation/sisyphus/rev/4ed1bcfcb168

Comment 19

[Jonathan Eads ( :jeads )]

Attachment: Integrated SQL encapsulation/webservice for bughunter Views

I integrated the SQL encapsulation module I've been working on.  It's called datasource, it's a generic way of managing multiple data sources.  

To install datasource:
git clone git://github.com/jeads/datasource
cd datasource
python setup.py install

There is a webservice in view.py sisyphus/webapp/bughunter that makes use of the module.  Here are some sample urls to try, you will need to login to bughunter first, they should all return json:

These three examples are from bc's last set of use cases:

Example 1
####
#urls by signature
####
http://localhost/bughunter/bughunter/views/urls?p=2011-08-25&rq=NPSWF32.dll@0x1ed283%20NPSWF32.dll@0x1795d6%20NPSWF32.dll@0x1ed76e%20NPSWF32.dll@0xd8dba%20NPSWF32.dll@0xe33b7

Example 2
http://localhost/bughunter/bughunter/views/crashdetail?p=2011-08-25&rq=NPSWF32.dll@0x39c163%20NPSWF32.dll@0x1ed2a5%20NPSWF32.dll@0x1795d6%20NPSWF32.dll@0x1ed76e%20NPSWF32.dll@0xd8dba

Example 3
####
#urls by fatal_message
####
http://localhost/bughunter/bughunter/views/fmurls?p=2011-08-25&rq=ABORT:%20CRT%20ASSERT%20dbgheap.c%281279%29%20:%20Assertion%20failed:%20_CrtIsValidHeapPointer%28pUserData%29

Example 4:
This is not a useful statement but it's easy to play with and see how the interface works.
http://localhost/bughunter/bughunter/views/crashes?r=100


Take a look at sisyphus/webapp/procs/bughunter.json to see the SQL.  You can add new views to the service by adding SQL in the view hash in bughunter.json.  You should not have to modify any python for new views that correspond to single SQL statements.  There's a big comment in sisyphus/webapp/bughunter/views.py in the method get_bhview() that explains how the service works.

A few notes about datasource:

I integrated it into bughunter by initializing a MySQL data hub in settings.py.  This was done so we would not need to hard code db info in another location.  You can also use datasource from any server side script or from the command line but you will need to set a datasource.json file somewhere.  There is a README in datasource that explains this in detail.

A few notes about SQL injection issues:
Passing in placeholders in the url should be safe but SQL replacements are not.  I'm looking into some strategies to resolve the security issue but have not finished with it yet.

Comment 20

[Bob Clary [:bc] (inactive)]

Comment on attachment 559386 [details] [diff] [review]
Integrated SQL encapsulation/webservice for bughunter Views

initial drive by comments...

missing procs/bughunter.json ?

>diff --git a/python/sisyphus/webapp/bughunter/urls.py b/python/sisyphus/webapp/bughunter/urls.py
>                        (r'^unittests/$', views.unittests),
>+                       (r'^views/*', views.get_bhview),

shouldn't the pattern be ^views/.* ?

>diff --git a/python/sisyphus/webapp/bughunter/views.py b/python/sisyphus/webapp/bughunter/views.py
...
>+   To add a new service method add a SQL to bughunter.json and your

nit: your -> you are or you're

Comment 21

[Jonathan Eads ( :jeads )]

Attachment: Integrated SQL encapsulation/webservice for bughunter Views

Integrated SQL encapsulation/webservice for bughunter Views

I integrated the SQL encapsulation module I've been working on.  It's called datasource, it's a generic way of managing multiple data sources.  

To install datasource:
git clone git://github.com/jeads/datasource
cd datasource
python setup.py install

There is a webservice in view.py sisyphus/webapp/bughunter that makes use of the module.  Here are some sample urls to try, you will need to login to bughunter first, they should all return json:

These three examples are from bc's last set of use cases:

Example 1
####
#urls by signature
####
http://localhost/bughunter/bughunter/views/urls?p=2011-08-25&rq=NPSWF32.dll@0x1ed283%20NPSWF32.dll@0x1795d6%20NPSWF32.dll@0x1ed76e%20NPSWF32.dll@0xd8dba%20NPSWF32.dll@0xe33b7

Example 2
http://localhost/bughunter/bughunter/views/crashdetail?p=2011-08-25&rq=NPSWF32.dll@0x39c163%20NPSWF32.dll@0x1ed2a5%20NPSWF32.dll@0x1795d6%20NPSWF32.dll@0x1ed76e%20NPSWF32.dll@0xd8dba

Example 3
####
#urls by fatal_message
####
http://localhost/bughunter/bughunter/views/fmurls?p=2011-08-25&rq=ABORT:%20CRT%20ASSERT%20dbgheap.c%281279%29%20:%20Assertion%20failed:%20_CrtIsValidHeapPointer%28pUserData%29

Example 4:
This is not a useful statement but it's easy to play with and see how the interface works.
http://localhost/bughunter/bughunter/views/crashes?r=100


Take a look at sisyphus/webapp/procs/bughunter.json to see the SQL.  You can add new views to the service by adding SQL in the view hash in bughunter.json.  You should not have to modify any python for new views that correspond to single SQL statements.  There's a big comment in sisyphus/webapp/bughunter/views.py in the method get_bhview() that explains how the service works.

A few notes about datasource:

I integrated it into bughunter by initializing a MySQL data hub in settings.py.  This was done so we would not need to hard code db info in another location.  You can also use datasource from any server side script or from the command line but you will need to set a datasource.json file somewhere.  There is a README in datasource that explains this in detail.

A few notes about SQL injection issues:
Passing in placeholders in the url should be safe but SQL replacements are not.  I'm looking into some strategies to resolve the security issue but have not finished with it yet.

Comment 22

[Jonathan Eads ( :jeads )]

Attachment: fixed settings.py import bug

Fixed settings.py import bug

Comment 23

[Mark Côté [:mcote]]

Comment on attachment 559508 [details] [diff] [review]
fixed settings.py import bug

This is pretty cool!  Looks like it'll improve modularity greatly.

I just have one (tiny, tiny) nit-pick: I find "'%s' % foo" to be more readable than "'%s'%foo".

Comment 24

[Jonathan Eads ( :jeads )]

Attachment: New Bughunter View UI

This patch is not ready for review but I wanted to make sure my changes are in bugzilla in case of disaster striking.

Comment 25

[Jonathan Eads ( :jeads )]

Attachment: New Bughunter View UI

Please take a look at the README in python/sisyphus/webapp/README.  There are a bunch of notes about the patch.  Also, bc, let me know when you apply the patch to qp-bughunter, there will likely be some local config issues that I can help sort through. As soon as it's up and running I would like to set up a meeting with bc, mcote (provided you are still available), and ctalbert, to go over a variety of things.  There are some significant deviations from the functional spec and a whole host of issues I would like to discuss.
The patch is artificially large, I changed the location of doc root to python/sisyphus/webapp/media and copied the admin app there rather than doing an actual move.  I'm not sure if this is the right thing to do, you might be using that directory for something else that i don't fully understand but the media directory is typically the home of all assets: CSS, js, images, and doc root accessible html. 
Jeads

Comment 26

[Jonathan Eads ( :jeads )]

Attachment: New Bughunter View UI

I forgot to "hg qrefresh" so I was missing modified files.

Comment 27

[Jonathan Eads ( :jeads )]

Attachment: New Bughunter View UI

Rebuilt patch after running "hg qrefresh"

Comment 28

[Jonathan Eads ( :jeads )]

Attachment: New Bughunter View UI

added missing README.

Comment 29

[Jonathan Eads ( :jeads )]

Attachment: New Bughunter View UI

Built a clean patch from a vm that matches the qp-bughunter environment.  Please let me know when you apply the patch, I would like to have a vidyo conference once it's up and running.  Thanks
Jeads

Comment 30

[Jonathan Eads ( :jeads )]

Attachment: View UI Fixes

1.) Fixed view pane resizing issues.  Any pane can be resized at any time and the correct layout is preserved.
2.) Added better defaults for table size and rows displayed.
3.) Fixed the search modal pop up bug.
4.) Modified all search boxes to work independently.
5.) Added regex support in the search box.
6.) Added column toggling.  This feature can be accessed by selecting the column dropdown button to the right of "Show entries".
7.) Reduced the size of the top Bughunter panel.
8.) Converted media link columns (crashreport, log) to retrieve data from qp-bughunter media.  If there are more links under media just let me know the column names.

Comment 31

[Jonathan Eads ( :jeads )]

Attachment: View UI Fixes

You will need to run "manage.py build_nav" and bounce the server after installing this patch.

Patch includes:
1.) fixed the control panel data initialization issue
2.) added context menus to table cells containing links.  These menus give the user options for launching the cell link in a new page.
3.) cleaned up the font size issues
4.) fixed a handful of additional UI bugs I found

Comment 32

[Jonathan Eads ( :jeads )]

Attachment: New View UI

INSTALL NOTES: You will need to run "manage.py build_nav", restart the server, and clear your browser's cache.  This patch contains the following changes:

1.) Resolved "No Data Available" issue.  This was caused by a character escaping problem.  Very special handling was required for URLs with non english characters in them.  To prevent the browser/jquery/django from double utf8 encoding them I had to use HTTP POST instead of GET.  This caused an issue with the csrf token that the application uses.  This has been resolved but requires the csrf token to be set in the cookie and passed in the POST header.  This is why you will likely need to clear you browser's cache to get it to run correctly.

2.) Added date range to "New Crashes"
3.) Resolved the 'Too much data returned' issue.  I think this was actually being caused by SQL statements that had an 'ORDER BY signature' which seems to perform horribly probably due to partial index.  All SQL statements have a hard limit of 5000 appended to them.  The only signature that required special handling was '(no signature)', when found it is not converted into a link.  This is far from bullet proof handling but will work for now.

Comment 33

[Bob Clary [:bc] (inactive)]

Attachment: datetime-1.patch

remove the now extraneous T from Mark's UI.

Comment 34

[Bob Clary [:bc] (inactive)]

Attachment: datetime-2.patch

format MySQL datetimes in selects. Jeads, you said you were rewriting this file, but I thought I would include it here for completeness.

Comment 35

[Bob Clary [:bc] (inactive)]

Attachment: gzipped-encoding.patch

I added gzip static encoding to the nginx configuration but needed to remove the .gz extension from the links in order for the browser to display the logs and crash reports rather than attempting to download them.

Comment 36

[Jonathan Eads ( :jeads )]

Attachment: Query Validation

This patch requires running "manage.py build_nav" and restarting the webserver.

1.) Modified all database queries to take new data types into account and to optimize performance and normalize query syntax for dynamic WHERE clause generation.

2.) Modified the admin application to take into account the new DATETIME data type.

3.) Modified signals/control panel to send the date range along with all signals in a post.

NOTE: To see the completely assembled SQL that the UI executes at run time do the following:

1.) Create the file /var/log/django/sisyphus.error.log on your local machine.
1.) Uncomment the three lines at the top of view.py that redirect stdout.
2.) Uncomment the lines containing the string "debug_show". 

You can "tail -f /var/log/django/sisyphus.error.log" to see all of the queries, with their full WHERE clause.  This is very helpful in detecting bugs and determining what the UI is doing.

Comment 37

[Jonathan Eads ( :jeads )]

Attachment: New View UI

This patch requires an update of datasource:
git clone https://github.com/jeads/datasource

1.) Added mysql string escaping calls to gaurd against SQL injection

2.) Contains query updates and a fix for the refresh bug.

Comment 38

[Jonathan Eads ( :jeads )]

Attachment: Query Validation

Missed the patch checkbox... Jeads

Comment 39

[Bob Clary [:bc] (inactive)]

jeads, something is wrong with these patches. everything appears borked.

Comment 40

[Jonathan Eads ( :jeads )]

(In reply to Bob Clary [:bc:] from comment #39)
> jeads, something is wrong with these patches. everything appears borked.

Let me know when you're available and I will troubleshoot.

Comment 41

[Bob Clary [:bc] (inactive)]

https://hg.mozilla.org/automation/sisyphus/rev/3fbbcc62f13c
datetime type change follow up - missed new to be reproduced crashes.

https://hg.mozilla.org/automation/sisyphus/rev/29241fcb6d29
Initial webapp landing

https://hg.mozilla.org/automation/sisyphus/rev/e829b363ffde
Query validation

https://hg.mozilla.org/automation/sisyphus/rev/e154ee0a466a
Strip .gz extension from urls so that nginx will serve the content as gzip encoded text.

https://hg.mozilla.org/automation/sisyphus/rev/523a4367e6a1
Remove extraneous T separator from datetime stamps in control panel widgets.

https://hg.mozilla.org/automation/sisyphus/rev/6da2a2afbec5
Enable debugging output via environment variable.

Comment 42

[Jonathan Eads ( :jeads )]

Attachment: Adjusted the date range for views

Adjusted the date range for views so that the end date is always tomorrow.  Also changed the python unquote function to urllib.unquote from urllib.unquote_plus.

Comment 43

[Jonathan Eads ( :jeads )]

Attachment: Added the ability to display signals sent or received.

Added a new panel on each view for display of signal data.

Comment 44

[Jonathan Eads ( :jeads )]

Attachment: Column order in view tables matches the column order in SQL

This patch requires an update of datasource:
git clone https://github.com/jeads/datasource

1.) Modified the data structure returned by the web service to deliver an additional array containing the column names, where the order of elements matches the order in the SELECT.  The view column order can now be controlled by modifying the SQL in procs/bughunter.json.

2.) bc noticed the space escape char used in the js, %2, should actually be %20. This change has been made.

3.) removed the special handling for "(no signature)"

Comment 45

[Jonathan Eads ( :jeads )]

Attachment: Merged new crashes and crashes, aggregated platform data

You will need to run "manage.py build_nav" and restart the webserver after this patch is installed.

1.) Added a placeholder'ish strategy to the SQL.
2.) Merged the views "New Crashes" and "Crashes"
3.) Aggregated the platform data in Crashes.  Set this as the new default view.
4.) Added a checkbox to the Crashes "Control Panel" to filter for new crash signatures only.
5.) Added code to update the default date range every 5 minutes.
6.) Added agreed upon handling for blank start/end dates and a maximum date range of 60 days.
7.) The Open Connection option menu now only contains views that share signals with the view from which the menu was launched.

Comment 46

[Jonathan Eads ( :jeads )]

Attachment: Added Parent/Child Relationships To Views and a bunch of fixes

You will need to run "manage.py build_nav" and clear your browser's cache before running this patch due to the change to all of the control panels.

1.) Added parent/child relationships to views.

2.) Fixed the column sort null bug in datatables.js

3.) Added extensive handling for corrupted dates and a date range reset button.

4.) Added messaging to control panel when server resets date range.

5.) Restyled all control panels to assist user in filling out form.

6.) Modified fgmenu.js to take an onClose callback.

7.) Modified the signal display to display date ranges on the views throughout all application states.

Comment 47

[Jonathan Eads ( :jeads )]

Attachment: Added Parent/Child Relationshipts to Views and fixed bugs

Fixed replace() call on undefined bug in DataAdapterCollection.js

Comment 48

[Bob Clary [:bc] (inactive)]

https://hg.mozilla.org/automation/sisyphus/rev/0687a4582e9c
https://hg.mozilla.org/automation/sisyphus/rev/312e819f2ed1
https://hg.mozilla.org/automation/sisyphus/rev/f922baf48f49
https://hg.mozilla.org/automation/sisyphus/rev/bdcc68bf4155
https://hg.mozilla.org/automation/sisyphus/rev/114e06282bde

Comment 49

[Jonathan Eads ( :jeads )]

Attachment: Enabled window to window signals

Added handling for sending signals from between windows.  You will need to clear your browser's cache before testing this patch.

Comment 50

[Jonathan Eads ( :jeads )]

Attachment: Enabled window to window signals

Fixed several bugs in the last patch.  You will need to clear your browser's cache before running this patch.

Comment 51

[Bob Clary [:bc] (inactive)]

Comment on attachment 574757 [details] [diff] [review]
Enabled window to window signals

Looks good.

Comment 52

[Bob Clary [:bc] (inactive)]

Attachment: bughunter-service-method-prefork.patch

Use prefork instead of threaded to get around the threading issues in python-mysql, and change the logging to use the proper arguments for outlog and errlog and to place them in /var/log/bughunter/. This also sets the flup debug variable to the value set in the environment variable SISYPHUS_DEBUG so we'll automatically turn flup debugging output on with the rest.

Comment 53

[Bob Clary [:bc] (inactive)]

Landed https://hg.mozilla.org/automation/sisyphus/rev/61227dd1484a which is an unrelated change to mozconfigs, then

https://hg.mozilla.org/automation/sisyphus/rev/ffff2650e6d6
https://hg.mozilla.org/automation/sisyphus/rev/a3ef940771bf

Comment 54

[Jonathan Eads ( :jeads )]

Attachment: Merged url views, added socorro_id signals, merged site test crash and crash details

You will need to run "python manage.py build_nav" restart the server and clear your browser's cache before installing this patch.

1.) Merged URL views into a single view.
2.) Added platform data aggregation to URL views.
3.) Removed unescaped escape characters displayed in fatal messages and signatures displayed in the signal display bar.
4.) Included build_cpu_name with cpu_name and summarized their data display
5.) Added socorro_id signal.
6.) Added handling for enter/return/tab to the control panel
7.) Merged site test crash with crash details.  
8.) Modified crash detail query use a temporary table strategy.
9.) Added handling for conditional display of columns in tables.

NOTES:
  I was unable to get logging working using the arguments to manage.py in service/bin/bughunter.  I had to comment out the "outlog" line and add the stdout redirect back to views.py.  I would much prefer to use bc's strategy utilizing manage.py to do the redirect but for some reason it does not work on my local machine.  Will investigate and fix in a subsequent patch.

  I did not get around to getting the socorro_id display to look pretty, I'm going to clean up the signal column names and the socorro_id signal in a subsequent patch.

Jeads

Comment 55

[Jonathan Eads ( :jeads )]

Attachment: Merged url views, added socorro_id signals, merged site test crash and crash details

You will need to run "python manage.py build_nav" and restart the web server before testing this patch.

This patch fixes:
1.) The url summary/fatal message signal bug.
2.) Un-commented/removed stdout redirect in views.py
3.) View names are now set/displayed as soon as user reloads one.
4.) Changed position of "Total Counts" in the URL summary.

Comment 56

[Jonathan Eads ( :jeads )]

Attachment: Resize column widths when window is resized

This patch re-sizes the column widths in tables when a user re-sizes a window.  Performance becomes a bit sluggish when 3+ views are open in a single window.  This might work for now, I think there are a few options for speeding things up in a future patch.

Comment 57

[Jonathan Eads ( :jeads )]

Attachment: Added signals: address, pluginfilename, pluginversion, and exploitability

You will need to run "python manage.py build_nav", clear your browser's cache, and restart the server before running this patch.

1.) Added new signals: address, pluginfilename, pluginversion, and exploitability.  They are accessible in the "Crash Details" view as clickable links that the other views will respond to.  The new signals are present in all of the control panels.

2.) Set the datetime column in "Crash Details" back to displayed by default.

3.) Decreased the default data date range to include the last 24 hours.

4.) Added separate display fields for signals sent and received.

Comment 58

[Jonathan Eads ( :jeads )]

Attachment: Fixes temptable bug

You will need to restart the server before testing this patch.  This patch forces all dynamically created temporary tables to have unique names and be deleted when a web service method is finished with them.

Comment 59

[Jonathan Eads ( :jeads )]

Attachment: Fixes new signals control panel bug and crash detail SQL bug

You will need to clear your browser cache and restart the server before testing this patch.

1.) Fixes new signals not being sent from control panel bug.

2.) Fixes Crash Detail SQL bug.

Comment 60

[Jonathan Eads ( :jeads )]

Attachment: Added views for assertions.

You will need to run "python manage.py build_nav", restart the server, and clear your browser's cache before running this patch.

1.) Added new views for assertions: Assertion, Assertion URL Summary, and Assertion Detail.

2.) Removed special handling for & in URLs.

Comment 61

[Jonathan Eads ( :jeads )]

Attachment: Assertion Fixes

You will need to run "python manage.py build_nav", restart the server, and clear your browser's cache before running this patch.

1.) Put SiteTestRun back in SQL that uses fatal_message.

2.) Change Crash/Assertion Details title to Related Crashes/Assertions

3.) Removed ORDER BY clause from Related Assertions' view.  Also removed the 
ORDER BY clause from Related Crashes' view due to similar performance problems.

4.) Add total occurrence column to Assertions and Assertions URL Summary

5.) In connections option menu ordered Assertions and Crashes together

6.) In preparation for adding Unit Test views re-organized all webservice methods,
added st suffixes to web service data adapters and SQL methods.

Comment 62

[Jonathan Eads ( :jeads )]

Attachment: Added Unit Test Assertions

You will need to run "python manage.py build_nav", restart your server, and clear your browser's cache before testing this patch.

1.) Added unit test assertion views.
2.) Added top level to the view navigation menu for Site Tests and Unit Tests.
3.) Added the signature/socorro id to the control panel of the Socorro View.
4.) Removed the ORDER BY clause from the socorro SQL due to performance problems when returning all signatures of a particular type.

Comment 63

[Bob Clary [:bc] (inactive)]

https://hg.mozilla.org/automation/sisyphus/rev/cab0c135009e
Merged url views, added platform data aggregation, added signal for socorro id, added enter/tab handling to control panel, merged site test crash with crash details

https://hg.mozilla.org/automation/sisyphus/rev/9ec392f21d9c
Dynamically resize column widths when the window is resized

https://hg.mozilla.org/automation/sisyphus/rev/72061e383c6d
Added new signals: address, pluginfilename, pluginversion, exploitability

https://hg.mozilla.org/automation/sisyphus/rev/f6c810d63e58
Adds unique names to temp tables and drops them after a query is finished

https://hg.mozilla.org/automation/sisyphus/rev/d3c4ff9b7938
Fixes new signals not sent from control panel and crash detail SQL bug

https://hg.mozilla.org/automation/sisyphus/rev/8f7e0a2088c6
Added new views: Assertions, Assertion URL Summary, and Assertion detail. Also removed special handling for & in URLs.

https://hg.mozilla.org/automation/sisyphus/rev/366c1d230f49
Fixed a variety of issues with Assertion views, prepared web service methods for the addition of unit testing views

https://hg.mozilla.org/automation/sisyphus/rev/c18288e130b6
Added Unit Test Assertion views and added the signature and socorro id to the Socorro View control panel

Comment 64

[Jonathan Eads ( :jeads )]

Attachment: Adds Unit Test Valgrind Queries

You will need to run "python manage.py build_nav", restart the server, and clear your browser's cache before testing this patch.

1.) Added Unit Test Valgrind Queries: Valgrinds, Valgrind URL Summary, and Related Valgrinds.

Comment 65

[Jonathan Eads ( :jeads )]

Attachment: Added view collections, a default_load data flag, and set the Crash Explorer collection to the default landing page.

You will need to run "python manage.py build_nav", restart the server, and clear your browser's cache before running this patch.

1.) Added view collections.  You can access them through the navigation menu.

2.) Added default_load flag to the view JSON structure to control which views load data in their default state.

3.) Set the Crash Explorer view as the default collection to load for the landing page.

Comment 66

[Jonathan Eads ( :jeads )]

Attachment: Added a new right click context menu for table cells.

You will need to clear your browser's cache before testing this patch.

1.) Removed old table cell/icon context menu and replaced with a right click based context menu.

2.) Added a "Select Cell Data" option to the new right click menu.

3.) Added all of the old context menu functionality to the new menu.

Comment 67

[Jonathan Eads ( :jeads )]

Attachment: Modified Collection Behavior

You will need to run "python manage.py build_nav", restart the server, and clear your browser's cache before testing this patch.

1.) When a collection is initialized it removes all of the views in the page before adding new ones.
2.) Forced a larger minimum size on data views.
3.) Added a copy feature to the right click context window.
4.) Added a new button to all data view panels that opens the panel in a new window.
5.) Added messaging for failed AJAX calls.
6.) Changed the "Open Connections Window" to be the "Open New View" and defaulted new views to open in a new page.
7.) Removed the send/received options from HTML and javascript.
8.) Added handling for using cached queries to facilitate rapid UI development when bughunter is inaccessible.

Comment 68

[Jonathan Eads ( :jeads )]

Attachment: Added SQL LIKE functionality

You will need to restart the server to test this patch.

1.) You can utilize an SQL LIKE clause by adding a % to the beginning or ending of any field found in the control panel and re-loading the control panel.

Comment 69

[Jonathan Eads ( :jeads )]

Attachment: Added a tree representation for platform distribution data.

You will need to run "python manage.py build_nav", restart the server, and clear your browser's cache before testing this patch.

1.) Added a tree representation to help visualize platform distribution data for Crashes, Assertions, and Valgrinds.

Comment 70

[Jonathan Eads ( :jeads )]

Attachment: Added the mozilla/HTML5 based context menu and a basic set of help documentation

You will need to restart the server and clear your browser cache to test this patch.

1.) Replaced jquery context menu with mozilla/html5 based menu.

2.) Added a basic set of help documentation for the application.  Started on data view specific help for crashes.  Will need some input from bc to finish this.  In the file sisyphus/webapp/templates/help/bughunter.generic.help.html I have added the string "BC" in locations where my understanding/knowledge is lacking and I need help filling in the information.  I would like to finish all of the data help for Crashes first, then I will templatize before moving onto Assertions and Valgrinds.

Comment 71

[Jonathan Eads ( :jeads )]

Attachment: Added Bugzilla/crash-stats search ability

You will need to clear your browser's cache to test this patch.

1.) Added Search options for bugzilla and crash-stats to the context menu.

Comment 72

[Jonathan Eads ( :jeads )]

Attachment: Fixed page naming system and context menu bugs

You will need to clear your browser's cache before testing this patch.

1.) Modified the page naming system to use a JSON string stored in localStorage.  

2.) Removed redundant "Bugzilla" prefixes from nested context menu items.

3.) Cleared the text selection when the context menu is opened on a new item that is different than the selected text.

4.) Added ALL to the bugzilla quicksearches using the summary: and comment: fields.  If we also need to add ALL to url: and sig: this will be a simple modification with the new link building code.

5.) Changed content searches to include both summary:"blah" AND comment:"blah"

6.) Modified context menu for fatal messages: removed bugzilla signatures and crash-stats from search options.

7.) Modified context menu for urls: removed bugzilla summary and crash-stats from search options

Comment 73

[Bob Clary [:bc] (inactive)]

Comment on attachment 584880 [details] [diff] [review]
Fixed page naming system and context menu bugs

Review of attachment 584880 [details] [diff] [review]:
-----------------------------------------------------------------

Your queries misplace the use of ALL. It is not used with the specific fields but is used as an overall qualifier to select bugs of all resolutions.

For example your current query when searching for fatal_message is:

 classification:"Client Software" OR classification:"Components" AND summary:ALL "Assertion failure: length != 0"

It should be

 ALL classification:"Client Software" OR classification:"Components" AND summary:"Assertion failure: length != 0"

Comment 74

[Jonathan Eads ( :jeads )]

Attachment: Fixed page naming system and context menu bugs

Moved the bugzilla ALL resolution specification to "quicksearch=ALL classification ..."

Comment 75

[Jonathan Eads ( :jeads )]

Attachment: Fixed bugzilla search logic bug, changed boolean AND to OR

Fixed bugzilla search logic bug, changed boolean AND to OR

Comment 76

[Jonathan Eads ( :jeads )]

Attachment: Placed data related help into templates, fixed some CSS/HTML bugs

You will need to clear your browser's cache before testing this patch.

1.) Fixed style on Platform Tree detail panel.

2.) Put the data related help into templates to avoid copying HTML repeatedly.

3.) Moved some inline styles into css.

Comment 77

[Jonathan Eads ( :jeads )]

Attachment: Fixed some help doc issues

You will need to clear your browser cache before testing this patch.

1.) Added the Socorro Record table to the help docs, it was left out in the first pass.  Fixed a handful of typos.

2.) Added the csrf guard to the help service method.

3.) Updated the README

Comment 78

[Bob Clary [:bc] (inactive)]

https://hg.mozilla.org/automation/sisyphus/rev/6d70649985fb
Added Unit Test Valgrind queries: Valgrinds, Valgrind URL Summary, and Related Valgrinds

https://hg.mozilla.org/automation/sisyphus/rev/4658ac27c6a3
Added view collections, a default_load data flag, and set the Crash Explorer collection to the default landing page

https://hg.mozilla.org/automation/sisyphus/rev/1ee72f1f422e
Added a new right click context menu for table cells. This menu supports selecting a data cells contents along with all of the old functionality the original cell/icon based menu did

https://hg.mozilla.org/automation/sisyphus/rev/354647744369
Modified collections to clear all views when opened, added a copy feature to the context menu, forced minimum size on views, added messaging for failed AJAX calls
(accidentally checked in under my username)

https://hg.mozilla.org/automation/sisyphus/rev/97e950126347
Added the following feature: An SQL LIKE will be used instead of = for any control panel field that begins or ends with %

https://hg.mozilla.org/automation/sisyphus/rev/95339b632085
Added a tree representation for Crashes, Assertions, and Valgrinds that helps visualize the platform distribution

https://hg.mozilla.org/automation/sisyphus/rev/142336dc2079
Added the mozilla/HTML5 based context menu and a basic set of help documentation accessible from the '?' icons in the UI

https://hg.mozilla.org/automation/sisyphus/rev/936eaf7dac0b
Added bugzilla/crash site search options to the context menu

https://hg.mozilla.org/automation/sisyphus/rev/71256ff1e5e3
Adapted bughunter page numbering system to use a JSON string stored in localStorage and fixed the context menu bugzilla/crash-stats search bugs

https://hg.mozilla.org/automation/sisyphus/rev/6a33391978dc
Fixed some help doc issues

Comment 79

[Jonathan Eads ( :jeads )]

Attachment: Removed files that are no longer used

This patch removes some HTML and javascript files that are no longer used by bughunter.

Comment 80

[Bob Clary [:bc] (inactive)]

Attachment: fix conflated unittest + sitetestrun issues

The UnitTests have no connection with the SiteTests. This fixes several views to use UnitTestRun instead of SiteTestRun.

Comment 81

[Bob Clary [:bc] (inactive)]

Attachment: help patch from bc

Comment 82

[Jonathan Eads ( :jeads )]

Attachment: Added addition external links, pre HTML tag wrapping, and context menu option for URL resubmission

You will need to run "python manage.py build_nav", restart the server, and clear your browsers cache before testing this patch.

1.) Added external link for the changeset column to the changeset display in mercurial.

2.) Wrapped steps and valgrind stack columns with <pre></pre> tags

3.) Added external link for Socorro uuid to the crash-stats crash report page.

4.) Added context menu option for resubmitting URLs to bughunter.  Note: when this option is selected it just shows an alert, this is a placeholder function to be filled out in a subsequent patch.

5.) refactored the DataAdapterCollection.js processData function to manage specialized column formatting in a more extensible manner.

Comment 83

[Bob Clary [:bc] (inactive)]

Comment on attachment 575577 [details] [diff] [review]
Merged url views, added socorro_id signals, merged site test crash and crash details

Actually landed this on 2011-12-05: https://hg.mozilla.org/automation/sisyphus/rev/cab0c135009e

Comment 84

[Bob Clary [:bc] (inactive)]

Comment on attachment 590265 [details] [diff] [review]
Added addition external links, pre HTML tag wrapping, and context menu option for URL resubmission

looks good.

Comment 85

[Jonathan Eads ( :jeads )]

Attachment: Added a URL resubmission functionality and new data views to examine resubmitted URLs

You will need to run "python manage.py build_nav", restart the server, and clear your browser's cache before testing this patch.  In addition, when this patch is pushed to local/production database, a user_id column needs to be added to SocorroRecord, and that column will need an index.

1.) Added two options to the context menu: "Resubmit URL" and "Resubmit All URLs".
2.) The new context menu options open a modal window that displays the URLs to be submitted and allows the user to enter a comment.
3.) Added two new data views under the Navigation Menu item called "User Data": "My URL Resubmissions" and "All URL Resubmissions".  These data views display user URL resubmissions.  This is a start, we will need to add data view collections that will enable a user to analyze the results but need to consult with bc before doing this.
4.) Modified automation/crashtest/crashurlloader.py by modularizing methods so that the module can be used from an import in addition to the command line interface.
5.) Added a user_id column to SocorroRecord with an index.  This will need to be done explicitly on local/production database to test this patch.

Comment 86

[Jonathan Eads ( :jeads )]

Attachment: Added URL resubmission functionality and new data views to examine resubmitted URLs

You will need to run "python manage.py build_nav", restart the server, and clear your browsers cache to test this patch.

1.) Added username option to crashurlloader.py, removed userid option and forced user to supply the username.

2.) Ressolved CSRF issue by adding @csrf_exempt back.

3.) Fixed duplicate URL submission bug

4.) Changed comments field back to signature.

5.) Added URL resubmission entry to help.

6.) Explained signature overloading in help

7.) Added textarea for user URL submissions to control panel of My URL Resubmissions and All URL Resubmissions.

8.) Built a set of collections for My URL Resubmissions.

9.) Added signature as a signal to My URL Resubmissions

Comment 87

[Bob Clary [:bc] (inactive)]

Comment on attachment 593171 [details] [diff] [review]
Added URL resubmission functionality and new data views to examine resubmitted URLs

r+ with minor changes to allow username or email for crashurlloader.

Comment 88

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/automation/sisyphus/rev/80c2429d3952
http://hg.mozilla.org/automation/sisyphus/rev/39664811c473
http://hg.mozilla.org/automation/sisyphus/rev/dbcb3d50c304
http://hg.mozilla.org/automation/sisyphus/rev/b4d5eb8188c3
http://hg.mozilla.org/automation/sisyphus/rev/a3e2d35debfa

Comment 89

[Jonathan Eads ( :jeads )]

Attachment: Fixes the js error that was occurring when a user submitted a URL from the control panel

Fixes the js error that was occurring when a user submitted a URL from the control panel

Comment 90

[Jonathan Eads ( :jeads )]

Attachment: Fixes js user submitted URL bug, js instance attribute bug, and multiple status row bug

You will need to restart your server and clear your browser's cache before testing this patch.

1.) Fixed the js class instance attribute bug, all class instance attributes that are set at runtime without a rational default are explicitly initialized to undefined.

2.) Fixed the multiple status row bug that occurred on URL resubmission.  All status counts are now displayed in a single row.

3.) Added date range constraints to My URL Resubmissions and All URL Resubmissions.

4.) When Submit URLs is clicked it now triggers an immediate close of the control panel and display of the spinner on the associated view to prevent the user from clicking while the view waits for the POST return.

Comment 91

[Jonathan Eads ( :jeads )]

Attachment: Fixed a contextmenu search bug, data in URLs was not escaped

You will need to clear your browsers cache to test this patch.

1.) Properly escaped data in bugzilla and crash-stats searches.  Special characters were breaking the search calls and needed to be escaped.

Comment 92

[Jonathan Eads ( :jeads )]

Attachment: Fixed a contextmenu search bug, data in URLs was not escaped

You will need to clear your browser's cache to test this patch.

1.) Fixed URL escape bug for url based searches from the context menu.

2.) Added a concatenation operator to the generation of the status field for URL resubmissions so that all of the status states can be seen in the field.

Comment 93

[Jonathan Eads ( :jeads )]

Attachment: Protect against user selecting a non-anchor region of the nav menu

You will need to clear your browser's cache to test this patch.

1.) Added handling for when a user clicks a non-anchor portion of the navigation menu.

Comment 98

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/automation/sisyphus/rev/b4d5eb8188c3
Added addition external links, pre HTML tag wrapping, and context menu option for URL resubmission

http://hg.mozilla.org/automation/sisyphus/rev/a3e2d35debfa
Added a URL resubmission functionality and new data views to examine resubmitted URLs

http://hg.mozilla.org/automation/sisyphus/rev/f0f11f3dc1c6
http://hg.mozilla.org/automation/sisyphus/rev/c86c87dae56d
Fixes the js error that was occuring when a user submitted a URL from the control panel

http://hg.mozilla.org/automation/sisyphus/rev/c91c248b5fc3
Fixed a contextmenu search bug, data in URLs was not escaped

http://hg.mozilla.org/automation/sisyphus/rev/831aa4fc6197
Protect against user selecting a non-anchor region of the nav menu