Closed Bug 375987 Opened 17 years ago Closed 12 years ago

Add testcase tagging support to Litmus

Categories

(Webtools Graveyard :: Litmus, defect, P3)

Product:
Webtools Graveyard
Component:
Litmus
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: coop, Unassigned)

References

(
URL
)

Details

This came up at our test automation meeting this week.

It would nice to be able to tag testcases based on "random" criteria. The example given in the meeting was by code module, but I could see many different types tags being useful for different groups, e.g. developers vs. QA.

The tagging model proposed was open-ended, similar in style to the tags used by flickr for photos.

We would also need to add "search by tag" to the  view tests page. We could also add some stock reports like "Top tags" etc.

This would require a new db table, testcase_tags. This would also require a new user flag, can_tag. Admins would automatically be able to tag, but users could be granted this flag independently of admin status.
I'm wondering whether a keyword model like Bugzilla uses with pre-defined keywords would be better then a tag model with flickr-style open-ended tags, so that different people don't end up with different tagging systems. 

Or, perhaps combining the best of both worlds, open-ended tags with AJAXy autocomplete goodness so that it's easy to find tags that have already been used. Scriptaculous's Ajax.Autocomplete is pretty good, since we don't already seem to have an autocomplete module in js/. 
(In reply to comment #1)
> I'm wondering whether a keyword model like Bugzilla uses with pre-defined
> keywords would be better then a tag model with flickr-style open-ended tags, so
> that different people don't end up with different tagging systems. 

This was raised explicitly at the meeting. We don't want to pre-define the tags because that puts the onus on one (or more) of us in the QA team to decide what tags make sense upfront. I can see tagging being very useful to many different classes of users, and I don't want to personally spend the time trying to figure that out in advance (and then maintaining it in perpetuity) when having a flexible user-based tag scheme will accomplish the same thing more easily and faster.

> Or, perhaps combining the best of both worlds, open-ended tags with AJAXy
> autocomplete goodness so that it's easy to find tags that have already been
> used. Scriptaculous's Ajax.Autocomplete is pretty good, since we don't already
> seem to have an autocomplete module in js/. 

Yes, that was the flickr model I mentioned: arbitrary tags for testcases, with some display of already-existing tags, be it by user, auto-complete, or whatever.
Severity: normal → enhancement
I'd love to raise the priority on this. This would make test case work on Litmus for FF3 a lot easier.
We've talked a bit about how we tags to work from the tag assignment side (e.g. flickr). I'm wondering about what kinds of tag search and display tools people would like to see as well. Again, flickr has some relevant tools for displaying photos associated with tags, but I'm open to other models.
Status: NEW → ASSIGNED
Sorry for the delay in replying. As a Flickr user, I'm pretty happy with their method. It's simple and freeform.

As far as search, a list of used tags as well as the ability to do a freeform text search would be good. 

Mainly, I'm wanting to be able to sub-group (beyond what we allow currently) test cases into useful sets (in Places, for example). Tagging makes this a lot easier and allows others to tag. 

We may want to have specific privs necessary to tag, just as to edit, a case, in order to keep it from ballooning out of control but that may not actually be much of a problem.
Priority: -- → P3
Severity: enhancement → major
I'll start with requiring product admin access to add testcase tags, but everyone will be able to search. I'll make it a simple flag to enable tagging for everyone too if we decide to enable it later.
Priority: P3 → P2
Assignee: litmus → ccooper
Status: ASSIGNED → NEW
Status: NEW → ASSIGNED
Landed the schema changes for this tonight. No interface yet.
Status: ASSIGNED → NEW
Priority: P2 → P3
QA Contact: ccooper → litmus
Assignee: ccooper → nobody
Can we get the priority raised on this?

A use-case for dev is to be able to mark tests that could have an automated test instead of a manual one. This would provide the ability to define a clear queue of work for potential test-script-writers.
Yes, raising the priority of this would be especially useful since we are being asked to add Theme related test cases that are OS specific.
So this still appears as P3, although it seems as if in our QA team mtg today there is a strong feeling that is should be higher in the queue.
Well, that discussion was not specifically on tagging, per se, but on being able to mark which version of Firefox, for example, a test case applies to (so we can have one testcase for multiple firefox versions). Tagging can do this but most test case managers just have a "version" section with checkboxes for applicable versions.
(In reply to comment #11)
> So this still appears as P3, although it seems as if in our QA team mtg today
> there is a strong feeling that is should be higher in the queue.

I recognize the interest in this. It has been the top priority item in the Litmus queue for over a year. I just don't have any free time to work on Litmus bugs. The only bugs that bubble up are crashes or exploits. All my other working time has been filled up with build/release team bugs ever since I've been reassigned away from QA.

If QA is serious about getting Litmus bugs fixed, they need to either:

a) get me on loan from the build team for long enough to fix the bugs; or,
b) hire another webdev.

timr and joduinn should talk about this.


So, I read this as there are no planned improvements for Litmus for the QA org because the resources for this are reallocated permanently. 

We need to do something about this or find a new test case manager.
(In reply to comment #13)
> (In reply to comment #11)
> > So this still appears as P3, although it seems as if in our QA team mtg today
> > there is a strong feeling that is should be higher in the queue.
> 
> I recognize the interest in this. It has been the top priority item in the
> Litmus queue for over a year. I just don't have any free time to work on Litmus
> bugs. The only bugs that bubble up are crashes or exploits. All my other
> working time has been filled up with build/release team bugs ever since I've
> been reassigned away from QA.
> 
> If QA is serious about getting Litmus bugs fixed, they need to either:
> 
> a) get me on loan from the build team for long enough to fix the bugs; or,
> b) hire another webdev.
> 
> timr and joduinn should talk about this.
> 
Coop, obviously we'd prefer to have you working on this since you know the code the best.  But if we need to branch out and train other people, we should start that process.  Where does the litmus code live?  I can try to recruit people to help with this.

Ok I found the code.  Geez, it's more than I expected. :-(
And all Perl!
(In reply to comment #14)
> So, I read this as there are no planned improvements for Litmus for the QA org
> because the resources for this are reallocated permanently. 
> 
> We need to do something about this or find a new test case manager.
> 
I'd like to see this as much as anyone else. But it really is just a "nice to have." Litmus is not going to go on the scrap heap any time soon for lack of this feature.
I'll see if I can put in a few cycles into this in the near future.
(In reply to comment #18)
> I'd like to see this as much as anyone else. But it really is just a "nice to
> have." Litmus is not going to go on the scrap heap any time soon for lack of
> this feature.

 No but the inability to mark what versions of Firefox a test case applies to or to select what platforms it applies to is a problem. No smoketest run on a non-Windows OS is ever 100% run because there are Windows specific cases that we have to mark as "not run" each time we do a run on Mac or Linux. 

 Litmus is, at best, 80% finished compared to the commercial test case managers on the market. I understand that it was written to fulfill certain needs, like being accessible through the web, that others don't always provide but the fact of the matter is that we've had a list of changes wanted or needed for most of a year now (long for this particular one) and no one being given time or direction to work on Litmus. It is always a tertiary priority after everything else so the work never gets done. This is very frustrating, at least to me, every time I use it or update test cases, etc.


We also have third party companies (like Flock, Miro, and Songbird) using Litmus for their test work. If we opened this up and made the code available as a project, we might actually get them to contribute bug fixes as well. I'm sure that each of these companies has a list of issues. If someone would like, I'd contact people but I'd like to be able to tell them that we'd accept code changes or the like. That would require someone to actually own this code, which is currently not the case.
Litmus is open source. It's available for use and development under the Mozilla Public License.
Yeah but there is no module owner or peer and, for example, Clint had to ask around to find the source. I'm not going to argue this anymore though.
(In reply to comment #19)
> I'll see if I can put in a few cycles into this in the near future.

A few cycles isn't going to cut it, and the recent comments in this bug are less about this specific feature than general disappointment with the stagnation of Litmus as a project.

Al: you seem to have to most interest/drive/bile here, and seem to know what you want. Compiling a list of the outstanding Litmus features (the missing 20%, as you say) would not be a bad idea if the QA team does decide go shopping for either a new webdev or a new management tool. Here's the list of Litmus bugs that are currently open as a seed:

http://tinyurl.com/duhc3
(In reply to comment #24)

> Al: you seem to have to most interest/drive/bile here, and seem to know what
> you want. Compiling a list of the outstanding Litmus features (the missing 20%,
> as you say) would not be a bad idea if the QA team does decide go shopping for
> either a new webdev or a new management tool. Here's the list of Litmus bugs
> that are currently open as a seed:
> 
> http://tinyurl.com/duhc3

I suppose that I will do so. I seem to remember a QA meeting or two in the last year where we had the entire QA Execution team work on such a list. I'm not sure where in the wiki such notes might be.
Litmus is being decommissioned in bug 802674. 

Visit https://moztrap.mozilla.org for your testing needs.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.