Closed Bug 669644 Opened 14 years ago Closed 14 years ago

a user should be able to enter test cases in batch mode given... when... format

Categories

(Mozilla QA Graveyard :: MozTrap, enhancement)

Product:
Mozilla QA Graveyard
Component:
MozTrap
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: camd, Assigned: carljm)

Details

1) Users should be able to enter test cases in a “As a... Given... When...” format such as: Test that a customer can purchase something As a paying customer Given that I have added an item to my shopping cart When I choose to checkout Then I should be shown a summary of my order And I should be prompted to pay for my purchase And when I click cancel Then I am asked to confirm my cancellation 2) A user should be able to enter this as free-form text, and it will place the text into the appropriate fields based on keywords (not case sensitive): Test that As Given when and then 3) These keywords should be highlighted in the pane (green? blue?) as the user types to give them the visual cue that they are “doing it right” 4) If the user enters more than one “Test that” entry, it should create a separate case for each. 5) After the user has typed everything, it should show the user a preview of each test case, showing how it will look for execution, so they can approve it. or they can go back and edit to fix it. 6) The page will need to show the keywords somewhere. A glossary, so to speak. Maybe as a popup on a “?” icon. Or perhaps a sidebar next to the text field? Just a list like the bullets shown above? Then hovering on one, it will give a description of what that means. On the testcase creation screen, the user can “switch modes” possibly like tabs to go into: Given.. when... format Wiki-text style format (another user story) Form field (like we have now) format
A Pivotal Tracker story has been created for this Bug: http://www.pivotaltracker.com/story/show/15412705
Cameron Dawson added a comment in Pivotal Tracker: let's see what other features we may need to postpone in order to make this for the July/August recess.
Cameron Dawson changed story state to started in Pivotal Tracker
Eric Meyer added a comment in Pivotal Tracker: Is there any reason for separate input fields for the keyword vs wiki-style text boxes? Could one field accept either format, and the glossary show both options?
Cameron Dawson added a comment in Pivotal Tracker: Eric: nope, no reason. I think one field is ideal. Not sure if it is more efficient for the text-parsing to know which kind of parsing it is expected to do ahead of time. Or to just try them both, and use what works.
Eric Meyer added a comment in Pivotal Tracker: How do we handle the product field in this plain-text format? Do we need a "for" keyword?
Cameron Dawson added a comment in Pivotal Tracker: I was thinking that we would have a separate drop-down field in both cases for the product. I think we should expect the product to be consistent for all cases entered by this mechanism. So I think we would have just 2 fields: 1) product, 2) big text area. I would love it if we could also have a Test Suite field that all cases entered this way are automagically added to a test suite. However, I don't know that the platform supports that, since the cases won't be "approved" yet.
Carl Meyer added a comment in Pivotal Tracker: If we want to actively circumvent the platform's approval feature (which is possible, but I think fighting the platform is a dangerous road to go down for the long-term health of the project), we could also have the UI code magically "approve" the entered test cases immediately (via the global admin user), and thus make it possible to have them immediately entered in a test suite. Also, I don't foresee it being particularly difficult to sort out which text format is being used in a single textbox. I think we should probably implement a single-format version first, though, and then expand it with a second format later.
Eric Meyer added a comment in Pivotal Tracker: Is that suite field a selection of existing suites (single or multiple) or a text field to add a new one, or an auto-completing text-field that allows for both, or something else entirely?
Cameron Dawson added a comment in Pivotal Tracker: Good question. While I think it would be neat-o to be able to create the Suite right there, if it's easier to just have a drop-down, that's cool, too. I think the only other field needed to create the suite would be the description. They could presumably edit the env stuff later. Carl's point about fighting the platform on this is a good one. It's just that this FEELS right to be able to add all the things I'm creating in this batch to a particular container (Suite). I wonder if we could require an admin password to do this?
Cameron Dawson added a comment in Pivotal Tracker: More thoughts: here's a scenario: We have a test-writing day where 20 people write tests for different areas at the same time. If they each wrote 10 tests, that'd be 200 new tests. Then an admin needs to go in and approve them all. But if we aren't able to specify which suite they go to in the first place, then we'd have to have ANOTHER pass through them to sort them into the Suites they should each go with. That sounds bad to me. We may need to discuss this live or something, but it is striking me that the user should specify the suite at time of test creation. Or at least be able to tag the case with the suite name it is intended for.
Carl Meyer added a comment in Pivotal Tracker: Cam: In your ideal world, how would this interact with the approval feature? That you'd be allowed to add test cases to a test suite before they're approved, and then you'd have to approve them before the test suite can be activated (that seems reasonable to me)? Or that certain users would have the privilege of creating test cases without requiring approval, and only those users would be able to use the batch UI and pre-assign to a suite? If this is something that's a priority, then we need to figure out the details of how it should work, file a platform bug to request that change in the platform, and then I can consider what workarounds, if any, are possible to make it work that way prior to the platform being updated.
Cameron Dawson added a comment in Pivotal Tracker: Carl: I'm realizing that this may be significant changes to the platform design.. sigh... But I think ideally, a suite should have any combination of active, locked, draft and unapproved cases. If that suite is added to a test run, then only the active cases would be added. If the suite has active cases, it would be considered active and use-able by a test run. Otherwise, the suite should show in the list that it's not active(draft would be distinct from inactive), and wouldn't be add-able to a test run.
Carl Meyer added a comment in Pivotal Tracker: Cam: Actually I think the first part of that probably wouldn't be a difficult change to make in the platform at all. It might even be something I could figure out how to do, though probably not a good use of my time. The latter part (where suites don't have their own draft/active state, but its automatically determined by the cases in them) would likely be more invasive and difficult. Unfortunately this approach is one I can't see how I would implement as a workaround; the platform simply won't let me add those unapproved/inactive cases to a suite. And if this is how it should work, I'm hesitant to implement a workaround that works fundamentally differently (auto-approval).
Cameron Dawson added a comment in Pivotal Tracker: Carl: oh, cool. That's encouraging! :) But agreed, this isn't a good use of your time, especially this week. Maybe Jerome could look at that. :) For this feature, this week, let's not worry about Suites. I'll create a new story for that feature. If we could add tags to the batch cases, that would be cool. But not necessary this week either. Perhaps the wireframes could show our "happy place" but hide the suite stuff in the impl? about draft / inactive: I wasn't clear in my description. I was thinking that we would still have draft or active state explicitly set on Suites. But if the suite had no cases in active status, then it might have some visual indication of that in the list, and not be add-able to a test run. Or, if you added it, you'd get an error saying: "Not gonna do it. Wouldn't be prudent."
Carl Meyer added a comment in Pivotal Tracker: Cool, all makes sense. We'll put the suites dropdown (or autocomplete) in the wireframe, and just have a page note to the effect that it's inactive, with a link to the relevant platform bug.
Eric Meyer added a comment in Pivotal Tracker: @j - this needs some basic slideshow handling JS. capture the links that point at slides, then toggle the slide and the active class on the link. I'd also like to replace the "add a step" link with some cooler automation. We can talk about how exactly that might work...
Carl Meyer added a comment in Pivotal Tracker: eric: I merged this branch (master-bulk-create) back into master; hope that's ok. It's just waiting for some JS from jonny, and it doesn't break anything that worked before (except env constraints are gone, but that's a good thing).
Cameron Dawson added a comment in Pivotal Tracker: This looks awesome. The single/bulk buttons look perfect. (not sure why that caught my eye so much) For the instructions on given when and markup, we could have little question mark icons next to it, and have it expand if they click it. We might explore having a drop-down for Suites. When I got to the page, and looked at "begin typing the name of a suite" I immediately thought, "uhh, what were those suites called again?" I know I was the one who asked for it that way though... :) And I know part of the idea for having it that way was that we could possibly create a new suite at this time, too. But let's go with it as is for now, and perhaps explore that later, if it feels wrong when we start using it. But this is awesome for now!
Eric Meyer added a comment in Pivotal Tracker: I actually think we have enough space to give decent instructions in-place without any help link. It's worth a try.
Carl Meyer added a comment in Pivotal Tracker: This is working for adding cases using the keyword syntax. It doesn't have any fancy syntax highlighting yet, and likely won't until August, but the backend is set up to make that doable when the time comes. I also noticed an oddity; the bulk syntax includes syntax for "description", but nowhere else on the site do we have descriptions for testcases, including on the existing (single) add-case page. If descriptions are desired, it'll take some time to work that in everywhere it should go. For now it'll just be a little odd that you put in a description in the bulk-create and then you never see it again. The precise syntax rules are as follows: White space at beginning or end of lines is ignored, and all keywords are case-insensitive. The first line must begin with "Test that ", and becomes the name of the testcase. There must be at least one description line following that, possibly more. It doesn't matter what the description lines look like (As... Given... are not enforced). Following the description line(s) there must be a line beginning with "When ". This marks the beginning of a step instruction. Additional lines can follow that are also part of the instruction, until a line beginning with "Then " - that marks the beginning of the expected result of that step. Any following lines are also part of that expected result, until either another "When " line (marking the start of the instruction for the next step), or another "Test that " line (marking the beginning of a new test case). The start-of-instruction for any step after the first may optionally use "And when " rather than just "When ". Alternatively, "And" may be used on a line by itself between steps, in which case it will be ignored. The only noticeable differences between this and the Google Doc are the lack of enforcement of As... Given... (if we do enforce that here should we enforce that the descriptions for all test cases follow that format?) and that the Google Doc includes a step (delimited by And) which has no instruction, only an expected result, which is not possible in the platform. My code would parse that as all one expected result, so the example in the Google Doc would have two steps, not three. There are some JS/UI issues remaining here; assigned to jonny next.
Cameron Dawson added a comment in Pivotal Tracker: updated the description to describe how it's actually working now. It was formerly "Given... when..." and now it's "Test that... when..."
Eric Meyer added a comment in Pivotal Tracker: @j - cam had a good-sounding solution to the auto-add-step task that he put here: http://www.pivotaltracker.com/story/show/12921589
Cameron Dawson added a comment in Pivotal Tracker: Jonny: for adding a new testcase to an existing test, could there be a button that shows on hover of an existing step that will insert a new step above it? Just a thought. Not sure what the icon for that. Perhaps something like the one attached here?
Cameron Dawson added a comment in Pivotal Tracker: Jonny: sorry, bad wording in my last comment. I meant to say "for adding a new step to an existing test"
Jonny Gerig Meyer added a comment in Pivotal Tracker: This story applies to bulk test case creation, and it appears to be done. The remaining issues related to single creation have been moved to https://www.pivotaltracker.com/story/show/12921589
Jonny Gerig Meyer changed story state to finished in Pivotal Tracker
Carl Meyer added a comment in Pivotal Tracker: At the very least here, we still need to fill in the instructions for the format on the right side of the bulk form. Jonny, you up for that? There are also other features that have been discussed, like syntax-highlighted preview. I think it would be best if such features got their own dedicated stories- Cam, I'll leave it up to you to create those stories and prioritize them as desired.
Carl Meyer changed story state to started in Pivotal Tracker
Carl Meyer added a comment in Pivotal Tracker: jonny: text is clear, thanks. eric, i think some keyword highlighting and better layout will help its readability a lot. jonny: there's an issue with the error-reload of the bulk form, made you a task.
Eric Meyer changed story state to finished in Pivotal Tracker
Carl Meyer changed story state to delivered in Pivotal Tracker
Eric Meyer changed story state to started and added a comment in Pivotal Tracker: triggered horizontal scrolling
Eric Meyer changed story state to started in Pivotal Tracker
Eric Meyer changed story state to finished in Pivotal Tracker
Carl Meyer changed story state to delivered in Pivotal Tracker
Cameron Dawson changed story state to accepted in Pivotal Tracker
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Target Milestone: --- → 0.4
Bumping to verified as [qa-] due to age of bug, shipped.
Status: RESOLVED → VERIFIED
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.