1826344 - Add QA milestones to the app

Status: RESOLVED FIXED

(Websites :: whattrainisitnow.com, enhancement)

Description

[Pascal Chevrel:pascalc]

Following a discussion on Slack, there is a desire to also have (some ?) QA milestones on the app.

I am seeing those currently but not sure yet which are really needed as we want the schedule to be clear and usable for all consumers.

QA Request & Feature technical documentation deadline
Feature Complete Milestone (if manual QA needed in Nightly)
Feature Complete Milestone (if manual QA starts in Beta)
QA pre-merge regression testing completed
Pre-release sign off

Comment 1

[Pascal Chevrel:pascalc]

After talking with QA, the milestones to add are:

• QA Request & Feature technical documentation deadline
• Feature Complete Milestone (if manual QA needed in Nightly)
• Feature Complete Milestone (if manual QA starts in Beta)
• QA pre-merge regression testing completed
• Pre-release sign off

Schedule rules:

Friday, Nightly week 1:

• QA Request & Feature technical documentation deadline
• Feature Complete Milestone (if manual QA needed in Nightly)

Friday, Nightly week 2:

• Feature Complete Milestone (if manual QA starts in Beta)

Friday, Nightly week 4:

• QA pre-merge regression testing completed

Wednesday, Beta week 3:

• Pre-release sign off

Notes:

• Milestones on the same date should be merged.
• Milestones should be reworded and indicate they are QA releated
• We need a short description for all the milestones

Tania, could you provide short descriptions for all the milestones? Thanks

Comment 2

[Tania Maity (:tmaity)]

Friday, Nightly week 1:

• QA Request & Feature technical documentation deadline:
  A QA Request needs to be filed in JIRA requesting manual QA assistance for features that are riding Fx train or require testing during the nightly and/or beta cycle. Along with the JIRA ticket, feature technical documentation is also required for QA to draft their Test Plan and Test Cases.

• Feature Complete Milestone (if manual QA needed in Nightly):
  Manual QA in Nightly cycle is optional but recommended. If feature owner requests Nightly testing (to be opted while filing QA request) then code should be ready for testing early so that QA has enough time to assess feature quality and provide a formal test report by the end of Week 3 of the nightly cycle.

Friday, Nightly week 2:

• Feature Complete Milestone (if manual QA starts in Beta) :
  If Nightly testing was skipped then feature should be in testable state by this time in preparation for Beta Testing.

Friday, Nightly week 4:

• QA pre-merge regression testing completed:
  QA assess build readiness for shipping to Beta by running a set of Regression test cases and shares a test report. Nightly Regression coverage is limited compared to Beta Regression testing (happens throughout Beta cycle).

Wednesday, Beta week 3:

• Pre-release sign off:
  The final, pre-release QA sign off which can be: Green (no quality concerns), Yellow (moderate quality concerns) or Red (severe quality concerns). A week before the final sign-off, QA shares preliminary test reports (mid beta assessment, forecast for the final sign-off) for each feature riding the train.

Comment 3

[Pascal Chevrel:pascalc]

Tania, when the dates say "Friday of week x" or "Wednesday of week y", is that deadline the start of the day or the end of the day? In which timezone? Thanks

Comment 4

[Pascal Chevrel:pascalc]

https://github.com/mozilla/releases_insights/commit/744eabc91f7fa7dc5bbd89b7bac5b9df680de03c

I am keeping the bug open as there is the follow up task of adding hours to QA milestones.

Comment 5

[Pascal Chevrel:pascalc]

First follow up patch based on feedback: https://github.com/mozilla/releases_insights/commit/80e048574f9087c86006536cbfb9cc5a8dc5c45f

Comment 6

[Tania Maity (:tmaity)]

(In reply to Pascal Chevrel:pascalc from comment #3)

Tania, when the dates say "Friday of week x" or "Wednesday of week y", is that deadline the start of the day or the end of the day? In which timezone? Thanks

These milestones would follow the same standard timezone rule as the other release milestones decided by RelMan except the QA reports ( "QA pre-merge regression testing completed" and "Pre-release sign off") .

These QA Reports/ Sign-offs are published by RO team and hence these two milestones are all as per RO timezone, their end of day.

Comment 7

[Pascal Chevrel:pascalc]

Hours added in https://github.com/mozilla/releases_insights/commit/c79855b32c6f697a668f160f5b340a21017d08f1