1441919 - Lando test plan

Status: RESOLVED FIXED

(Conduit :: Documentation, enhancement)

Description

[Mark Côté [:mcote]]

We need a test plan for Lando similar to https://wiki.mozilla.org/Phabricator/TestPlan.

It should cover everything that has landed so far (or are just about to), notably, complete landings, blocking errors, and (display of) nonblocking warnings.

Comment 1

[Israel Madueme [:imadueme]]

Will QA be doing these tests on the production Phabricator and production Lando instances? Or is there a -dev transplant out there somewhere?

If they will be doing this on the production instances then I would like to not block our initial release on this. That way I/whoever can create the test plan in exactly the same way that we expect QA to run it (i.e. with the deployed instances instead of my local development setup). 

I know that most things would remain the same, but, it's only when you actually do the real thing that you find the problems you missed beforehand.

Comment 2

[Israel Madueme [:imadueme]]

Do our QA people even have L3 to land stuff? And if we only have the production instance, then I guess we have to have QA test with a local development setup right? It wouldn't make sense to test in production; for a pre-release yeah, but, going forward we'd need to have a better solution.

I can investigate updating the demo environment to include the transplant service (fairly easy with glob's prior work) and update lando so that QA can do the test plan in localdev.

Comment 3

[Mark Côté [:mcote]]

Yes, ckolos is setting up a transplant-dev.  That will be the testing path going forward.  Until that is ready, we'll use production, to a QA repo.

We thought about using the demo environment for testing originally, but it's too complicated and fragile.  And regardless, QA should be testing against our deployed instances.  Local testing is for developers, generally speaking.

You raise a good point with L3 access.  I'm not sure what permission is required to push to the QA repo, but it shouldn't be L3.  However for now I don't think Lando understands that different repos have different permissions.  We'll need that for try.  Let's find out if QA already have L3 access; if so, we're good.  If not, we either have to get QA to get L3, or we have to build the permission mapping now.

Comment 4

[Phabricator Automation]

Attachment: test plan: add initial QA test plan

Comment 5

[Phabricator Automation]

Comment on attachment 8958059 [details]
test plan: add initial QA test plan

Piotr Zalewa [:zalun] has approved the revision.

https://phabricator.services.mozilla.com/D707