636816 - Too many results generated when assigning test case with limited environment groups

Status: VERIFIED FIXED

(Mozilla QA :: Case Conductor-Platform, defect)

Description

[Carl Meyer (:carljm)]

It seems that the environment profile of a test case is not being taken into account when including it in a test run or assigning it. 

If I have a test case with a limited profile (only two environment groups) and I add it to a test run that has a larger set of environment groups, the resulting IncludedTestCase has the testrun's full set of environment groups. And if I assign that IncludedTestCase, it generates results for all of the test run's environment groups, including ones that are not valid for the test case.

I would expect that the IncludedTestCase should have an environment profile consisting only of the intersection between the test run's environment groups and the test case's environment groups, and that on assignment results should only be generated for the environment groups in that intersection.

Comment 1

[Vadim]

Yes, right now environments are based on container profile in cases when it's defined, if it's not there test case profile is used.

I was thinking about intersecting them at first, but it's not as simple when groups within both environments aren't matched exactly. I have to think about this one some more...

Comment 2

[Carl Meyer (:carljm)]

(In reply to comment #1)
> I was thinking about intersecting them at first, but it's not as simple when
> groups within both environments aren't matched exactly. I have to think about
> this one some more...

I think this is pretty important to add, otherwise defining environments on testcases is pointless. The UI will autogenerate environment groups at the product level - this means the "container profile" always exists (because it inherits from the product), and therefore environment profile of testcase is always disregarded.

I'm not sure I understand what is meant by "both environments aren't matched exactly". At least in our use case, environments are generated at the product level, and both test cases and cycles/runs inherit from that same initial set of groups. I think the intersection should simply happen at the environment-group level, with no need to look inside at the member environments. If there are no groups in common between the test case and the test run, trying to add that test case to the test run should be an error.

Comment 3

[Vadim]

(In reply to comment #2)
> (In reply to comment #1)
> > I was thinking about intersecting them at first, but it's not as simple when
> > groups within both environments aren't matched exactly. I have to think about
> > this one some more...
> 
> I think this is pretty important to add, otherwise defining environments on
> testcases is pointless. The UI will autogenerate environment groups at the
> product level - this means the "container profile" always exists (because it
> inherits from the product), and therefore environment profile of testcase is
> always disregarded.
> 
> I'm not sure I understand what is meant by "both environments aren't matched
> exactly". At least in our use case, environments are generated at the product
> level, and both test cases and cycles/runs inherit from that same initial set
> of groups. I think the intersection should simply happen at the
> environment-group level, with no need to look inside at the member
> environments. If there are no groups in common between the test case and the
> test run, trying to add that test case to the test run should be an error.

The API supports creating and updating groups by assigning any number environments to them. It goes for Test Cases as well as executable entities. You're referring to auto-generate scenario, which doesn't cover all the possibilities. 

Defining environments at test case level is not useless, because users may choose not to define them at test run level and just take default test case environments.

So, intersecting them is the right approach, it's just as simple as it seems...

Comment 4

[Vadim]

ment to say not as simple...

Comment 5

[Carl Meyer (:carljm)]

(In reply to comment #3)
> The API supports creating and updating groups by assigning any number
> environments to them. It goes for Test Cases as well as executable entities.
> You're referring to auto-generate scenario, which doesn't cover all the
> possibilities. 

Yes, I understand that. I'm just pointing out that autogenerate is most likely the only scenario the UI will support in 1.0 anyway. Even when it does support arbitrary creation of environment groups, I still don't think that makes it necessary for the intersection algorithm to implement any special intelligence looking within the environment groups. Even a custom-created environment group should either be used (as itself) on both sides, or should not be included in the generated assignment. So I still think it's adequate to do the intersection purely at the environment-group-identity level, which is relatively simple to implement.
 
> Defining environments at test case level is not useless, because users may
> choose not to define them at test run level and just take default test case
> environments.

This does not match my observation. The test run I tried with did not have any environments defined at test run level. But it automatically inherited them from the product level nonetheless, and the test case environments were still ignored in favor of those environments inherited from the product. So it appears to me that currently, if the product has environment groups associated with it, defining environments on a test case will never have an effect.

Comment 6

[Vadim]

(In reply to comment #5)
> (In reply to comment #3)
> > The API supports creating and updating groups by assigning any number
> > environments to them. It goes for Test Cases as well as executable entities.
> > You're referring to auto-generate scenario, which doesn't cover all the
> > possibilities. 
> 
> Yes, I understand that. I'm just pointing out that autogenerate is most likely
> the only scenario the UI will support in 1.0 anyway. Even when it does support
> arbitrary creation of environment groups, I still don't think that makes it
> necessary for the intersection algorithm to implement any special intelligence
> looking within the environment groups. Even a custom-created environment group
> should either be used (as itself) on both sides, or should not be included in
> the generated assignment. So I still think it's adequate to do the intersection
> purely at the environment-group-identity level, which is relatively simple to
> implement.
> 
> > Defining environments at test case level is not useless, because users may
> > choose not to define them at test run level and just take default test case
> > environments.
> 
> This does not match my observation. The test run I tried with did not have any
> environments defined at test run level. But it automatically inherited them
> from the product level nonetheless, and the test case environments were still
> ignored in favor of those environments inherited from the product. So it
> appears to me that currently, if the product has environment groups associated
> with it, defining environments on a test case will never have an effect.

That's the key here - product doesn't have to have environments. It should, but it's not mandatory. The only way to enforce assigning groups as a whole would be
to remove API calls for adjusting group's compositions at any level below product. Otherwise, it'll still be potential discrepancies...

Comment 7

[Vadim]

Added simple matching based on group ids only. More advanced matching based on individual environments within each group may be needed in the future.