question

NairDeepak-6020 avatar image
0 Votes"
NairDeepak-6020 asked JarvanZhang-MSFT edited

Xamarin UI Test- Inconsistent Behavior

We have developed a Xamarin(Xamarin.Android & Xamarin.iOS) application. We have written UI Test cases for the same using Xamarin.UITest test suite.We are using the latest version of Xamarin.UI Test nuget package(3.0.12) and using Microsoft Visual Studio Enterprise 2019 Version 16.8.4

. We have noticed inconsistent behaviour for our test runs. Please find our findings below.
The same test case which was passed for one test run is seen failed on the next test run. - inconsisitent
Please see the below screen shots for Test Case 05(a).
• Sometimes we could see some kind of a time lag on the device which we run that the test explorer status is not matching with the device screen. For example, in the test explorer, we might see Test case no: 4 is being executed, but in the device we could only see the test case no: 3 is just being executed. Once this happens, we could find all the next Test cases fail.
• Also for some events (for example, the item click of the Navigation drawer) it takes more time to trigger the event. This too happens intermittently that same type of events consume different timings.

63994-test-run-3.png


63956-test-run-1.png


63849-test-run-2.png


dotnet-xamarinvs-testing
test-run-3.png (133.7 KiB)
test-run-1.png (135.3 KiB)
test-run-2.png (159.9 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

hi, NairDeepak-6020. Are all test cases working properly? And will they influence each other? Please make sure that the test cases work well and try to reduce the number of cases to test to check the issue.

Here are some advice on improve the performace of UiTest, you could refer to: https://devblogs.microsoft.com/xamarin/best-practices-tips-xamarin-uitest/#be-as-deterministic-as-possible

0 Votes 0 ·

hi JarvanZhang,
Thank you for the suggestion. I will definitely refer the link you have provided. But, I do have a question here. I am seeing inconsistent results for the Test case. Say, Test case 2 is a success and 3 is a failure on my first test run. On next test runs, I may see Test Case 2 has failed and 3 got passed successfully. Any idea why we may see this?

0 Votes 0 ·

Do the test case 2 and 3 rely on another case? Please make sure every test is independent.

0 Votes 0 ·

1 Answer

JohnHardman-6852 avatar image
0 Votes"
JohnHardman-6852 answered NairDeepak-6020 commented

You haven't shown the source of your tests, so these are generalities:
(1) Don't have any test rely on state resulting from another test. Tests should be independent, and should clean up anything that might be left over from a previous test.
(2) If a test waits a certain length of time for something to happen, make sure that the test checks that whatever it is happened, rather than just continuing on the assumption that it happened. e.g. don't wait 3 seconds for a page to be populated and ready to accept user input. Instead, wait for the view that you want to interact with to be visible and check that it has focus. If it doesn't get focus within 60 seconds (not 3), fail that test. Tests should be designed to be resilient, not fragile, so should (except in very specific circumstances) not fail because the test runs slowly.
(3) Read "The Art of Unit Testing" by Roy Osherove. I don't agree with everything in it, but it contains a lot of useful advice that applies to UI testing too.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank You JohnHardman-6852,
I will review my code with the valuable suggestions you have provided.

0 Votes 0 ·