Configure automated integration and functional test automation
The first thing that comes to mind about Continuous Delivery is that everything needs to be automated.
Otherwise, you can't deploy multiple times a day. But how to deal with testing, then?
Many companies still have a broad suite of manual tests to be run before delivering to production. Somehow these tests need to run every time a new release is created.
Instead of automating all your manual tests into automated UI tests, you need to rethink your testing strategy.
Lisa Crispin describes in her book Agile Testing that you can divide your tests into multiple categories.
Source: https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants
We can make four quadrants where each side of the square defines our targets with our tests.
- Business facing - the tests are more functional and often executed by end users of the system or by specialized testers that know the problem domain well.
- Supporting the Team - it helps a development team get constant feedback on the product to find bugs quickly and deliver a quality build-in product.
- Technology facing - the tests are rather technical and non-meaningful to business people. They're typical tests written and executed by the developers in a development team.
- Critique Product - tests that validate a product's workings on its functional and non-functional requirements.
Now we can place different test types we see in the other quadrants. For example, we can put Unit tests, Component tests, and System or integration tests in the first quadrant.
We can place functional tests, Story tests, prototypes, and simulations in quadrant two. These tests support the team in delivering the correct functionality and are business-facing since they're more functional.
In quadrant three, we can place tests like exploratory, usability, acceptance, etc.
We place performance, load, security, and other non-functional requirements tests in quadrant four.
Looking at these quadrants, specific tests are easy to automate or automated by nature. These tests are in quadrants 1 and 4. Tests that are automatable but mostly not automated by nature are the tests in quadrant 2. Tests that are the hardest to automate are in quadrant 3.
We also see that the tests that can't be automated or are hard to automate are tests that can be executed in an earlier phase and not after release.
We call shift-left, where we move the testing process towards the development cycle.
We need to automate as many tests as possible and test them.
A few of the principles we can use are:
- Tests should be written at the lowest level possible.
- Write once, and run anywhere, including the production system.
- The product is designed for testability.
- Test code is product code; only reliable tests survive.
- Test ownership follows product ownership.
By testing at the lowest level possible, you'll find many tests that don't require infrastructure or applications to be deployed.
We can use the pipeline to execute the tests that need an app or infrastructure. We can run scripts or use specific test tools to perform tests within the pipeline.
On many occasions, you execute these external tools from the pipeline, like Owasp ZAP, SpecFlow, or Selenium.
You can use test functionality from a platform like Azure on other occasions. For example, Availability or Load Tests executed from within the cloud platform.
When you want to write your automated tests, choose the language that resembles the language from your code.
In most cases, the application developers should also write the test, so it makes sense to use the same language. For example, write tests for your .NET application in .NET and your Angular application in Angular.
The build and release agent can handle it to execute Unit Tests or other low-level tests that don't need a deployed application or infrastructure.
When you need to do tests with a UI or other specialized functionality, you need a Test agent to run the test and report the results. Installation of the test agent then needs to be done upfront or as part of the execution of your pipeline.