The goal of this post is to underscore the significance of the Test Phase in the DevOps infinity loop. To go over the tools, frameworks and methods, types of tests and overview of the AWS ecosystem and services available for the implementation of DevOps Best practices and the Test Phase in particular. This isn’t going to be a technical post, but rather more general, informative one. It would not include technical details on how to setup a pipeline and/or deploy the code to your infrastructure, but explain highlights, options and services involved and options available.
Test Phase
The DevOps practices require a fast release of software, with possible multiple deployments per day. To meet these requirements teams have to be able to test the code within minutes and as frequent as possible. The Test Phase will determine if the build is ready to go to the next stage or it will block it based on the tests results without human intervention.
Testing software is an essential step in the Software Development Life Cycle (SDLC), being able to detect bugs early and resolve prior to a production release is invaluable. Including a Test Phase into your pipeline gives you a huge benefit of identifying existing bugs. In the Test Phase, we use automated testing tools, which vary depending on the application. For instance tools like Newman will be an easy way of testing your API public methods, JUnit/Xunit or Jest would be a great for unit testing of your code/components and Playwright or Cypress are ideal for a full e2e tests implementations up to UI level if your application is a Web application.
The Test Phase is also a phase where you can integrate test management tools like TestRail. This tool generates comprehensive reports on the codebase and testing results, providing stakeholders with insight into the development cycle and application maturity. Such abilities can provide the stakeholders in your company with reporting and help learning more about the development cycle and application maturity. One thing to adopt is a mindset where the Quality lives everywhere across the Organization and not only within the boundaries of a QA Department. This mindset will help you to achieve the best results , the Test Phase will help you to build the path to success in this journey.
Why Test Phase is important
The most important/critical benefit of Test Phase is enabling testing, quality validation, and performance evaluation to happen as early as possible in the SDLC, this shift left practice is a departure from traditional or manual testing, which occurs mostly at the end of it.
Here are some obvious benefits of Test Phase :
- Implementing tests early in the SDLC enables a continuous testing process, allowing for faster and continuous delivery of software. This approach supports testing quickly, frequently, and early, reducing the risk of errors going undetected.
- Automating testing eliminates the human factor of skipping errors, which would necessitate the maintenance and update of test scenarios. Consequently, there may be a need to have a dedicated team of software developers in testing.
- Each step of the SDLC involves different forms of testing (unit testing , API testing, e2e and UI testing). This minimizes backtracking in the case that you’ve detected an error.
Implementing this Phase reduces the time to deliver applications, and errors and bugs are minimized, helping implement a shared quality responsibility model where the entire Engineering Department is responsible for the app’s quality.
About the Test Pyramid
The Test Pyramid is a well-known concept in software development and has been discussed in different articles and writings, but the original description was made by Mike Cohn in his book Succeeding with Agile. The metaphor of the pyramid helps to visualize the different layers of testing that should be implemented in software development.
Behavior-Driven tests are the most effective, as they incorporate test scenarios and expected results. These tests assume that they cover the integration between components, making integration tests the most valuable as they test the system as a whole or in isolated and complex modules, rather than just individual units. Therefore it might sound like the investment into the UI e2e tests (run on top of the system after it has been deployed and fully integrated) is a great idea, which is actually true. It also has some pitfalls, example: having only UI e2e tests could potentially lead to disproportionate spend, maintenance headache and stress.
Investing solely in UI end-to-end tests often leads to an “ice cream cone” anti-pattern in the test pyramid, also known as an atypical or flipped test pyramid..
Maintenance of this cone will become a nightmare, the cost could get out of proportions, the tests results could became inconsistent and your organization will probably loose the benefits of a proper Test Phase.
Now we can go over the layers of Test Pyramid, popular test types and the benefits of each.
- Unit Tests : The basic and most granular ones, focusing on one unit of work , usually a method or a component. Those tests live in the bottom of the Test Pyramid and could be complied and executed within your Build stage. Those tests are easy to implement, they are cheap and easy and can provide a first line of defence when it comes to the quality of code.
- Integration Tests/API Tests : Integration made using the API so testing this layer is crucial to your systems, these tests have to validate the ability of the system to work in integration (therefor validate that unit of work can exists and live in in integration between them). Could be implemented by developers or quality practitioners, depends on the organizational structure. At the end of the day no one would like to onboard the aircraft that was only unit tested..
- UI e2e Tests > The most comprehensive tests, should follow the BDD approach to support the scenarios, these tests will require the full integration of your system including FrontEnd, BackEnd, DataBase and Networking. Those tests usually written by quality practitioners in collaboration with business or product owners. Because of the nature of these they are the most costly ones as well as the most slow and require more maintenance as the business requirements and test scenarios are tend to change more frequently
Test Automation is…
When you have an ever-increasing amount of software in your possession or even if you don’t and it is pretty static one you would have a hard times testing your application manually. Ensuring the repetitive process is done appropriately and reducing human error factor is close to impossible when it is done in a manual manner. So this where the Automation is coming to your help, Automation has become a standard with pretty much everything, from from infrastructure via orchestration of your builds and testing your code/applications
In practice, this means the developers will focus on writing unit tests and integration tests as part of the deliverables or tasks, while the quality practitioners will focus on writing the UI e2e tests based on scenarios provided by the business or product owners, the test discovery phase, mainly done using manual testing would be done as a prerequisite stage prior to the Automation
In order to achieve the Automation of the Test Phase you can use one of the available tools on the market today, the possibilities are endless and mainly depend on your needs and the skills of your development organization. Playwright and Cypress will help you to achieve the goals with targeting UI e2e tests , both tools have pros and cons and nowadays you can even utilize them as one shop stop (testing all layers of your Test Pyramid) for your testing needs if your stack is JS for example. Another layer is Integration one, and the variety of tools also wide open : Katalon, Newman by Postman and others. Always review the features and take into consideration the complexity, interoperability and the ability to be integrated into CI/CD pipeline (as the whole purpose of having those is ability to integrate them into you Test Phase, which means include them in your CD/CD pipeline). The last but not the least is a layer of Unit tests, to write and execute those you would need to use the tools that work best with the you codebase: for .Net code you can use XUnit, for Java use JUnit and for code written in JS or TS you can utilize Jest (or as I have mentioned above tools like Playwright are modern and versatile and if your codebase is written in JS or TS you can use Playwright to test your entire pyramid. Please refer to the original documentation of Playwright).
AWS CodePipeline for your Test Phase implementation
Test Phase could be implemented using different tools/services that exist on the market today. Services like Jenkins and CircleCI will let you do the same steps in order to achieve the desire results : you need to pull your code from the Source code > Build you code > Test it > Deploy to Pre-Production (Staging) > Optionally Test Again > Deploy to Production. Here you go the deployment has completed.
Many tools/services exist out there but let’s talk about AWS services. AWS CodePipeline is a fully managed continuous delivery (CD) service that helps to build pipelines, orchestrate stages and deliver updates to your application and infrastructure. AWS CodePipeline works great with other AWS DevOps services like AWS CodeDeploy, AWS CodeCommit, AWS CodeBuild as well as known third-party action providers such as Jenkins and Github.
The simple use case for the pipeline is : Pull Code > Build it > Deploy it. That looks simple and in this case you can only execute the unit tests on the Build stage of the pipeline. Great! But what about having more complex pipeline when the Test Phase will include integration tests and even the UI e2e tests, so the pipeline will look like this : Pull Code > Build it > Deploy to Staging > Test it > Deploy to Prod. The difference here is that extending the pipeline we are adding the testing stage that could run integration or/and UI e2e tests agains our Staging environment we deployed in the previous Stage of the pipeline.Now this is much better!! Adding the Test Phase here would be a straight forward as adding the extra step (Stage) via the Edit page of the existing pipeline and configuring the execution of your tests using the native test framework execution (depends on the test tool you have).
As this service is versatile and can interact with many other AWS and third-party services it is hard to pick one and use an example, but it is important to mention some useful and important functionalities of the AWS CodePipeline :
Detection option: is a property of a pipeline that will start the pipeline and based on the source location of the artifacts AWS recommends to use Github webhooks for Github or Amazon CloudWatch Events for artifacts stored in AWS S3 bucket for example.
Disable/Enable transition: The transition links the stages of the pipeline and it is enabled by default. If for some reason you don’t want to automatically proceed from one stage to another you can disable by clicking “Disable transition” button and by doing that, it prevents the continuous execution of the pipeline. It is easy re-enable it back.
Add Stage: in order to edit the sequence of the pipeline and introduce the new stage or remove/update the existing stage, you need to “Edit” the pipeline it will navigate you into the Edit page where you can add actions to your stage serially or in parallel to your existing actions. This functionality makes your pipeline flexible and extendable.
Approval Action: when you want to manage the stages of your pipeline, for example having someone to approve the deploy stage, you can use this functionality, that will hold the pipeline till the approval and will resume the execution after the approval happened.
Other DevOps AWS Services that CodePipeline works and integrates well are:
- AWS CodeCommit
- AWS CodeDeploy
- AWS CodeBuild
Summary and Conclusion
No pipeline should exist without a Test Phase, and no release should exist without a Test Phase. Minimize the human factor, rely on the automation and the tools given to achieve this that what the focus of the DevOps specialist responsible for the CI/CD should be. Not only the DevOps specialist who would probably build the pipeline, but the whole Development organization including Developers, QA personal and DevOps team, the right approach is to see the Quality as responsibility of all and to see the Test Phase as an available, convenient, flexible and bulletproof Phase. The use of a right service like AWS CodePipeline will help to achieve the desired results!
Links and Resources:
