How to Shift Left your test strategy? - Part 1

Modern software products are being delivered as a service (SaaS) using cloud technologies. This secular move to cloud and SaaS has changed the way software is developed and deployed. SaaS products give you the means to deliver features and functionality to all your customers very quickly and very often. This also means that any quality issues or outages of your SaaS product will result in large number of unhappy customers. If you look at the root-cause analysis (RCA) of quality issues and defects, you will find that most bugs could have been caught during the design and coding stages. In order to meet the tough quality criteria for SaaS products, we need to test early and test often. This movement towards testing early is what is being called “Shift Left”.

With Shift Left you give your developers the processes, tools, and systems to build and test the product in the most effective manner. However this is easier said than done because Shift Left requires a change in the engineering culture. In a 3-part blog series I will share ideas and resources that will help you shift your testing to the left - closer to design and development phases. And in the process bring about a necessary culture change in your cloud product engineering teams.

Key principles for Shift Left Testing

Before we look at specific ideas about shift left testing, let us look few key steps that will help change the mindset of your software development team.

• Remove the silos between software development and test team - this might be hardest if your organization is designed for separation between Dev and Test. But instead of thinking about this as an organizational change activity, ensure all engineers understand the importance of quality and take responsibility for quality.
• Define clear ownership - Shift left requires developers to embrace testing and testers to write code for fast and reliable automation. However it is more important to define clear ownership of different test areas. The teams also need to work jointly in creating tools, simulators and CICD pipelines.
• Elevate the importance of test strategy - all software engineering teams know the importance of architecture, design etc. These are reviewed with great interest whereas test architecture and design is relegated to the backend of the software development process. When a new product or a major change is introduced, develop the test strategy while developing the architecture and design. This will also ensure that senior test engineers are aware of the bigger picture.
• Set clear performance goals for your tests - Every software developer (and tester) understands that a product must undergo tests to meant the performance requirements. What about performance requirements for your tests? How quickly should your tests run? With CI/CD, the performance of automated tests is a key indicator of the time it takes to fix an issue and deploy it.
• Focus on the criticality of test infrastructure and services - Instead of considering test automation as a bunch of scripts, encourage engineers to think of test infrastructure just like they would think about production infrastructure. The availability, reliability, and stability of test infrastructure has direct impact on the quality of testing.

Start with the Test Pyramid

We will start our journey of Shift Left testing by looking the types of test cases and how to align them with your test strategy. Test pyramids such as the ones depicted below, have been used frequently to define various test types (or layers).

On the left side you have the ideal test pyramid - for Agile team. And on the right you have the pyramid that is most commonly seen in waterfall model of software engineering. The reality is going to somewhere in the middle but still closer to the right.

The “integration” test layer of the pyramid is often the least understood and also has a wide ranging definition. This leads to poorer coverage of integration tests and hence resulting in a test hourglass (instead of a pyramid). Therefore it is important to be more specific about integration tests. Is it API tests? Does it require a subset of services and infrastructure?

With this exercise, you may end up with more than 3 tiers of test cases, but the key takeaway should be that test strategy needs to “shift left” towards the Agile test pyramid. Why? The test pyramid not only helps in segregating tests based on purpose, it also classifies test cases based on the time taken. The Unit tests are larger in number but they usually take less time when compared to end-to-end tests. Therefore writing more UT not only gives you more coverage, it gives you more test coverage in relatively lesser time. The time taken by tests is a critical factor in CI/CD pipelines, therefore organizing your test cases into tiers is vital to Shift left strategy.

Next up

We have seen some important steps in creating the right test culture and how organizing your tests into tiers is useful for the Shift-left test strategy. In the next blog, I will share highlights from a curated set of blogs that will help you understand the scope and benefits of each test tier.