Sanity testing is a kind of software testing that occurs when a new software build is developed or when minor changes in code or functionality are made to an existing build.
In this article, we’re going to deep dive into the sanity testing definition and details, exploring what sanity testing is, how sanity testing can be approached, and what tools can make sanity testing software simpler and more efficient.
What is sanity testing?
Sanity testing is a kind of software testing that’s performed by testers to ensure that a new software build is working as it should. It’s a quick process that can prevent developers and QA teams from wasting time and resources on more rigorous testing on software builds that aren’t ready yet.
Sanity testing is often used after bug fixes or repairs have been carried out, and it’s designed to test whether these fixes have worked and whether the core functionalities that have been changed now work as they should. After installing the build, testers carry out sanity testing instead of full regression testing to ensure that the build is functional, and the changes have been implemented correctly.
If the bug fixes implemented by developers are working as they should be, the testers will consider the sanity test to have been passed. If they are not working as they should, the build will be rejected and sent back to developers for further changes before deeper testing is carried out.
1. When do you need to do sanity testing?
Sanity testing is usually carried out on software that is stable but not necessarily functional; for example, after minor changes have been made to a software build, software testers may carry out sanity tests to ensure that these changes are working properly before moving on to full regression testing.
Sanity testing takes place after smoke testing, which can ascertain whether a build is stable or not, but before regression testing. For example, if smoke testing reveals instabilities that require repairs, sanity testing could be implemented after changes have been made to fix these bugs to identify whether the changes are working as expected.
2. When you don’t need to do sanity testing
Sanity testing should take place after any changes have been made to a stable software build to verify the functionality of these changes. If you haven’t made any changes to a software build, or if you’re in the middle of implementing changes that aren’t finalized yet, there is no need to sanity test a build.
If you choose not to carry out sanity testing after making changes to a software build, you could save yourself time in the short term, but you run the risk of finding bigger problems later on during testing that halt development and cause serious delays.
It’s always worth carrying out sanity testing after making changes that could affect performance because it’s much better to identify any potential bugs or issues early, before wasting money and resources on more thorough QA testing.
3. Who is involved in sanity testing
Sanity testing is usually carried out by testers after they receive a stable software build for further testing. QA testers carry out sanity testing on individual aspects of the build, for example on single functionalities that have been changed or specific bugs that have been fixed.
In this way, sanity testing offers relatively detailed feedback on very specific areas of the software build. If the tests pass, testers carry out further regression testing. If they fail, the build is returned to developers for further work.
The benefits of sanity testing
Sanity testing saves a lot of time and effort because it prevents QA teams from wasting time on deeper tests before ascertaining that the core functions of the software build are working as they should.
Sanity testing is fast, cost-effective, and necessary if development and testing teams want to create bug-free software efficiently and quickly.
● It saves time and resources
● No documentation efforts are required
● It can help to identify missing objects
● It prevents major issues later
1. It’s efficient and fast
Sanity testing is a fast and efficient way to ascertain whether key functionalities of a software build are working as you expect.
You can carry out simple sanity tests in less than an hour, and if your sanity test passes this gives your QA team the go-ahead to continue with further testing.
2. It does not require documentation
Most sanity testing is unscripted, which means that testers do not have strict requirements to write out the pass/fail criteria of each test or write up documentation to present the results of the sanity test. This means it can be done relatively quickly and casually, without causing significant disruptions to work.
3. It can identify missing objects
Sanity testing can help testers to identify related or missing objects that could be crucial for the function of the build. Because sanity testing is used to test specific functionalities individually, it’s easier to identify individual bugs and issues when sanity testing compared to when carrying out smoke testing and other initial software tests.
4. It prevents major issues later
Sanity check testing can help you to identify issues early on during the testing process and avoid the incidence of major, show-stopping bugs later on in development. Identifying problems early on can help you to stay on schedule during development and prevent costly mistakes.
The challenges of sanity testing
Sanity testing is not without its challenges. Sanity testing software can help testers to identify some of the major bugs in a build before continuing with further testing, but it’s not a reliable way to identify every issue that could arise.
Some of the challenges of sanity testing include:
● It has a relatively narrow scope and can miss some issues.
● Sanity testing is unscripted.
● Developers don’t always know how to fix the bugs found in sanity testing.
● Sanity testing only focuses on commands and functions of the software.
1. It has a narrow scope
Sanity testing has a very narrow scope compared to many other types of testing. The purpose of sanity testing is to test specific functionalities or changes to make sure they’re working properly. Outside of these changes, sanity testing doesn’t offer any insight into the overall functionality of the software build.
2. It’s unscripted
While some testers might consider this to be an advantage, the fact that sanity testing is unscripted means that there is no documentation to look back on in the future if developers or testers want to check the results of a sanity test. Sanity testing has limited use beyond its immediate impact.
3. It only tests functions and commands
Sanity testing is only used to test functions and commands in a software build. You can’t test how the software functions at the design structure level in sanity testing, meaning it’s not always easy for developers to identify where the issues that arise are and how to fix them.
The characteristics of sanity testing
Sanity testing can be differentiated from other forms of software testing based on its key features and characteristics. It’s possible to define sanity testing by considering its characteristics, which are:
● Simple
● Unscripted
● Undocumented
● Deep
● Narrow
● Carried out by testers
1. Simple
Sanity testing is a simple form of software testing that is meant to be easy to design and just as easy to perform. This means that QA sanity testing can be carried out quickly as and when it’s needed without testing teams having to schedule informal tests.
2. Unscripted and undocumented
Sanity testing is usually both unscripted and undocumented, which also contributes to the casual way in which sanity testing can be conducted in most testing environments.
Sanity testing is an informal process that exists primarily to sanity check that changed functions and features work as expected.
3. Deep and narrow
Sanity testing is a type of software testing that is considered to be both deep and narrow. This means that sanity testing covers only a narrow view of a software build but goes into depth on those aspects of the build that it does test.
For example, software testers may sanity test the function of a single feature in detail rather than testing all core features at a basic level.
4. Carried out by testers
Sanity testing is almost always carried out by testers. This differentiates sanity testing from other common forms of software testing like smoke testing, which can be carried out either by QA teams or developers.
Sanity testing vs smoke testing vs regression testing
Sanity testing, smoke testing, and regression testing are often spoken about together and some people may confuse different types of testing if they don’t understand the differences between the sanity testing definition and other types of tests.
Smoke and sanity testing are both fast tests carried out to determine whether a software build is functioning properly. However, sanity tests are different from both smoke tests and regression tests.
1. What is smoke testing?
Smoke testing in QA is a type of software testing that is carried out on new software builds to check for functionality and behaviour. A smoke test is a fast test that runs through the core functionalities of the software to ensure that they work correctly.
For example, suppose you’re testing a mobile shopping application. In that case, you may use a smoke test to check whether customers can sign in, add items to their baskets, and checkout without encountering major bugs or errors.
Smoke tests are also carried out after changes have been made to code in development that could impact the functionality of a build.
2. What is regression testing?
Regression testing is a type of software testing that exists to confirm that the recent changes made to code have not negatively affected the software’s features or functionality.
Sanity testing is a subset of regression testing because it involves testing the function of individual features or modules.
Regression testing is the detailed testing of all areas that have been changed or modified since the last build.
3. What is the difference between smoke and sanity testing?
Like smoke testing, sanity testing ascertains whether certain functionalities are working as they should.
However, unlike smoke testing, sanity testing is focused only on one or two functionalities, usually those which have been most recently changed or repaired. One difference between smoke and sanity testing is that smoke testing gives a broader view of the functionality of a software build, whereas sanity testing gives a narrower but deeper view of a single aspect of the build.
Sanity testing is ultimately a subset of regression testing, which is a type of software testing that testers use to ascertain how a software build functions after changes have been made.
The biggest difference between smoke and regression testing is that smoke testing in QA is carried out on initial or unstable builds, while regression testing is always carried out on stable builds.
Either testers or developers can carry out smoke testing while testers always carry out regression testing.
4. What is the difference between sanity and regression testing?
Regression testing is a superset of sanity testing, meaning that a sanity test is essentially a single small element of a full regression test.
The biggest difference between sanity and regression testing is that sanity testing only tests a few, select areas of code that have been altered to ‘sanity check’ the status of the build, while regression testing tests all areas of changed code to make sure they’re working as expected.
Another difference between sanity and regression testing is that sanity testing is carried out first, with full regression testing only taking place if sanity tests are passed.
5. Smoke, sanity, and regression testing: conclusion
Smoke testing, sanity testing, and regression testing are types of software testing that can help developers and testers to identify errors in code at an early stage of development.
Smoke testing is the first type of testing to take place, and it can be carried out either by developers or by testers on unstable builds. This is the biggest difference between smoke and regression testing.
Sanity testing takes place next, with full regression taking place if both of these first tests pass.
All three types of tests are essential to ensure that development teams and QA teams don’t waste time and resources on software builds with show-stopping bugs that could cause major delays if they’re only found later on in development.
Manual vs automated sanity tests
Modern automation technology makes it possible to automate sanity testing to reduce the amount of time that testers have to spend carrying out these necessary tests.
However, automating sanity tests usually requires more technical resources than manual testing and it can be hard to spare the development time to create and run automated sanity tests without using sanity testing tools.
Often, the best choice is to combine regular automated testing with manual sanity testing to explore core functions in more detail.
1. Manual sanity testing: benefits, challenges, and process
Manual sanity testing is any kind of sanity testing that is carried out manually by human testers. When testing manually, testers validate the key features of the software build themselves by testing the outcomes of various test cases and checking these against the expected outcomes.
Manual testing is often considered to be more detailed than automated testing because it allows for more exploratory testing. While automated tests simply follow a set script, manual testers can use their own insight and judgment to explore features and processes that may require further investigation. In other words, they can go ‘off script’.
The pros of manual sanity testing include:
● Manual testing can easily be carried out by non-technical QA staff
● It’s easy to set up a manual sanity test without specific resources
● Testers can explore different elements of the software build during a manual test
However, there are also a lot of disadvantages to manual sanity testing as well:
● Manual testing is time-consuming and can’t be carried out as regularly as automated testing
● Testing may be less detailed if testers want to save time
● Test coverage may be narrower
● There is room for human error in manual sanity testing
2. Sanity test automation: benefits, challenges, and process
Automated testing is becoming more popular among testing teams with the resources and skills to implement it. Automating sanity testing allows testing teams to conduct sanity tests more regularly and to standardise the sanity testing process across multiple tests.
Sanity testing software using automation tools is one of the fastest and most efficient ways to carry out sanity testing, but it does require that software teams allocate technical resources to the creation and management of automation processes.
In smaller teams, this could draw resources away from crucial processes like development and bug-fixing.
The pros of automated sanity testing include:
● Automated sanity testing is much more efficient than manual testing
● There are no limits to how regularly you can sanity test when using automation
● There’s little room for human error in automating sanity testing
● Automated sanity tests can cover a wider range of samples
However, there are also drawbacks to automated testing, including:
● Automated testing allows no room for subjectivity
● Automated tests can’t explore outside their scripted scenarios
● Automating sanity testing costs resources
● Not all testing teams have the technical skills to automate sanity check testing
3. Conclusion: Manual or sanity test automation?
Ideally, development teams and testers can combine manual QA sanity testing with automated testing for the best results. This allows software teams to benefit from the consistency of automated testing and the flexibility of manual testing.
In the case of both smoke and sanity testing, it costs resources and technical skills to automate sanity testing, meaning this isn’t always possible, especially for smaller software teams or in the case of one-off sanity tests.
Testing teams wanting to explore automated testing can use sanity testing tools to simplify the automation process and reduce the need for additional development staff.
What you need to start sanity testing
Before you start sanity testing, it’s important to determine how you’re going to approach testing and define sanity testing parameters and objectives. You don’t need a lot of actual tools to sanity test, and sanity testing can be largely unplanned.
Most often, sanity testing is carried out because changes have been made to a stable software build and testers want to verify that these changes work as expected.
In this case, you will start a sanity test by outlining the changes made, the processes you’re going to use to test them, and the expected results of each test.
1. A stable build
Sanity tests are carried out once the software build has been tested for stability via smoke testing. It’s the responsibility of developers and testers to ensure that a software build is stable before conducting further tests.
2. Test case scenarios
Before you start sanity check testing, you’ll need to outline the test case scenarios that you’re going to test regardless of whether you’re going to carry out manual or automated sanity tests.
If you’re conducting sanity testing after bugs have been fixed, you’ll want to define test cases that verify the quality of the fixes.
3. Sanity testing tools
You don’t need any special tools to carry out sanity testing, but sanity testing tools can make it easier to carry out tests during the normal course of the working day.
If you want to transition to regular sanity tests throughout the day, or if your development team makes multiple modifications to a software build every day, sanity testing tools could help. For example, you could use testing tools to implement robotic process automation.
The sanity testing process
Software sanity testing is usually a relatively fast process that can be carried out in less than an hour. Automating sanity tests may take longer to begin with, but once your automation script is set up you can carry out sanity tests in no time at all.
Follow the steps below to understand how to carry out a manual sanity test and which steps you’ll need to take at each stage of the testing process.
1. Identify modified components
The purpose of sanity testing is to test the functionality of specific features and components after changes have been made to a build.
Before you can start software sanity testing, it’s important to identify which components have been modified or added to the build and which aspects of the code have been changed since the last round of testing.
2. Evaluate each component
Once you have identified the components that require testing, you can analyze each component individually to understand its attributes and how it’s supposed to work.
This helps testers to understand the expected outcomes of sanity testing and make sense of the results of their tests.
3. Define the sanity testing approach
At this stage, it’s necessary to define your approach to sanity testing. Are you going to carry out manual testing or automated testing?
If you’re using an automated approach, the tools that you use to automate the testing should help you to create test scripts to test the components that you’ve already identified.
If you’re testing manually, consider how to test the functions that you need to verify.
4. Perform sanity testing
The next stage of sanity testing is conducting the testing itself.
Testers conduct manual sanity check testing by assessing all of the components, linked parameters, and functions of the modules that have been edited, added, or modified since the last test.
When sanity testing software, compare the results of each sanity test against the expected outcomes of the test to ascertain whether each component is working properly.
5. Next steps
After carrying out your sanity test, consider whether or not the build has passed or failed. If the sanity tests result in unexpected behaviours or outcomes, return the build to developers for further work.
If the build passes sanity testing, meaning that all of the build components behave in the way that you expect, further regression testing can take place.
Best practices for sanity testing
Because sanity testing is both unscripted and undocumented, testers can conduct sanity testing as and when they need to. There aren’t many recommended best practices for sanity testing because it’s such a casual type of software testing, but there are some rules that you can follow to help you to ensure that you get the most out of your sanity tests.
1. Always sanity test after adding new functionality
Software sanity testing is a necessity when new functions or commands are added to a stable software build.
The most important best practice of sanity testing is to always carry out sanity testing every time a component is modified or added, or when a bug is repaired.
2. Focus on relevant functions and commands
Part of the sanity testing definition is its focus on functions and commands, but when you’re carrying out sanity testing it’s important to focus on those functions and commands that are most important to the function of your software build.
Like smoke testing, sanity testing is best used to assess core functionalities that could cause serious disruptions if not identified at this stage.
3. Always automate tests where possible
If you have the resources, tools, and technical skills required to automate your sanity tests, this can help to both speed up the process of testing and standardize testing methodologies.
This doesn’t mean that automated testing should always be used in place of manual testing, but that implementing some kind of automated testing alongside manual testing is always best.
Types of outputs from a sanity test
Most of the time, the output of a sanity test will simply be a binary pass or fail decision depending on how the components that you test behave under test conditions.
1. Pass
If the code that has been modified has no bugs or logic errors, this should result in your sanity test passing. Passing simply means that when you conduct your sanity test, the modules behave in the way that you would expect them to.
If the sanity test passes, testers continue with further testing and a full set of regression tests.
2. Fail
If the functions that you test don’t behave as you expect them to when you carry out your sanity test, this indicates that the test has failed.
Testers will then pass the software build back to the development team to continue development, repair bugs, and fix any errors in the code that could be causing the tests to fail.
Examples of sanity tests
Learning how to sanity test with example tests is the best way to understand how sanity tests work and how to conduct a sanity test manually.
Below are two illustrations of sanity testing with example test cases.
1. Sanity testing after a bug fix
During smoke testing, developers found bugs within an eCommerce application that prevented customers from adding new items to their baskets.
After carrying out repairs to fix this bug, the build was passed on to QA testers for a sanity test. The sanity test involved testing the functionality of adding new items to the basket to make sure that this worked as expected.
2. Sanity testing after modifications
A team of developers has been working on an update for a shopping list app that allows users to categorize lists with different labels. This involves adding a lot of new code to the existing build to implement this feature.
Once the code has been added, testers carry out sanity testing to assess the new feature and test its performance. A bug arises that prevents users from recategorizing a list once they have already added a label to it, so the build is sent back to the developers for further work.
Types of errors and bugs detected through sanity testing
Sanity testing is commonly used to test the rationality of a software build after modifications have been made that might affect the functionality of the software.
As such, software sanity testing can help QA testers to identify various bugs and errors in computer code.
1. Logic errors
Sanity tests can help testers and developers identify logic errors within the new code. These errors may cause core functions to behave unexpectedly or even cause the software to crash.
2. Bugs
Bugs in computer code can be small or large; in some cases, they may simply impact usability and convenience while in others they can prevent the entire application from functioning at all.
Sanity tests can identify bugs or reveal whether a bug has been fixed adequately or not.
Common sanity testing metrics
Metrics in any kind of software testing should be countable and quantifiable. When you carry out sanity testing, it’s important to keep track of metrics that can help you to assess the output or result of your sanity test objectively.
This is particularly important if you want to automate sanity testing at some point in the future.
Some examples of sanity testing metrics include:
● Test cases not executed
● Test cases passed
● Test cases failed
● Test cases blocked
Effectively, measurable metrics include any results that provide quantitative data that reflects how well your software build has performed during the sanity test.
5 Best Free Sanity Testing tools
If you’re interested in implementing free sanity testing tools to help plan, run, and automate sanity tests on stable software builds, below is a list of some of the best sanity testing tools available online for free today.
1. ZAPTEST FREE edition
ZAPTEST is a free testing tool suite that is available as both a free version and a paid enterprise edition.
The ZAPTEST FREE tool is a software testing tool that allows users to automate sanity tests, smoke tests, and other types of software tests to test applications for Mac, Windows, Android, and other platforms.
It’s easy to operate and the ideal way to try out sanity testing automation without paying a premium.
In brief, ZAPTEST’s 1SCRIPT technology allows for test automation on any software application, cross-platform, cross-browser, cross device, and in a codeless interface that is ideal both for beginner and very experienced testers.
2. QA Wolf
If you’re looking for simplicity, QA Wolf is a delightfully simple QA testing application that’s fully hosted in your browser, which means you won’t need to download anything to use it. You can use QA Wolf to carry out automated tests no matter what your skill level is.
3. Selenium
Selenium is another testing tool that’s available as both a free edition and a paid for version. Selenium is compatible with a lot of programming languages, which makes it a great choice for development teams using less common languages, and it can be used to automate sanity testing and other types of testing for web applications.
4. Watir
If you want to get started writing your own automated software tests but you don’t know where to start, Watir is an open-source tool that makes it easy to write simple and maintainable automated sanity tests.
5. Windmill
Windmill is an open-source testing tool that was created to automate the testing and debugging of web applications. It’s an effective tool for sanity testers wanting to check that web applications have been properly debugged during the development stage.
Sanity testing checklist
Before you conduct your first sanity tests, make sure that you understand how to define sanity testing and what you’ll need before you start sanity testing.
● Do you know what new functionality has been added to the build?
● Do you understand how the new functionality is supposed to work?
● What are your criteria for passing and failing the sanity test?
● Do you need to acquire any sanity testing tools before you begin?
● How do you plan to relay the results of your test to developers?
● Do you know how to repeat the sanity tests if needed?
Once you know all of the answers to these questions, you’re ready to start your first sanity test.
Conclusion
Sanity testing is a necessary step in software testing that allows testers to assess whether recently modified components are working correctly. Sanity testing is always carried out by testers rather than developers, and it’s possible to automate sanity testing or carry it out manually.
As more software teams move towards hyperautomation, automated sanity testing is becoming increasingly common. Ideally, software teams may aim to carry out manual, exploratory testing when testing new components while using automated testing to test small changes throughout the working day.
FAQs and resources
If you want to further your knowledge of sanity testing, check out some of the resources and FAQs below.
1. Best courses on sanity test automation
You can learn more about sanity testing and other types of software testing by looking for online courses in sanity testing. You can find courses online at websites like:
● Coursera
● Uplatz
● Course Line
● Edureka
Some online courses are offered for free, while others may offer certifications or qualifications upon completion for a fee.
2. Best books on sanity testing
You can improve your knowledge of sanity testing by reading books about sanity testing and software testing.
● Software Testing, by Ron Patton
● How to Break Software, by James Whittaker
● Software Testing Techniques, by Boris Beizer
● Software Test Automation, by Mark Fewster and Dorothy Graham
● Agile Testing, by Lisa Crispin and Janet Gregory
3. What are the top 5 interview questions on sanity testing
Before you apply for QA jobs that may involve sanity testing, you can prepare answers to common sanity testing interview questions.
● What is the difference between smoke and sanity testing?
● When should you carry out sanity testing?
● How do you determine whether a sanity test has failed?
● When might you carry out manual vs automated testing?
● What are the advantages of sanity testing?
4. Best YouTube tutorials on sanity testing
You can learn more about sanity testing from these YouTube videos:
● What is Sanity Testing?
● The Difference Between Smoke and Sanity Testing
● What is Sanity Testing? Purushotam Academy
● Smoke Testing vs Sanity Testing with Examples
5. How to maintain sanity tests
Because sanity tests are usually used to verify modifications made to code, each time you run a sanity test you may be testing different elements of the code or adapting your test to assess different functionalities.
For this reason, it’s important to stay on top of your sanity test maintenance to ensure that you’re ready to test whenever it’s needed.
● Update test cases as the functionality of your software build evolves
● Always follow test design best practices
● Re-evaluate your tests on a regular basis
● Keep future projects in mind when creating new tests
6. What is sanity testing in QA?
Sanity testing in QA Is a type of software testing that involves testing newly modified or added components of a stable software build to make sure they’re behaving correctly.
This sanity testing definition distinguishes sanity testing from smoke testing because smoke testing is carried out on unstable builds.
Sanity testing software is always carried out by testers rather than developers, and one of the most common reasons to carry out a sanity test is because a bug has been fixed or repaired. In this way, testers ensure that the fix is working correctly and further testing can begin.
Of course, if you an organization require Enterprise level software testing + services do get in touch! ZAPTEST is the leading automation tool on any platform, including Linux, Windows, Android, iOS, web. It allows for any test, including load tests, performance tests, UI tests, unit tests, functional tests, integration testing, UI tests, complex API tests, and many many more!
