Software Testing Skills Must Shift with AI

Artificial intelligence (AI) is touching many parts of our lives and careers. Rama Anem discusses the impact of AI on software testing and how testers’ skills must adapt.

Every technical revolution has had its own followers and others who do not welcome it. Fear of change, unknown futures, and loss of jobs are completely understandable. This is nothing new, and we hope this time history will not repeat itself and a new crop of Luddites will not show up as was seen in 19th century England.

Revolution does not start with the invention of new technology, but with its wide adoption in industry. Invention that stays on the shelf and does not provide any benefit to society will never change anything, even if it is “The answer to the Ultimate Question of Life, the Universe, and Everything” (1).

Machine learning, a form of AI (Artificial Intelligence) has been around 60 years and was used mostly in areas that were too boring or too difficult for people (high risk of human error). Even expert systems, which are not AI in our current understanding (it could not learn), could provide an answer or diagnosis that is more precise than the answer from an average expert. One of the earliest examples is MYCIN, which was created more than 40 years ago to identify bacteria causing severe infections (2).

What is AI?

The word AI is appearing more and more in technology articles, newspapers and websites, and raising discussions in society. AI is already here, not in the form of Skynet (3), of course, but the one helping people.

AI or artificial intelligence is the simulation of human cognitive functions by machines. This includes learning (the ability to make new decision based on processed information), reasoning (using the rules to reach approximate or definite conclusions), self-correction (changing behavior based on outcome).

There are many applications of A.I., but the ones that have active development now are:

  1. Computer vision
  2. Speech recognition
  3. Expert system
  4. Game bots
  5. Machine learning
  6. Data mining
  7. Natural language processing

Of course, this division is not strict, all these applications are interrelated, and one is not imaginable without the other. Machine learning is already widely used, especially in image processing (medical imaging processing, face recognition), voice recognition and translation. Machines can train themselves new skills knowing the expected correct outcome.

How quickly it evolves these days

Any form of AI requires computing power, data storage and, for it to be used by people, some type of interface. There are several identifiable reasons why AI made such development leaps in the past decade:

  1. Moore’s law (4). Computing power doubles every 2 years, making operations cheaper for end customers, which allows using computers and automation in more and more industries.
  2. Demand on workers. There is higher demand on workers in almost any industry, and this requires making processes more efficient. Computer/AI assisted approaches could save lots of time and money and free some resources to work on new products.
  3. Availability of internet. Internet access can work both ways: it provides unlimited access to data for AI and makes AI more available for end users.
  4. Big data. Another new term. Analysis of large volumes of data is virtually impossible for a human, so this is where AI comes in to find patterns in the data.

These demands inspire development of new services and applications. The main goal is to increase efficiency and productivity, leaving more space to creativity.

Another groundbreaking new approach is AI that builds AI and does it better than humans: Google’s AutoML (5). Why would such AI be required? Industry is moving forward way too fast, and there are not enough specialists to satisfy market demands.

How will AI affect software testing?

AI will not steal QA (quality assurance) jobs as many people think and say these days (at least in the observable future), but it will have a huge impact on QA process and techniques. QA engineers will either have to learn new skills or change professions to a field of work that is more conservative.

There are already some frameworks and services that can generate and execute test scenarios without any or minimal human involvement (Appdiff (6), ReTest (7), myWizard (8)). These are not business logic tests yet, because AI cannot know what is correct logic without providing any additional info (like human QA). However, AI does understand gestures (for mobile phone apps), input requirements for each type of field and overall user input scenarios. All it has to do is to find all fields and run all combinations of tests. Maybe, with the experience of finding bugs, it could do some tests earlier to find bugs faster.

AI with a different experience (knowledge base) will have different skills and, unfortunately, the outcome using one or the other tool will not be predictable.

What does that mean for testers?

Their jobs will change. Most of the boring day-to-day tasks will be automated, and QA will have to concentrate on constant process improvement, creative tests, and analysis of business logic. The most important task will be training AI using new experience. This is a new skill that QA will have to learn.

“You have to be smarter than the programmer to find problems in the code.” (9) said somebody on the Internet. This is already true. Testers already must know the application as well as the developers and in some cases even better:

  • Proficient I business logic
  • Know DB/API side of application
  • Know security testing
  • Load testing
  • Understand industry standards of design
  • Create automation tests
  • And a lot more.

The introduction of AI will add more to this list: new frameworks, new ways of thinking and new skills.

The question is how should it be done. Currently there are not many people who are experts in AI creation. This will be a new skill to choose the right facts, data and “feed it to” to AI engine.

One thing will not change: testers will need to think to go above and beyond to find design and requirement bugs to ensure excellence in the product, which cannot be done by AI yet.

What benefits will be gained by adapting these changes?

There are several ways that QA processes could benefit from using AI: helping to do your work, doing your work instead of you, and doing analysis to make your work more effective.

How it could help you to do your work? AI could help to organize test sets: analyze coverage and remove duplicates and overlapping areas, which eventually will reduce the cost of automating and maintaining test sets.

Data analysis will help to identify areas with high concentrations of defects producing defect heat maps for the application, that will help in choosing a more effective test strategy. This also could help to improve the development process by identifying weak parts of and applications and the most common errors.

What about AI doing work instead of you doing it? Existing AI frameworks can already provide automated test generation and execution. What does that mean for QA? More time for creative tests, business logic tests and other projects.

A.I. works fast, has virtually unlimited memory, and is tireless. It’s a perfect worker to test all the combination of user actions, validate behavior of every control and test all combinations of input data. This could increase test coverage without increasing execution time.

Another benefit of using AI assisted test frameworks is reducing maintenance cost for automation, which currently is a challenge for companies that try to balance between test coverage and test set maintenance after UI/API/Data changes. AI should be able to map desired outcomes with new inputs and validate if a change in results falls into an acceptable range.

AI assisted automation would also be a game changer for applications with non-detrimental responses, which are challenging to automate. One of the examples is IoT (internet of things) automation. Sensors give ranges of values, so AI will be helpful in the validation of acceptable ranges and degrading change.

To summarize:

There will be less work for QA workers per line of code, but there will be more code and products.

  1. Coverage will increase: AI can automatically create test scripts for every boring test scenario often omitted by manual testers/automated tests created by humans
  2. As software becomes less deterministic ordinary automation and test cases (TC) will not work – validation requires intelligence and knowledge base. Example – you cannot create asserts for data from sensor because you cannot know the exact values, sent by device at this moment. Another example – testing of integration with search engines: results may vary with new information flowing into the system
  3. The difference between AI and current automation is simple: current automation just follows scenarios generated by users, but AI may develop scenarios to do the job to achieve some task itself by providing measures of success.

What changes should software testers acquire to cope up with it?

Testers will have to accept a new reality as seen with automation, devops and others. Some changes will be similar to the ones required to adopt automation.

  1. New tools. The bar for effectiveness of testing will be raised. To compete with others, testers will have to adopt new tools, learn how to effectively use them, and understand where new tools can be used.
  2. The volume of deterministic tests will decrease. Software becomes less predictable and more connected to the world and may change behavior over the time. The classic if-then-assert tests will not be able to the job anymore.
  3. AI will take over a part of the testing load itself. And this is good. No more boring tests, great coverage and enough time for creativity.
  4. Testing AI with new skills that will require understanding AI basics and how information is processed.
  5. ‘Clickers’ will definitely die out completely. Even now Google offers services that will provide a list of improvements and errors on a webpage. With more advanced AI tools, tasks like UI validation, load tests, and speed tests of apps with UI will be and should be done by machines. As a paradox, testers will do less testing. The same thing happened with the adoption of automation: testers work with code more. This does not mean that testers will die out completely, but the nature of their job will change again.

The advice is to learn these skills before it is too late to jump on the AI adoption train.


As many technologies that were feared by people, AI just becomes another very helpful tool which benefits society in everyday life (personal assistance, translation services etc.), healthcare (e.g. evaluation of tissue health on CT or MRI images) or on professional levels (automated QA). As with any tool, it is useless without a goal, and future QA professionals will be the ones to give it goals, to improve it, and to choose relevant inputs to test business logic. People are not going to be obsolete, they just will be able to work faster and more efficiently. Testing will become more interesting for people, since machines will do most of the routine, boring tasks.


  1. Phrases from The Hitchhiker’s Guide to the Galaxy
  2. MYCIN: Artificial Intelligence Program
  3. Skynet
  4. Moore’s Law
  5. Google’s AutoML AI Won’t Destroy the World (Yet)
  6. AppDiff
  7. ReTest
  8. Accenture myWizard
  9. Will AI Bots Steal Your QA Testing Job?