Concurrent testing

<h2 id="selected-history--approaches-of-testing-concurrency">Selected history & approaches of testing concurrency</h2>
Approaches to concurrency testing may be on a limited unit test level right up to system test level.<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a>
Some approaches to research and application of testing program/software concurrency have been:

<ul><li>Execute a test once.<a class="footnote-ref" id="fnref:9" href="#fn:9">9</a>: 63 </li></ul>
This was considered to be ineffective for testing concurrency in a non-deterministic system and was equivalent to the testing of a sequential non-concurrent program on a system
<ul><li>Execution of the same test sequence multiple times.<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a>: 63 </li></ul>
Considered likely to find some issues in non-deterministic software execution.
This later became called non-deterministic testing.<a class="footnote-ref" id="fnref:11" href="#fn:11">11</a>
<ul><li>Deterministic testing.<a class="footnote-ref" id="fnref:12" href="#fn:12">12</a>: 63 </li></ul>
This is an approach to set the system into a particular state so that code can be executed in a known order.
<ul><li>Reachability testing<a class="footnote-ref" id="fnref:13" href="#fn:13">13</a><a class="footnote-ref" id="fnref:14" href="#fn:14">14</a></li></ul>
An attempt to test synchronisation sequence combinations for a specified input (shared variable access not being corrupted, effectively testing race conditions variables). The sequence is typically derived for non-deterministic test execution.
<ul><li>Structural Approaches / Static Analysis</li></ul>
Analysis of code structure and static analysis tools.
An example was a heuristic approach<a class="footnote-ref" id="fnref:15" href="#fn:15">15</a>
This led to code checker development, for example jlint.<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a> Research and comparison of static analysis and code checkers for concurrency bugs <a class="footnote-ref" id="fnref:17" href="#fn:17">17</a>
See also <a href="/facts/List_of_tools_for_static_code_analysis/E9bcvnrD">List of tools for static code analysis</a>
<ul><li>Multi-user approach</li></ul>
This is an approach to testing program concurrency by looking at multiple user access, either serving different users or tasks simultaneously.<a class="footnote-ref" id="fnref:18" href="#fn:18">18</a><a class="footnote-ref" id="fnref:19" href="#fn:19">19</a> : 745 
Testing software and system concurrency should not be confused with <a href="/facts/Stress_testing/zdLHAs0G">stress testing</a>, which is usually associated with loading a system beyond its defined limits. Testing of concurrent programs can exhibit problems when a system is performing within its defined limits. Most of the approaches above do not rely on overloading a system. Some literature<a class="footnote-ref" id="fnref:20" href="#fn:20">20</a>: 745  states that testing of concurrency is a pre-requisite to stress testing.

<h2 id="lessons-learned-from-concurrency-bug-characteristics-study">Lessons learned from concurrency bug characteristics study</h2>
A study in 2008<a class="footnote-ref" id="fnref:21" href="#fn:21">21</a> analysed bug databases in a selection of open source software. It was thought to be the first real-world study of concurrency bugs. 105 bugs were classified as concurrency bugs and analysed, split as 31 being deadlock bugs and 74 non-deadlock bugs. The study had several findings, for potential follow-up and investigation:

<ul><li>Approximately one-third of the concurrency bugs cause crashes or hanging programs.</li>
<li>Most non-deadlock concurrency bugs are <a href="/facts/Atomicity_(programming)/u3pnNCvs">atomicity</a> or order violation.</li></ul>
I.e. focusing on atomicity (protected use of shared data) or sequence will potentially find most non-deadlock type bugs.
<ul><li>Most concurrency bugs involve 1 or 2 threads.</li></ul>
I.e. Heavy simultaneous users/usage is not the trigger for these bugs. There is a suggestion that pairwise testing may be effective to catch these types of bugs.
<ul><li>Over 20% (7/31) deadlock bugs occurred with a single thread.</li>
<li>Most deadlock concurrency bugs (30/31) involved only one or two resources.</li></ul>
An implication that pairwise testing from a resource usage perspective could be applied to reveal deadlocks.
<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Software_testing/VoXzG9Tb">Software testing</a></li>
<li><a href="/facts/Scalability_testing/tz1jpRkL">Scalability testing</a></li>
<li><a href="/facts/Load_testing/fHRPN97k">Load testing</a></li>
<li><a href="/facts/Software_performance_testing/qtlCrHme">Software performance testing</a></li>
<li><a href="/facts/Scenario_analysis/SHMWDWGv">Scenario analysis</a></li>
<li><a href="/facts/Simulation/Q4QJV1oI">Simulation</a></li>
<li><a href="/facts/Stress_test_(hardware)/5CNpjlW8">Stress test (hardware)</a></li>
<li><a href="/facts/System_testing/ThxRDsAy">System testing</a></li></ul>

<h2 id="general-references">General References</h2>
<ul><li><a href="https://web.archive.org/web/20160819141940/http://www.softwaretestpro.com/Item/5227/What-is-Concurrent-Testing/Testing-Management-Best-Practices-Quality-Assurance-Software-Project-Management-Tools">What is Concurrent Testing?</a> at the <a href="/facts/Wayback_Machine/nmQ3a6JC">Wayback Machine</a> (archived 19 August 2016)</li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Wang, Chao; Said, Mahmoud; Gupta, Aarti (21–28 May 2011). Coverage guided systematic concurrency testing. ICSE '11 Proceedings of the 33rd International Conference on Software Engineering. Waikiki. pp. 221–230. <a href="/wiki/Aarti_Gupta_(computer_scientist)" target="_blank">/wiki/Aarti_Gupta_(computer_scientist)</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Dustin, Elfriede (28 December 2002). Effective Software Testing: 50 Ways to Improve Your Software Testing. Addison-Wesley Longman. p. 186. ISBN 0201794292. <a href="0201794292" target="_blank">0201794292</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Leiner, A.L.; Notz, W.A.; Smith, J.L.; Weinberger, A. (July 1959). "PILOT—A New Multiple Computer System". Journal of the ACM. 6 (3): 313–335. doi:10.1145/320986.320987. S2CID 19867617. <a href="https://doi.org/10.1145%2F320986.320987" target="_blank">https://doi.org/10.1145%2F320986.320987</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Dijkstra, Edsger W. (May 1968). "The structure of the "THE"-multiprogramming system". Communications of the ACM. 11 (5): 341–346. doi:10.1145/363095.363143. S2CID 2021311. <a href="https://doi.org/10.1145%2F363095.363143" target="_blank">https://doi.org/10.1145%2F363095.363143</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">"Concurrent Software Testing: A Systematic Review" (PDF). Archived from the original on 24 September 2015. Retrieved 4 March 2014.{{cite web}}: CS1 maint: bot: original URL status unknown (link) <a href="https://web.archive.org/web/20150924220657/http://www.icmc.usp.br/CMS/Arquivos/arquivos_enviados/BIBLIOTECA_113_RT_359.pdf" target="_blank">https://web.archive.org/web/20150924220657/http://www.icmc.usp.br/CMS/Arquivos/arquivos_enviados/BIBLIOTECA_113_RT_359.pdf</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Wang, Chao; Said, Mahmoud; Gupta, Aarti (21–28 May 2011). Coverage guided systematic concurrency testing. ICSE '11 Proceedings of the 33rd International Conference on Software Engineering. Waikiki. pp. 221–230. <a href="/wiki/Aarti_Gupta_(computer_scientist)" target="_blank">/wiki/Aarti_Gupta_(computer_scientist)</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Binder, Robert V. (1999). Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Longman. ISBN 0-201-80938-9. <a href="0-201-80938-9" target="_blank">0-201-80938-9</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Melo, Silvana Morita; Souza, Simone do Rocio Senger de; Souza, Paulo Sérgio Lopes de; Carver, Jeffrey C. (2017). How to test your concurrent software: an approach for the selection of testing techniques. Conference on Systems, Programming, Languages, and Applications: Software for Humanity - SPLASH. <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">K.C., Tai (20–22 September 1989). Testing of concurrent software. Proceedings of the Thirteenth Annual International Computer Software & Applications Conference. Orlando, FL, USA, USA. pp. 62–64. <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">K.C., Tai (20–22 September 1989). Testing of concurrent software. Proceedings of the Thirteenth Annual International Computer Software & Applications Conference. Orlando, FL, USA, USA. pp. 62–64. <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">Hwang, Gwan-Hwan; Tai, Kuo-Chung; Huang, Ting-Lu (1995). "Reachability Testing: An Approach To Testing Concurrent Software". International Journal of Software Engineering and Knowledge Engineering. 5 (4): 493–510. doi:10.1142/S0218194095000241. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">K.C., Tai (20–22 September 1989). Testing of concurrent software. Proceedings of the Thirteenth Annual International Computer Software & Applications Conference. Orlando, FL, USA, USA. pp. 62–64. <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">Hwang, Gwan-Hwan; Tai, Kuo-Chung; Huang, Ting-Lu (1995). "Reachability Testing: An Approach To Testing Concurrent Software". International Journal of Software Engineering and Knowledge Engineering. 5 (4): 493–510. doi:10.1142/S0218194095000241. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Qi, Xiaofang; Li, Yueran (23–24 November 2018). Parallel Reachability Testing Based on Hadoop MapReduce. th International Conference, SATE 2018. Shenzhen, Guangdong, China. pp. 173–184. doi:10.1007/978-3-030-04272-1_11. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Lu, Shan; Park, Soyeon; Seo, Eunsoo; Zhou, Yuanyuan (1–5 March 2008). Learning from mistakes: a comprehensive study on real world concurrency bug characteristics. ASPLOS XIII Proceedings of the 13th international conference on Architectural support for programming languages and operating systems. Seattle, WA, USA. pp. 329–339. <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Artho, Cyrille; Biere, Armin (27–28 August 2001). Applying static analysis to large-scale, multi-threaded Java programs. Proceedings 2001 Australian Software Engineering Conference. Canberra, ACT, Australia, Australia. pp. 68–75. <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">Manzoor, Numan; Munir, Hussan; Moayyed, Misagh (27–30 November 2012). Comparison of Static Analysis Tools for Finding Concurrency Bugs. 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops. Dallas, TX, USA. pp. 129–133. <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Dustin, Elfriede (28 December 2002). Effective Software Testing: 50 Ways to Improve Your Software Testing. Addison-Wesley Longman. p. 186. ISBN 0201794292. <a href="0201794292" target="_blank">0201794292</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Binder, Robert V. (1999). Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Longman. ISBN 0-201-80938-9. <a href="0-201-80938-9" target="_blank">0-201-80938-9</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
<li id="fn:20">Binder, Robert V. (1999). Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Longman. ISBN 0-201-80938-9. <a href="0-201-80938-9" target="_blank">0-201-80938-9</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></li>
<li id="fn:21">Lu, Shan; Park, Soyeon; Seo, Eunsoo; Zhou, Yuanyuan (1–5 March 2008). Learning from mistakes: a comprehensive study on real world concurrency bug characteristics. ASPLOS XIII Proceedings of the 13th international conference on Architectural support for programming languages and operating systems. Seattle, WA, USA. pp. 329–339. <a href="#fnref:21" class="footnote-back-ref">↩</a></li>
</ol>

Concurrent testing open-in-new

Concurrent testing