Abstract

The web application security scanner is a computer program that assessed web application security with penetration testing technique. The benefit of automated web application penetration testing is huge, which web application security scanner not only reduced the time, cost, and resource required for web application penetration testing but also eliminate test engineer reliance on human knowledge. Nevertheless, web application security scanners are possessing weaknesses of low test coverage, and the scanners are generating inaccurate test results. Consequently, experimentations are frequently held to quantitatively quantify web application security scanner's quality to investigate the web application security scanner's strengths and limitations. However, there is a discovery that neither a standard methodology nor criterion is available for quantifying the web application security scanner's quality. Hence, in this paper systematic review is conducted and analysed the methodology and criterion used for quantifying web application security scanners' quality. In this survey, the experiment methodologies and criterions that had been used to quantify web application security scanner's quality is classified and review using the preferred reporting items for systematic reviews and meta-analyses (PRISMA) protocol. The objectives are to provide practitioners with the understanding of methodologies and criterions that available for measuring web application security scanners’ test coverage, attack coverage, and vulnerability detection rate, while provides the critical hint for development of the next testing framework, model, methodology, or criterions, to measure web application security scanner quality.

Keyword

Web application security scanner, Penetration testing, Quality criteria, PRISMA.

Cite this article

Refference

[1][1]Roche X. Httrack website copier. Citato a. 2012.

[2][2]Hai-Jew S. Conducting surface web-based research with maltego carbon. Retrieve from: http://scalar.usc.edu/works/conducting-surface-web-based-research-with-maltego-carbon/index. Accessed 15 May 2018.

[3][3]https://www.acunetix.com/Websitesecurity/Cros%20s-Site-Scripting. Accessed 15 May 2018.

[4][4]Meucci M, Keary E, Cuthbert D. The OWASP testing guide v2. OWASP Foundation 2008.

[5][5]Jovanovic N, Kruegel C, Kirda E. Pixy: a static analysis tool for detecting web application vulnerabilities (short paper). Symposium on security and privacy 2006 (pp.258-63). IEEE.

[6][6]Suto L. Analyzing the accuracy and time costs of web application security scanners. San Francisco. 2010.

[7][7]Vieira M, Antunes N, Madeira H. Using web security scanners to detect vulnerabilities in web services. In international conference on dependable systems & networks 2009 (pp. 566-71). IEEE.

[8][8]Antunes N, Vieira M. Detecting SQL injection vulnerabilities in web services. In Latin-American symposium on dependable computing 2009 (pp. 17-24). IEEE.

[9][9]Antunes N, Vieira M. Comparing the effectiveness of penetration testing and static code analysis on the detection of SQL injection vulnerabilities in web services. In international symposium on dependable computing 2009 (pp. 301-6). IEEE.

[10][10]Antunes N, Vieira M. Defending against web application vulnerabilities. Computer. 2012; 45(2):66-72.

[11][11]http://projects.webappsec.org/w/page/13246986/Web%20Application%20Security%20Scanner%20Evaluation%20Criteria. Accessed 25 February 2018.

[12][12]Black PE, Fong E, Okun V, Gaucher R. Software assurance tools: web application security scanner functional specification version 1.0. Special Publication, National Institute of Standards and Technology. 2008.

[13][13]Qianqian W, Xiangjun L. Research and design on web application vulnerability scanning service. In international conference on software engineering and service science 2014 (pp. 671-4). IEEE.

[14][14]Baral P. Web application scanners: a review of related articles [Essay]. IEEE Potentials. 2011; 30(2):10-4.

[15][15]Fong E, Okun V. Web application scanners: definitions and functions. In annual Hawaii international conference on system sciences 2007. IEEE.

[16][16]Curphey M, Arawo R. Web application security assessment tools. IEEE Security & Privacy. 2006; 4(4):32-41.

[17][17]Tian-yang G, Yin-Sheng S, You-yuan F. Research on software security testing. World Academy of Science, Engineering and Technology. 2010; 4(9):1446-50.

[18][18]Tung YH, Tseng SS, Shih JF, Shan HL. W-VST: a testbed for evaluating web vulnerability scanner. In international conference on quality software 2014 (pp. 228-33). IEEE.

[19][19]Gol D, Shah N. Detection of web application vulnerability based on RUP model. In national conference on recent advances in electronics & computer engineering 2015 (pp. 96-100). IEEE.

[20][20]Chen JM, Wu CL. An automated vulnerability scanner for injection attack based on injection point. In international computer symposium 2010 (pp. 113-8). IEEE.

[21][21]Alssir FT, Ahmed M. Web security testing approaches: comparison framework. In proceedings of the international congress on computer applications and computational science 2012 (pp. 163-9). Springer, Berlin, Heidelberg.

[22][22]Muñoz FR, Cortes II, Villalba LJ. Enlargement of vulnerable web applications for testing. The Journal of Supercomputing. 2017:1-20.

[23][23]Bazzoli E, Criscione C, Maggi F, Zanero S. XSS peeker: a systematic analysis of cross-site scripting vulnerability scanners. arXiv preprint arXiv:1410.4207. 2014.

[24][24]Patil S, Marathe N, Padiya P. Design of efficient web vulnerability scanner. In international conference on inventive computation technologies 2016 (pp. 1-6). IEEE.

[25][25]Fung AP, Wang T, Cheung KW, Wong TY. Scanning of real-world web applications for parameter tampering vulnerabilities. In proceedings of the ACM symposium on information, computer and communications security 2014 (pp. 341-52). ACM.

[26][26]Khoury N, Zavarsky P, Lindskog D, Ruhl R. Testing and assessing web vulnerability scanners for persistent SQL injection attacks. In proceedings of the first international workshop on security and privacy preserving in e-societies 2011 (pp. 12-8). ACM.

[27][27]Medeiros I, Neves NF, Correia M. Automatic detection and correction of web application vulnerabilities using data mining to predict false positives. In proceedings of the international conference on world wide web 2014 (pp. 63-74). ACM.

[28][28]DURIC Z. WAPTT-Web application penetration testing tool. Advances in Electrical and Computer Engineering. 2014; 14(1):93-102.

[29][29]Balduzzi M, Gimenez CT, Balzarotti D, Kirda E. Automated discovery of parameter pollution vulnerabilities in web applications. In NDSS 2011.

[30][30]Makino Y, Klyuev V. Evaluation of web vulnerability scanners. In international conference on intelligent data acquisition and advanced computing systems: technology and applications 2015 (pp. 399-402). IEEE.

[31][31]Aliero MS, Ghani I. A component based SQL injection vulnerability detection tool. In Malaysian software engineering conference 2015 (pp. 224-9). IEEE.

[32][32]Auronen L. Tool-based approach to assessing web application security. Helsinki University of Technology. 2002 (pp. 1-20).

[33][33]Antunes N, Vieira M. Benchmarking vulnerability detection tools for web services. In international conference on web services 2010 (pp. 203-10). IEEE.

[34][34]Fong E, Gaucher R, Okun V, Black PE, Dalci E. Building a test suite for web application scanners. In proceedings of the Hawaii international conference on system sciences 2008 (pp. 1-8). IEEE.

[35][35]Cardwell K. Building virtual pentesting labs for advanced penetration testing. Packt Publishing Ltd; 2014.

[36][36]Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of Internal Medicine. 2009; 151(4):264-9.

[37][37]Bau J, Bursztein E, Gupta D, Mitchell J. State of the art: automated black-box web application vulnerability testing. In symposium on security and privacy 2010 (pp. 332-45). IEEE.

[38][38]Bau J, Wang F, Bursztein E, Mutchler P, Mitchell JC. Vulnerability factors in new web applications: audit tools, developer selection & languages. Stanford, Tech. Rep. 2012.

[39][39]Shahriar H, Zulkernine M. Automatic testing of program security vulnerabilities. In international conference on computer software and applications 2009 (pp. 550-5). IEEE.

[40][40]Khoury N, Zavarsky P, Lindskog D, Ruhl R. An analysis of black-box web application security scanners against stored SQL injection. In third international conference on privacy, security, risk and trust (PASSAT) and social computing (SocialCom) 2011 (pp. 1095-101). IEEE.

[41][41]Parvez M, Zavarsky P, Khoury N. Analysis of effectiveness of black-box web application scanners in detection of stored SQL injection and stored XSS vulnerabilities. In international conference for internet technology and secured transactions 2015 (pp. 186-91). IEEE.

[42][42]Diaz G, Bermejo JR. Static analysis of source code security: assessment of tools against SAMATE tests. Information and Software Technology. 2013; 55(8):1462-76.

[43][43]Garn B, Kapsalis I, Simos DE, Winkler S. On the applicability of combinatorial testing to web application security testing: a case study. In proceedings of the workshop on joining AcadeMiA and industry contributions to test automation and model-based testing 2014 (pp. 16-21). ACM.

[44][44]Alsaleh M, Alomar N, Alshreef M, Alarifi A, Al-Salman A. Performance-based comparative assessment of open source web vulnerability scanners. Security and Communication Networks. 2017:1-14.

[45][45]Idrissi SE, Berbiche N, Guerouate F, Shibi M. Performance evaluation of web application security scanners for prevention and protection against vulnerabilities. International Journal of Applied Engineering Research. 2017; 12(21):11068-76.

[46][46]Doupe A, Cova M, Vigna G. Why Johnny cant pentest: An analysis of black-box web vulnerability scanners. In international conference on detection of intrusions and malware, and vulnerability assessment 2010 (pp. 111-31). Springer, Berlin, Heidelberg.

[47][47]Huiyao A, Yang S, Tao Y, Hui L, Peng Z, Jun Z. A new architecture of AJAX web application security crawler with finite-state machine. In international conference on cyber-enabled distributed computing and knowledge discovery 2014 (pp. 112-7). IEEE.

[48][48]Jensen T, Pedersen H, Olesen MC, Hansen RR. Thaps: automated vulnerability scanning of PHP applications. In Nordic conference on secure IT systems 2012 (pp. 31-46). Springer, Berlin, Heidelberg.

[49][49]Wang X, Wang L, Wei G, Zhang D, Yang Y. Hidden web crawling for SQL injection detection. In international conference on broadband network and multimedia technology 2010 (pp. 14-8). IEEE.

[50][50]Tripp O, Weisman O, Guy L. Finding your way in the testing jungle: a learning approach to web security testing. In proceedings of the international symposium on software testing and analysis 2013 (pp. 347-57). ACM.

[51][51]Li N, Xie T, Jin M, Liu C. Perturbation-based user-input-validation testing of web applications. Journal of Systems and Software. 2010; 83(11):2263-74.

[52][52]Kosuga Y, Kono K, Hanaoka M, Hishiyama M, Takahama Y. Sania: syntactic and semantic analysis for automated testing against SQL injection. In computer security applications conference 2007 (pp. 107-17). IEEE.

[53][53]Duchene F, Rawat S, Richier JL, Groz R. LigRE: reverse-engineering of control and data flow models for black-box XSS detection. In working conference on reverse engineering 2013 (pp. 252-61). IEEE.

[54][54]Rocha TS, Souto E. ETSSDetector: a tool to automatically detect cross-site scripting vulnerabilities. In international symposium on network computing and applications 2014 (pp. 306-9). IEEE.

[55][55]Dao TB, Shibayama E. Idea: automatic security testing for web applications. In international symposium on engineering secure software and systems 2009 (pp. 180-4). Springer, Berlin, Heidelberg.

[56][56]Avancini A, Ceccato M. Comparison and integration of genetic algorithms and dynamic symbolic execution for security testing of cross-site scripting vulnerabilities. Information and Software Technology. 2013; 55(12):2209-22.

[57][57]Palsetia N, Deepa G, Khan FA, Thilagam PS, Pais AR. Securing native XML database-driven web applications from XQuery injection vulnerabilities. Journal of Systems and Software. 2016; 122:93-109.

[58][58]Thome J, Gorla A, Zeller A. Search-based security testing of web applications. In proceedings of the international workshop on search-based software testing 2014 (pp. 5-14). ACM.

[59][59]Duchene F, Rawat S, Richier JL, Groz R. KameleonFuzz: evolutionary fuzzing for black-box XSS detection. In proceedings of the conference on data and application security and privacy 2014 (pp. 37-48). ACM.

[60][60]Deepa G, Thilagam PS, Khan FA, Praseed A, Pais AR, Palsetia N. Black-box detection of XQuery injection and parameter tampering vulnerabilities in web applications. International Journal of Information Security. 2018:17(1):105-20.

[61][61]Antunes N, Laranjeiro N, Vieira M, Madeira H. Effective detection of SQL/XPath injection vulnerabilities in web services. In international conference on services computing 2009 (pp. 260-7). IEEE.

[62][62]Antunes N, Vieira M. Enhancing penetration testing with attack signatures and interface monitoring for the detection of injection vulnerabilities in web services. In international conference on services computing 2011 (pp. 104-11). IEEE.

[63][63]Antunes N, Vieira M. Penetration testing for web services. Computer. 2014; 47(2):30-6.

[64][64]Antunes N, Vieira M. Designing vulnerability testing tools for web services: approach, components, and tools. International Journal of Information Security. 2017; 16(4):435-57.

[65][65]Su Z, Wassermann G. The essence of command injection attacks in web applications. In SIGPLAN notices 2006 (pp. 372-82). ACM.

[66][66]Dessiatnikoff A, Akrout R, Alata E, Kaâniche M, Nicomette V. A clustering approach for web vulnerabilities detection. In Pacific Rim international symposium on dependable computing 2011 (pp. 194-203). IEEE Computer Society.

[67][67]Lounis O, Guermeche SE, Saoudi L, Benaicha SE. A new algorithm for detecting SQL injection attack in web application. In science and information conference (SAI) 2014 (pp.43-51).

[68][68]Akrout R, Alata E, Kaaniche M, Nicomette V. An automated black box approach for web vulnerability identification and attack scenario generation. Journal of the Brazilian Computer Society. 2014; 20(4):1-16.

[69][69]Nanda S, Lam LC, Chiueh TC. Dynamic multi-process information flow tracking for web application security. In proceedings of the international conference on Middleware companion 2007. ACM.

[70][70]Wei K, Muthuprasanna M, Kothari S. Preventing SQL injection attacks in stored procedures. In software engineering conference 2006. IEEE.

[71][71]Antunes N, Vieira M. SOA-scanner: an integrated tool to detect vulnerabilities in service-based infrastructures. In international conference on services computing 2013 (pp. 280-7). IEEE.

[72][72]Djuric Z. A black-box testing tool for detecting SQL injection vulnerabilities. In international conference on informatics and applications 2013 (pp. 216-21). IEEE.

[73][73]Singh AK, Roy S. A network based vulnerability scanner for detecting SQLI attacks in web applications. In international conference on recent advances in information technology 2012 (pp. 585-90). IEEE.

[74][74]Vernotte A, Dadeau F, Lebeau F, Legeard B, Peureux F, Piat F. Efficient detection of multi-step cross-site scripting vulnerabilities. In international conference on information systems security 2014 (pp. 358-77). Springer, Cham.

[75][75]Saleh AZ, Rozali NA, Buja AG, Jalil KA, Ali FH, Rahman TF. A method for web application vulnerabilities detection by using boyer-moore string matching algorithm. Procedia Computer Science. 2015; 72:112-21.

[76][76]Lee M, Lee Y, Yoon H. An enhanced rule-based web scanner based on similarity score. Advances in Electrical and Computer Engineering. 2016; 16(3):9-14.

[77][77]Liu L, Su G, Xu J, Zhang B, Kang J, Xu S, et a. An inferential metamorphic testing approach to reduce false positives in SQLIV penetration test. In computer software and applications conference 2017 (pp. 675-80). IEEE.

[78][78]Backes M, Rieck K, Skoruppa M, Stock B, Yamaguchi F. Efficient and flexible discovery of PHP application vulnerabilities. In European symposium on security and privacy 2017 (pp. 334-49). IEEE.

[79][79]De Meo F, Rocchetto M, Vigano L. Formal analysis of vulnerabilities of web applications based on SQL injection. In international workshop on security and trust management 2016 (pp. 179-95). Springer, Cham.

[80][80]Huang YW, Yu F, Hang C, Tsai CH, Lee DT, Kuo SY. Securing web application code by static analysis and runtime protection. In proceedings of the international conference on world wide web 2004 (pp. 40-52). ACM.

[81][81]Huang YW, Tsai CH, Lee DT, Kuo SY. Non-detrimental web application security scanning. In international symposium on software reliability engineering 2004 (pp. 219-30). IEEE.

[82][82]Huang YW, Huang SK, Lin TP, Tsai CH. Web application security assessment by fault injection and behavior monitoring. In proceedings of the international conference on world wide web 2003 (pp. 148-59). ACM.

[83][83]Huang YW, Tsai CH, Lin TP, Huang SK, Lee DT, Kuo SY. A testing framework for web application security assessment. Computer Networks. 2005; 48(5):739-61.

[84][84]Huang YW, Lee DT. Web application security-past, present, and future. In computer security in the 21st century 2005 (pp. 183-227). Springer, Boston, MA.

[85][85]Kals S, Kirda E, Kruegel C, Jovanovic N. Secubat: a web vulnerability scanner. In proceedings of the international conference on world wide web 2006 (pp. 247-56). ACM.

[86][86]Viega J, Bloch JT, Kohno Y, McGraw G. ITS4: a static vulnerability scanner for C and C++ code. In annual conference on computer security applications 2000 (pp. 257-67). IEEE.

[87][87]Balzarotti D, Cova M, Felmetsger V, Jovanovic N, Kirda E, Kruegel C, et al. Saner: composing static and dynamic analysis to validate sanitization in web applications. In symposium on security and privacy 2008 (pp. 387-401). IEEE.

[88][88]Tripp O, Pistoia M, Cousot P, Cousot R, Guarnieri S. Andromeda: accurate and scalable security analysis of web applications. In international conference on fundamental approaches to software engineering 2013 (pp. 210-25). Springer, Berlin, Heidelberg.

[89][89]Galan E, Alcaide A, Orfila A, Blasco J. A multi-agent scanner to detect stored-XSS vulnerabilities. Internet Technology and Secured Transactions 2010 (pp.332-7).

[90][90]Suto L. Analyzing the effectiveness and coverage of web application security scanners. San Francisco. 2007.

[91][91]Razzaq A, Latif K, Ahmad HF, Hur A, Anwar Z, Bloodsworth PC. Semantic security against web application attacks. Information Sciences. 2014; 254:19-38.

[92][92]Mainka C, Somorovsky J, Schwenk J. Penetration testing tool for web services security. In world congress on services 2012 (pp. 163-70). IEEE.

[93][93]Balduzzi M, Egele M, Kirda E, Balzarotti D, Kruegel C. A solution for the automated detection of clickjacking attacks. In proceedings of the symposium on information, computer and communications security 2010 (pp. 135-44). ACM.

[94][94]Huyam AA, El-Qawasmeh E. Discovering security vulnerabilities and leaks in ASP. NET websites. In international conference on cyber security, cyber warfare and digital forensic 2012 (pp. 329-33). IEEE.

[95][95]Eshete B, Villafiorita A, Weldemariam K, Zulkernine M. Confeagle: automated analysis of configuration vulnerabilities in web applications. In international conference on software security and reliability 2013 (pp. 188-97). IEEE.

[96][96]Vithanage NM, Jeyamohan N. WebGuardia-an integrated penetration testing system to detect web application vulnerabilities. In international conference on wireless communications, signal processing and networking 2016 (pp. 221-7). IEEE.

[97][97]Laranjeiro N, Vieira M, Madeira H. Protecting database centric web services against SQL/XPath injection attacks. In international conference on database and expert systems applications 2009 (pp. 271-8). Springer, Berlin, Heidelberg.

[98][98]McAllister S, Kirda E, Kruegel C. Leveraging user interactions for in-depth testing of web applications. In international workshop on recent advances in intrusion detection 2008 (pp. 191-210). Springer, Berlin, Heidelberg.

[99][99]Doupe A, Cavedon L, Kruegel C, Vigna G. Enemy of the state: a state-aware black-box web vulnerability scanner. In USENIX security symposium 2012.

[100][100]Pellegrino G, Tschurtz C, Bodden E, Rossow C. JAk: using dynamic analysis to crawl and test modern web applications. In international workshop on recent advances in intrusion detection 2015 (pp. 295-316). Springer, Cham.

[101][101]Fonseca J, Vieira M, Madeira H. Vulnerability & attack injection for web applications. In international conference on dependable systems & networks 2009 (pp. 93-102). IEEE.

[102][102]Fonseca J, Vieira M, Madeira H. Evaluation of web security mechanisms using vulnerability and attack injection. IEEE Transactions on Dependable and Secure Computing. 2014; 11(5):440-53.

[103][103]Fonseca J, Matarese F. Using vulnerability injection to improve web security. In innovative technologies for dependable OTS-based critical systems 2013 (pp. 145-57). Springer, Milano.

[104][104]Tung YH, Tseng SS, Shih JF, Shan HL. A cost-effective approach to evaluating security vulnerability scanner. In network operations and management symposium 2013 (pp. 1-3). IEEE.

[105][105]Dao TB, Shibayama E. Security sensitive data flow coverage criterion for automatic security testing of web applications. In international symposium on engineering secure software and systems 2011 (pp. 101-13). Springer, Berlin, Heidelberg.

[106][106]Loh PK, Subramanian D. Fuzzy classification metrics for scanner assessment and vulnerability reporting. IEEE Transactions on Information Forensics and Security. 2010; 5(4):613-24.

[107][107]OWASP T. Application Security Risks 2017.

[108][108]Deepa G, Thilagam PS, Praseed A, Pais AR. DetLogic: a black-box approach for detecting logic vulnerabilities in web applications. Journal of Network and Computer Applications. 2018; 109:89-109.