Testing Experience Magazine Vol.12 Dec 2010

ISSN 1866-5705 www.testingexperience.com free digital version print version 8,00 € printed in Germany 12 The Magazine for Professional Testers Open Source Tools December 2010© diego cervo - Fotolia.com

Response Time Accuracy: open-source or Proprietary Performance Test Tool by Muhammad Dhiauddin Mohamed Suffian, Vivek Kumar & Redzuan AbdullahPerformance testing is a critical test approach in achieving the it is important to demonstrate that there are differences in re-full coverage of software product testing, apart from functiona- sponse time between various performance testing tools and tolity and other non-functional aspects. Performance testing will understand the tools’ behaviors including their influence on thedetermine that the software under test is capable of performing machine resource utilization. We started off by planning the cor-within an acceptable time according to the end-users’ expecta- rect test setup for the study.tions. Our experiences in performance testing, particularly in theload testing of various types of software, ranging from mobile The test setup involved two identical machines that serve as cli-applications, desktop, web-based to cloud-based applications ent with performance testing tools installed and a server whichand using different performance testing tools, have triggered us hosts the static web pages. These machines are connected via PC-to ask which tool actually gives the most accurate results. The to-PC LAN cable (commonly referred as cross cable). The reasonanswer is not simple, and it is not that the proprietary tool always for connecting it via cross cable is to minimize network factorsgives the right and accurate response times compared to open- from influencing the performance test results. The diagram forsource tools, or vice-versa. Thus, before answering the question, the setup is as follows: Figure 1: Test setup for comparison of performance testing toolsAs shown in the diagram, only three performance testing tools cookies. Running processes on this machine were kept to a mi-were selected and installed on the client machine. For the scope nimum, i.e. unnecessary processes and server-side logging wereof the study, the tools are only referred as Tool A, Tool B and Tool disabled. Load testing was performed from 1 user access and in-C. While the test was running for a particular tool, the services creased to 100, 200, 300, 400 and 500 concurrent users with zerofor the other remaining tools were stopped or disabled. As for ramped-up time. The most important thing for this experiment isthe server, it only hosted static HTML pages with no cache and to reboot both the client and server machines for each test.58 The Magazine for Professional Testers www.testingexperience.com

Server machineFrom the technical perspective, the hardware specification forboth machines was as follows: Operating system: Windows Server 2003 Enterprise Edition SP1 CPU/processor: Intel Pentium D 3.4 GHz Java JDK: JDK 1.6.0 update 21 RAM/memory: 2 GB Web server: Internet Information Services 6 Hard disk storage: 80 GB HTML page size: 65.8 KB (page: 7 KB; image 1: 25.2 KB; imageThe configuration of the software specification was as follows: 2: 33.6 KB) Client machine The test was started with Tool A and with one simulating user ac- cessing the HTML page at the server machine using cross cable. At Operating system: Windows XP SP2 the same time, we recorded CPU, memory and disk I/O utilization Java JDK: JDK 1.6.0 update 21 for both client and server machines. The same approach was done Tool: Tool A (open-source); Tool B (open-source); Tool C (pro- three times per virtual user set until the number reached 500 vir- prietary) tual users. The same kinds of load test were performed against Tool B and Tool C. The metrics collected from this exercise were response time, average CPU utilization, memory utilization and disk I/O utilization. The details of the results obtained from this activity are listed below: a) Tool A No of Con- Response Time Resource Utilization (Client) Resource Utilization (Server) current Users (seconds) CPU Memory Disk I/O CPU Memory Disk I/O 1 0.103 0% 3% 22% 21% 22% 0% 1 0.100 0% 3% 22% 23% 22% 0% 1 0.103 0% 3% 22% 20% 22% 0% 100 0.258 6% 3% 22% 65% 23% 0% 100 0.29 5% 3% 22% 65% 23% 0% 100 0.231 1% 3% 22% 67% 22% 0% 200 0.198 7% 3% 22% 71% 23% 1% 200 0.231 7% 3% 22% 83% 23% 2% 200 0.538 4% 3% 22% 93% 23% 0% 300 0.337 3% 3% 22% 94% 23% 0% 300 0.265 10% 3% 22% 95% 23% 0% 300 0.229 3% 4% 22% 97% 23% 3% 400 0.169 7% 4% 22% 93% 23% 0% 400 0.534 8% 3% 22% 92% 23% 3% 400 0.278 6% 3% 22% 95% 23% 2% 500 0.394 14% 4% 22% 96% 22% 8% 500 0.35 14% 4% 22% 97% 22% 9% 500 0.306 14% 4% 22% 95% 23% 10% b) Tool B No of Con- Response Time Resource Utilization (Client) Resource Utilization (Server) current Users (seconds) CPU Memory Disk I/O CPU Memory Disk I/O 1 0.015 0% 4% 22% 17% 7% 2% 1 0.015 0% 4% 22% 17% 7% 2% 1 0.015 0% 4% 22% 17% 7% 2% 100 1.423 1% 4% 22% 99% 6% 12% 100 1.211 3% 4% 22% 99% 6% 7% 100 1.403 1% 4% 22% 99% 6% 48% 200 3.489 3% 4% 22% 99% 6% 55% 200 4.478 4% 4% 22% 99% 7% 53% 200 5.123 4% 4% 22% 98% 6% 31%www.testingexperience.com The Magazine for Professional Testers 59

300 6.158 7% 4% 22% 99% 6% 14% 300 7.068 5% 4% 22% 99% 7% 33% 300 5.394 2% 4% 22% 99% 6% 69% 400 8.597 3% 4% 22% 99% 6% 32% 400 8.164 10% 4% 22% 97% 6% 32% 400 8.757 3% 4% 22% 98% 6% 36% 500 11.316 5% 4% 22% 98% 6% 47% 500 11.17 5% 4% 22% 98% 7% 27% 500 8.901 8% 4% 22% 97% 6% 28% c) Tool C No of Con- Response Time Resource Utilization (Client) Resource Utilization (Server) current Users (seconds) CPU Memory Disk I/O CPU Memory Disk I/O 1 0.047 0% 3% 22% 35% 19% 3% 1 0.078 0% 3% 22% 38% 19% 3% 1 0.047 0% 3% 22% 25% 20% 3% 100 1.487 3% 3% 22% 100% 19% 59% 100 2.174 3% 3% 22% 100% 19% 14% 100 1.52 1% 3% 22% 100% 19% 22% 200 3.007 3% 3% 22% 100% 19% 27% 200 3.614 2% 3% 22% 100% 18% 15% 200 4.021 5% 3% 22% 100% 19% 38% 300 5.997 2% 3% 22% 100% 18% 22% 300 5.947 3% 3% 22% 100% 17% 10% 300 4.979 3% 3% 22% 100% 17% 18% 400 6.272 11% 3% 22% 100% 22% 15% 400 7.042 2% 3% 22% 100% 17% 6% 400 7.332 3% 3% 22% 100% 17% 27% 500 7.771 4% 4% 22% 88% 18% 7% 500 7.106 7% 4% 22% 100% 18% 24% 500 7.604 7% 4% 22% 100% 16% 10%Based on the metrics collected from the three tools, we plotted times under different user loads for three rounds of performancethe graphs to compare and understand the trends of the re- testing. The graphs representing the results detailed in the previ-sponse times. The graphs focus only on comparing the response ous tables are shown below:Figure 2: Performance (Load) Test – Round 1 Figure 3: Performance (Load) Test – Round 2 Figure 4: Performance (Load) Test – Round 360 The Magazine for Professional Testers www.testingexperience.com

From the graphs plotted, we could observe the differences in re- ed and proved that differences do exist. In our next research ef-sponse time between the three different tools after running a per- fort we will try to answer why this is the case. Some fundamentalformance (load) test for the same HTML web page. This result has understandings that are already in place concern the languageachieved our research objective, which was to prove that, when with which the tools have been developed, the architecture of theusing different performance test tools, the response times we get respective tools, the recording mechanism for each tool, captur-are not similar or even close to each other, especially as the num- ing and simulating the load used for the performance test, andber of concurrent users increases. As mentioned previously, Tool the method of calculating metrics gathered by each tool. HavingA and Tool B are open-source tools while Tool C is a proprietary this research in place will allow further understanding of the be-tool, which leads us to another research question: “Why do differ- havior of various performance tools and could help in selectingent performance testing tools give different response times when the most suitable tool for different contexts of testing.testing the same web page?”. In this article, we have demonstrat-BiographyDhiauddin is currently a Senior Engi- Vivek Kumar works as a Senior Test Redzuan is currently the Director of theneer and Test Team Lead for the test Engineer at one of the most reputable test department in one of the leadingdepartment in one of the leading R&D R&D agencies in Malaysia. He has more R&D agencies in Malaysia. He has moreagencies in Malaysia. He has almost than 7 years of experience in Software than 14 years of working experience6 years of experience in the software/ QA both in manual testing and in test in software development, technicalsystem development and software automation. During his QA/testing support and software testing in bothtesting / quality assurance fields. With career he has been working in various local and multi-national organizations..a working experience in automotive, domains, such as mobile telecommuni- Redzuan has vast knowledge in variousbanking and research & development cations, banking, agriculture, semantic programming languages, CMMI, Sixcompanies, he obtained his technical technology, and ERP systems. He holds Sigma, telecommunication and projectand management skills from various an Engineering degree in Electronics management with international certifi-types of project. He holds an M.Sc. & Instrumentation from the Dr. B. R. cations in TOGAF, ASP, Red Hat Certifiedin Real Time Software Engineering Ambedkar University, Agra (Formerly Engineer (RHCE), British Standard Infor-from the Centre for Advanced Soft- Agra University, Agra). He is a Certified mation Security Auditor (BS7799) andware Engineering (CASE®), Universiti Tester- Foundation Level (CTFL) from Six Sigma Green Belt among others.Teknologi Malaysia. He is a Certified the International Software TestingTester - Foundation Level (CTFL) and Qualification Board (ISTQB®). He hasCertified Tester Advanced Level – Test undertaken and managed variousManager (CTAL-TM) of the International projects from different domains andSoftware Testing Qualification Board been involved in the complete project(ISTQB®). He also has vast knowledge in life cycle starting from project kick-offCMMI®, Test Process and Methodolo- to release to the customer. Vivek has agies as well as Software Development sound knowledge in CMMI, STLC, SDLC,Life Cycle (SDLC). He has just completed automation testing and automationhis Six Sigma Green Belt project on the framework (keyword driven and datatest defect prediction model. He has driven) with Selenium, HP Quick Testapplied and managed various testing Professional 10 and Rational Functionalstrategies in his projects, including Tester. His tireless dedication to thefunctional, performance, security, usa- advancement of software testing inbility and compatibility testing both general and to performance testingfor system test and system integration in particular is often referred to as atest. His interests are in software engi- hobby and not only a job due to theneering and software testing, particu- enjoyment he gets from his efforts.larly in performance testing and testmanagement.www.testingexperience.com The Magazine for Professional Testers 61

Testing Experience Magazine Vol.12 Dec 2010

by Open University Malaysia/Universiti Teknologi Malaysia on Apr 22, 2011

Statistics

Views

Actions

1 Embed 4

Accessibility

Categories

Upload Details

Usage Rights

Report content

Testing Experience Magazine Vol.12 Dec 2010 Document Transcript