所需积分/C币:27 2013-11-19 10:25:13 197KB PDF

rapid_bottleneck_identifation,you should read it
UNDERSTANDING BOTTLENECKS. THROUGHPUT AND CONCURRENCY Before delving into the specifics of the RBI methodology, we must first establish a common understanding of bottlenecks--and where they are found-as well as draw a distinction between throughput and concurrency testing Bottlenecks-Key Performance Inhibitors Any system resource--such as hardware, software, or bandwidth--that places defining limits on data flow or processing speed creates a bottleneck. In Web applications, bottlenecks directly affect performance and scalability by limiting the amount of data throughput or restricting the number of application connections These problems occur at all levels of the system architecture, including the network layer, the Web server, the application server, and the database server. Historically, based on our experience testing actual customer applications, bottlenecks have been distributed across these components as shown in figure 1. web Server(s) 10% In Web applications, bottlenecks directly Network affect performance and scalability by limiting the amount of data throughput or restricting the number of appllcatlon connections Applieation Dab通 Servers 20% Figure 1. Estimated distribution of bottlenecks across the system infrastructure The Compounding Impact of Testing Complexity The testing approach you choose directly impacts the difficulty of isolating and resolving bottlenecks. Unfortunately, too many testing procedures begin with complex usage scenarios where testers try to simulate exactly how the application will be utilized in production. This may involve running several different transactions to simulate different types of users who interact with the application n different ways. Unfortunately, this creates a significant testing roadblock because scenarios that arc highcr in complexity and involve multiplc diffcrent transactions introduce more bottlenecks into the test which makes it difficult to identify root causes For example, the graph in Figure 2 illustrates the test results of a standard c-commcrcc application that bottlcnccked at approximately 2,000 concurrent users.In this sample test, the usage scenarios involved browsing, searching, and adding items to a shopping cart to complete a purchase. Although there were only Rapid bottleneck Identification a Better Way to do load testing page 3 three transactions being tested, each transaction interacted with all levels of the application architecture and any one of them could have caused the bottleneck To further complicate matters, the bottleneck could also have been caused by a system issue. Ultimately, the more variables involved in a test, the more difficult it to determine thc causc of the problcm 性 作图 Figure 2. Graphical results of pplication test. If the problem can be in any tier of the architecture, and the likelihood of it being in any one tier is not substantially greater than in any other, where else can you look fo Two Primary Issues: Throughput and Concurrency Throughput is the amount of data flow a system can support, measured in hits per A majority of all system and application second, pages per second, and megabits of data per second. Concurrency is the performance issues result from limitations in throughput number of independent users simultaneously connected and using an application. In our experience, a majority of all system and application performance issues result from limitations in throughput. However, concurrency issues are also critical to application performance and can be even morc difficult to isolate Testing for Throughpu Testing for throughput involves minimizing the number of user connections to a system and maximizing the amount of work being done by those users. This pushes the system and application to capacity so that all issues will be revealed For throughput testing at the system level, basic files can be added to the Web and application servers for testing purposes. The load test can then be set up to request these test files to assess maximum system throughput at each tier picallv testers use a large image file for bandwidth tests, a small text file or image for hit Rapid bottleneck Identification a Better Way to do load testing Page 4 rate tests, and a very simple application page-a"Hello World?"page, for instance for page rate testing. If the system does not meet basic application performance requirements-juist requesting these simple test pages--testing should cease until the system itself has been improved, either through tuning the settings, incrcasing the hardwarc capacity, or incrcasing the allocated bandwidth Throughput testing of the actual application then involves hitting key pages and user transactions in the application itself with limited delay between requests to find the page-per-second capacity limit of the various functional components Obviously the pages or transactions with the poorest page throughput need the nost tuning. Testing for Concurrency On the system and application levels, concurrency is limited by sessions and socket connections. Code flaws and incorrect server contiguration settings can also limit concurrency. Concurrency tests involve ramping up a number of users on the system and using realistic page-delay times at a ramp-up speed slow enough to gather useful data throughout the testing at each level of load. As with throughput testing, it is important to test the key pages and user transactions in the application under test The Difference Between Throughput and Concurrency Tests The load generated from a 100 virtual user load test with 1-second think times is not equivalent to a 1,000 virtual user load test with 10-second think times. As Figure 3 illustrates, the two tests are identical in terms of throughput; however, in Although most bottlenecks are found in terms of concurrency they are vastly different throughput testing, to properly test an application you must test for bot Bottleneck Scenario Throughput Concurrency throughput and concurrency Point 100 Users Pages/Second 1 Second/ page (100vU x 1 Page/VU: 1 Second =100 Connections Pages/ Second 1. 000 Users Pages/ second 1,000 10 Seconds Page(1000VU x 1 Page/VU)= 10 Seconds=100 onnections ages/Second Figure 3. 100 virtual user load test versus 1,000 virtual user load test In the first scenario, the throughput test, the application bottlenecked at 50 pages per second. In the second scenario, however, a concurrency test of the same Is, the application bottle Pages pe d. The only y differences between these two tests were the number of users on the system and the length of time those users stayed on the pages. In the throughput test with fewer users and shorter page view delays, the application had more throughput capacity; the second test shows the application was limited in its concurrency. If Rapid bottleneck Identification a Better Way to do load testing page 5 the testers had checked only for throughput, the concurrency issue would not have been discovered until the application was in production Figure 4 and Figure 5 on the following page show the results of each test and highlight the importance of testing for both throughput and concurrency The only differences between these two tests were the number of users on the system and the length of time those users stayed on the pages Figure 4. Throughput tests for 100 users with 1-second page views show a bottleneck at 50 pages per second Figure 5. Concurrency tests for 1,000 users with 10-second page views show a bottleneck at 25 pages per second Rapid bottleneck Identification a Better Way to do load testing page 6 THE RBI TESTING APPROACH Traditionally, performance testers focused on concurrent users as the key metric for application scalability. However, if a majority of application and system-level issues are found in throughput tests, a new approach is needed These three principles form the foundation of the rBi methodolog All Web applications have bottlenecks These bottlenecks can only be uncovered one at a time Focus should be placed where the bottlenecks are most likely to occur Although recognizing the importance of concurrency testing the rbl methodology first focuses on throughput testing to root out the most common bottlenecks, followed by concurrency testing to assess performance under load conditions that reflect the actual number of users expected on the application. RBI testing also starts with the simple tests and then builds in complexity so that when an issue appears, all the other possible causes have been ruled out. Focusing on throughput testing, followed by concurrency testing, and using a structured approach to the test process ensures that bottlenecks are quickly isolated, which improves cfficicncy and reduces cost Benefits of RBI Testing The rBi methodology enables rapid yet thorough testing that systematically uncovers all system and application issues-both simple and complex Reduce Testing Time How much time can you save by focusing initially on throughput testing? Take an example of a system expected to handle 5,000 concurrent users, with users spending an avcragc of 45 seconds on cach pagc. If the application has a bottleneck that will limit its scalability to 25 pages per second, a concurrency test will find this bottleneck at approximately 1, 125 users(25 pages per second at 45 seconds per page In the interest of not biasing the data, a typical concurrency test ramp up should proceed slowly. For example, you may consider ramping one user every five By uncovering bottlenecks using a tiered seconds. In this example, the bottleneck would have occurred 5, 625 seconds or 94 approach, you can qulckly ldentify issues minutes into the test( 1, 125 uscrs at 5 seconds pcr uscr). Howcvcr, to validate the as well as isolate issues in components of bottleneck, the test would have to continue beyond that point to prove that the which you have limited knowledge throughput was not climbing as users were added. A throughput test could have found this problem in less than 60 seconds Rapid bottleneck Identification a Better Way to do load testing Page 7 Eliminate Initial Testing Complexity Very often performance testing begins with overly complex scenarios exercising too many components at the same time, making it easy for bottlenecks to hide The rBi methodology begins with system-level testing that can be carried out before the application is even deployed Improve QA Efficiency The rbi methodology tests the simplest test cases first and then moves on to those with increased complexity. If the simplest test case works and the next level of complexity fails the bottleneck lies in the newly added complexity By uncovering bottlenecks using a tiered approach, you can quickly identify issues as well as isolate issues in componcnts of which you have limited knowledge Enhance Testing Effectiveness with Knowledge Aggregation The modular and iterative nature of the methodology means that when a bottleneck appears, all the previously tested components have already been ruled out. For instance, if hitting the home page shows no bottlenecks but hitting the home page plus executing a search shows very poor performance, the cause of the bottleneck lies in the search functionality. RBI Testing for Common System Bottlenecks Any performance testing should begin with an assessment of the basic network infrastructure supporting the application. If this basic system cannot support the anticipated user load on a system, even infinitely scalable application code will bottleneck. Basic system-level tests should be run to validate bandwidth, hit rate, and connections. Additionally, simple test application pages should be exercised- le“hell for instal The Applic After validating that the system infrastructure meets the most basic needs of the end users, turn to the application itself. Once again, start with the simplest possible test casc If testing has progressed this far without uncovering system level issues(or those issues have been resolved), any remaining problems are caused by the application itself. For example, if a test application page achieved 100 pages per second and thc homc page bottlenecks at only 10 pages pcr sccond, the problcm lics in the overhead required to display the home page Rapid bottleneck Identification a Better Way to do load testing page 8 At this point, the test application page test provides two valuable pieces of nformation First, because we know that the system itself is not the bottleneck, the The difference between the performance of culprit can only be the code on the home page. Second, we can see how much a test page and the performance of an tuning the home page could improve performance. The difference between the actual production application page performance of the test application page (100 pages pcr sccond) and the home determines the potential gain in page (10 pages per second) determines the maximum performance improvement performance improvement from tuning tuning could provide. Likewise, multipage transactions can be assessed by breaking down the performance of individual pages in the transaction and evaluating how each contributes to the performance of the overall transaction Since any real-world application page likely requires more processing power than a hello world"test page, it is reasonable to expect some drop-off in performance However, the grcatcr that drop-off, the grcatcr the nccd for-and potential gain trom--tuning. It is also important to note that if the drop-off between the test page and an actual application page is not substantial and the performance is still insufficient to meet the needs of the application user base, you need to add more hardware capacity Up to this point no mention has been made of page-response times. Although response times are a key metric of overall performance, response times will be the same for one user as they are for 1,000 or 100,000 users unless a bottleneck is encountered. So in this methodology, response times are only useful as an indicator that a bottleneck has been reached (if response times begin to spike) or as failure criteria(if response times exceed some predefined threshold), with poor performing pages(those that experience errors or high response times)most in need of code optimization RBI Testing for Application Bottlenecks As with the system-level testing, the rbi methodology begins application testing with the simplest possible test case and then builds in complexity. In a typical e-commerce application, you would test the home page first and then add pages and busincss functions until complctc, rcal-world transactions arc bcing tested, first degradation in response times or page throughput will have been caused by th 2. individually and then in complex scenario usage patterns. As steps are added, any newly added step, making it easier to isolate what code needs to be investigated Once cach of the business functions and transactions has bccn tcstcd and optimized(as necessary), the transactions can be combined into complete scenario Proper concurrency tests scenarios must concurrency tests. These concurrency tests must focus on two key components accurately reflect what real users do on the First, the concurrency test must accurately reflect what real users do on the site- site and replicate those actions at the same browse, search, register, login, and purchase. Second, the steps in those user pace transactions must be performed at the same pace as real-world visitors with appropriate "think times"between each step. This data can be gathered with a Web logging tool that exposes session length, pages viewed per session (to determine the user pacing), and percentages of pages hit(to determine the actual business functions used) Rapid bottleneck Identification a Better Way to do load testing page 9 Once the test has been designed from real-world data--or educated assumptions for an application not yet deployed the test must be executed in a way that gathers valuable information at various user load levels. If the site is expected to handle 1,000 concurrent users, then it's important not to start those users all at oncc. Instcad ramp your test slowly, adding onc or more users at defincd timc intervals, until you reach 1000. This will allow you to determine overal timc performance at each level of user load and also make it easier to identify performance problems when they begin to occur. CONCLUSION The rbI methodology for load testing improves testing efficiency by focusing first on Where bottlenecks most often occur--in the throughput. Once throughput has ssess performance under realistic user loads. By following a structured approac o been thoroughly tested, you can test the system and application for concurrency to from system testing to application testing and slowly, systematically introducing complexity into the test cases, you can quickly isolate bottlenecks and their associated root cause Although this paper focuses on methodology it is important to point out that much of this process can and should be automated using an automated testing tool. Oracle Application Testing Suite is the centerpiece of Oracle enterprise Manager for functional and load testing for Web and service-oriented architecture applications CONTACT US For more information about Oracle Application Testing Suite and Oracle Enterprise Manager please visit oracle. com or call +1 800. ORACLEI to speak to an Oracle representative. Also, please visit Oracle Technology Network at oracle. com/technology/products/oem, Rapid bottleneck identification-a Better Way to do load testing

试读 11P rapid_bottleneck_identifation
  • 分享精英


关注 私信 TA的资源

    rapid_bottleneck_identifation 27积分/C币 立即下载


    27积分/C币 立即下载 >