Storm Forecast
Blog on APIs, Load Testing, Performance and StormForger.
https://stormforger.com/blog
2020-07-29T23:00:00+00:00
Various authors mainly by StormForger GmbH, https://stormforge.io
Open? Or Closed? On Workload Models for Performance Testing
https://stormforger.com/blog/open-closed-workloads/
2020-07-29T23:00:00+00:00
2020-07-29T23:00:00+00:00
Sebastian Cohnen
<p>There are many differentiating properties when it comes to load and performance testing tools. The <em>general workload model</em> is one aspect that is often overlooked. However, tests that run using the wrong workload model can vastly underestimate latencies and provide a false sense of security. In the following we explain what workload models are and we shed some light on <em>why</em> we believe that StormForger is using the “right” approach for many, if not most cases.</p>
<p></p>
<p>There are many differentiating properties when it comes to load and performance testing tools. The <em>general workload model</em> is one aspect that is often overlooked. However, tests that run using the wrong workload model can vastly underestimate latencies and provide a false sense of security. In the following we explain what workload models are and we shed some light on <em>why</em> we believe that StormForger is using the “right” approach for many, if not most cases.</p>
<p></p>
<p>Workload models are a topic that we talk about a lot when giving presentations, making product demos or onboarding our customers. We think it is essential to have a basic understanding about this for having good and realistic performance tests.</p>
<p>Workload models at first sight are a boring, apparently "theoretical" topic. As with many theoretical topics it turns out that workload models have a <strong>very substantial impact</strong> in practice. Workload models greatly influence <em>what</em> you are actually testing and are probably way more relevant than one might think. It also explains why you get vastly different results with different performance testing tools (because they implement different models). I'd also argue that this fundamental principle is often simply overlooked when choosing a tool to run any kind of performance test.</p>
<p><strong>Workload</strong> describes a unit of work that is being executed, e.g., by a simulated agent and is usually a series of requests and other steps. This could be one or more business transactions like "load start page", "search for product", "add to basket", "begin checkout", …</p>
<p><strong>Workload models</strong> describe the basic principle <em>how</em> a defined workload is executed in order to perform a performance test. In this article, we want to differentiate between <em>open</em> and <em>closed</em> workload models.</p>
<h2 id="closed-workloads">Closed Workloads</h2>
<p>In a closed workload model, you define a <strong>fixed number of concurrent agents</strong>, isolated from one another, each performing a defined sequence of tasks (the workload) over and over again in a loop. There may be a pause between iterations, but that is not important for this article.</p>
<p>Here is some pseudo code to give you an idea how a closed workload tool works in principle:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="k">for</span> <span class="mi">1</span> <span class="n">to</span> <span class="vg">$concurrency</span> <span class="k">do</span>
<span class="nb">fork</span> <span class="k">do</span>
<span class="k">while</span> <span class="k">do</span>
<span class="n">executeWorkload</span><span class="p">()</span>
<span class="nb">sleep</span><span class="p">(</span><span class="vg">$iterationDelay</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>A typical, well known and very simple example for a closed model is <a href="https://en.wikipedia.org/wiki/ApacheBench">Apache Bench</a> (or <code>ab</code> for short). With <code>ab</code> you basically state how much concurrency you want and what you want to hit and it will try to perform those requests as fast as possible. <a href="https://jmeter.apache.org">JMeter</a> is another example of a closed model load generator.</p>
<h2 id="open-workloads">Open Workloads</h2>
<p>In an open workload model, you have a <strong>defined rate of arrival</strong> at which new agents are spawned. Agents are isolated from one another, performing a defined sequence of tasks and are terminated when they have finished. Started agents are independent from one another and new agents are launched regardless of the state of currently active agents.</p>
<p>Again, here is an example in pseudo code how an open model implementation looks like:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="k">while</span> <span class="k">do</span>
<span class="nb">fork</span> <span class="k">do</span>
<span class="n">executeWorkload</span><span class="p">()</span>
<span class="k">end</span>
<span class="nb">sleep</span><span class="p">(</span><span class="mi">1</span><span class="o">/</span><span class="vg">$rateOfArrival</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div>
<p>StormForger uses an open workload model with the ability to mimic a closed model.<sup id="fnref:sf_closed_workload" role="doc-noteref"><a href="#fn:sf_closed_workload" class="footnote" rel="footnote">1</a></sup> Another example of an open workload model is <a href="http://tsung.erlang-projects.org">tsung</a> (which we've been using internally <a href="/blog/load-testing-engine-evolution/">for many years</a>).</p>
<h2 id="whats-the-big-deal">What’s the big deal?</h2>
<p>One obvious difference between the models is that closed workload systems are easier to reason about: You know the number of agents in the system beforehand (it's constant). For open models this number is a function of <a href="https://en.wikipedia.org/wiki/Little's_law">Little's Law</a>.<sup id="fnref:littles_law" role="doc-noteref"><a href="#fn:littles_law" class="footnote" rel="footnote">2</a></sup></p>
<p>To quickly re-iterate: Closed model systems have a fixed number of agents executing workloads. New work is only scheduled for an agent, when it is done with the previous one. This leads to a fundamental drawback: <strong>The system under test (SuT) coordinates the test itself.</strong> If the SuT is slowing down or stalls, then the entire test is impacted. During this time, all agents waiting for responses also stall, no new requests are made and load is taken away from the SuT which in turn allows the system to recover. Currently active agents in an open model system would also be impacted, but the crucial difference is, that new agents continue to arrive at the system. This keeps the pressure up at the target and better mimics real world situations like marketing campaigns: A slow shop experience will not stop customers from clicking ads, hitting reload or opening newsletters.</p>
<p>Imagine what happens to the <em>rate of new transactions</em> per time when the System under Test has a hick-up:</p>
<p><img src="/blog/open-closed-workloads/open-closed-ae0ead2f.png" alt="New transactions started with open VS closed models" /></p>
<p>There are also other issues regarding closed models, like the "Coordinated Omission Problem", a term coined by Gil Tene.<sup id="fnref:coordinated_omissions" role="doc-noteref"><a href="#fn:coordinated_omissions" class="footnote" rel="footnote">3</a></sup></p>
<p>True open models only exist in theory though. Resources are limited and there are practical upper limits on the number of active clients, which is technically unbounded. In case you are hitting resource limits with your open model tests, they should be considered inconclusive, discarded and repeated with more testing resources (or with a lower traffic model). This is the main reason why it is critically important to monitor the test itself closely which is what StormForger does automatically for every executed test.</p>
<h2 id="conclusion">Conclusion</h2>
<p>There are good reasons to use open and closed workload models. It is important to know the difference though.</p>
<p>We at StormForger believe that the open workload model is the one you probably want to use for many scenarios. It reflects better what happens in real world situations and it is better suited to detect problems with your system under test in many cases. Open models are harder to reason about but it's worth it to ensure your load tests are realistic and your system can perform in production as expected.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:sf_closed_workload" role="doc-endnote">
<p>Mimicking a closed workload model in StormForger is possible. You can specify an upper bound of clients to start per arrival phase. Using this, in combination with an endless loop around each scenario you effectively get a closed workload model. <a href="#fnref:sf_closed_workload" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:littles_law" role="doc-endnote">
<p>Little's Law (queueing theory) states, that for a stable system <em>L = λW</em>, where <em>L</em> is the average number of waiting customers in a stable system, <em>λ</em> is the average rate of arrival and <em>W</em> the average wait time of a customer.</p>
<p>This can be directly translated to an open model performance test: With an arrival rate of 100 per second, with an average time in the system of 15 sec (including requests and think times) the average number of active agents is: 100 arrivals/sec * 15 sec = 1500 clients. <a href="#fnref:littles_law" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:coordinated_omissions" role="doc-endnote">
<p>The <em>"Coordinated Omission Problem"</em>, a term coined by Gil Tene, see <a href="https://www.youtube.com/watch?v=lJ8ydIuPFeU">"How NOT to Measure Latency" (video)</a>.</p>
<p>Gil talks at length about what happens to your measurements and how it leads to false assumptions especially if systems under test stall for closed workload tests. While compensation is possible to a certain degree in those cases, it quickly gets difficult for non-trivial workload models. <a href="#fnref:coordinated_omissions" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Cookie Domain Mapping
https://stormforger.com/blog/cookie-domain-mapping/
2020-06-24T23:00:00+00:00
2020-06-24T23:00:00+00:00
Sebastian Cohnen
<p><a href="/blog/cookie-domain-mapping/"><img src="/blog/cookie-domain-mapping/inline-left-cookies-a2fba0b3.jpg" alt="Cookie"></a>
Performance testing often happens on non-production systems, like a QA or dedicated "perf" environments. In some cases it is easier to directly access those systems for testing instead through a load balancer serving traffic under a shared domain. This approach leads to a typical problem though: Cookie domain mismatches and lost sessions.</p>
<p></p>
<p><a href="/blog/cookie-domain-mapping/"><img src="/blog/cookie-domain-mapping/inline-left-cookies-a2fba0b3.jpg" alt="Cookie" /></a>
Performance testing often happens on non-production systems, like a QA or dedicated "perf" environments. In some cases it is easier to directly access those systems for testing instead through a load balancer serving traffic under a shared domain. This approach leads to a typical problem though: Cookie domain mismatches and lost sessions.</p>
<p></p>
<p><img src="/blog/cookie-domain-mapping/cookie-map-8837235e.png" alt="Cookies!" /></p>
<p>Let's say you are running an ecommerce site and have a QA environment under <code>shop.qa.example.com</code>. The shopping cart is managed by dedicated service running under <code>cart.qa.example.com</code>. In production the entire shop, including the cart component, is served from the same domain under <code>https://example.com</code>. Requests to <code>/cart/*</code> are routed to the shopping cart, while the rest is routed to the shop system.</p>
<p>In production, when everything is served under the same domain, you don't have any problems. For performance testing there can be any numbers of reasons to directly test against upstream systems. There is usually one problem though: Cookies. The standard cookie rules will not send cookies set for one system to another if the domain does not match.<sup id="fnref:rfc6265" role="doc-noteref"><a href="#fn:rfc6265" class="footnote" rel="footnote">1</a></sup></p>
<p>For this reason, we recently added a small feature called <a href="https://docs.stormforger.com/reference/sessions/#cookie-domain-mapping">"Cookie Domain Mapping"</a>. Cookie Domain Mapping allows to change the rules how cookies are handled in a StormForger test by providing mapping configuration for the <em>domain property</em> of cookies.</p>
<p>How can we apply this to our ecommerce example we've already described? The goal is, that cookies set by either system <code>{shop,cart}.qa.example.com</code> have to be available to every other system. To achieve this, you can provide a mapping like this:</p>
<div class="highlight"><pre class="highlight javascript"><code><span class="nx">session</span><span class="p">.</span><span class="nx">setOption</span><span class="p">(</span><span class="dl">"</span><span class="s2">cookie_domain_map</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">cart.qa.example.com</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">shop.qa.example.com</span><span class="dl">"</span><span class="p">,</span>
<span class="p">});</span>
</code></pre></div>
<p>Cookies set by <code>{shop,cart}.qa.example.com</code> will be handled like as if they were set by <code>shop.qa.example.com</code> (storing cookies). The same happens for the decision logic if cookies should be send back to the server (reading cookies). Note that cookies won't be sent to other domains, like <code>static.qa.example.com</code>.</p>
<h2 id="example">Example</h2>
<p>To get a more complete picture, let's take a look at a simple user journey with three steps: Visit start page, view a product and add a product to the shopping cart.</p>
<p><strong>Visit start page</strong>: A new client visits the start page. In the background a session cookie is generated and returned with a <code>Set-Cookie</code> header:</p>
<div class="highlight"><pre class="highlight javascript"><code><span class="nx">session</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">"</span><span class="s2">http://shop.qa.example.com/</span><span class="dl">"</span><span class="p">);</span>
</code></pre></div>
<p>Nothing special happens here in terms of cookie handling.</p>
<p><strong>View a product</strong>: The client now visits a product page. The value from the session cookie is for example used to fill the "recently viewed articles" list.</p>
<p>There is also no special treatment required for this, as the start page was served from the same domain and cookie handling works automatically in those cases:</p>
<div class="highlight"><pre class="highlight javascript"><code><span class="nx">session</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">"</span><span class="s2">http://shop.qa.example.com/4711</span><span class="dl">"</span><span class="p">);</span>
</code></pre></div>
<p><strong>Add product to shopping cart</strong>: Now we want to add the item to the shopping cart by performing a <code>POST</code> request. The session cookie is required to internally associate the cart to the correct user. <strong>But the domains do not match.</strong> We have to configure a domain map to send the cookie regardless of the mismatch:</p>
<div class="highlight"><pre class="highlight javascript"><code><span class="nx">session</span><span class="p">.</span><span class="nx">setOption</span><span class="p">(</span><span class="dl">"</span><span class="s2">cookie_domain_map</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">cart.qa.example.com</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">shop.qa.example.com</span><span class="dl">"</span><span class="p">,</span>
<span class="p">});</span>
<span class="nx">session</span><span class="p">.</span><span class="nx">post</span><span class="p">(</span><span class="dl">"</span><span class="s2">http://cart.qa.example.com/add</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="na">payload</span><span class="p">:</span> <span class="dl">"</span><span class="s2">ean=4711</span><span class="dl">"</span> <span class="p">});</span>
</code></pre></div>
<p>🎉</p>
<h2 id="conclusion">Conclusion</h2>
<p>Cookie Domain Mapping is a simple feature to manipulate the handling of cookies being saved and looked up by simulated clients in a StormForger test. It allows to make Cookies seamlessly work across different domain names which can become handy when testing non-production environments.</p>
<p>Check out our documentation on <a href="https://docs.stormforger.com/reference/sessions/#cookie-domain-mapping">"Cookie Domain Mapping"</a> to learn more.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:rfc6265" role="doc-endnote">
<p>This is of course a gross simplification. 😀 If you really want to understand how this works, check out <a href="https://tools.ietf.org/html/rfc6265">RFC6265: HTTP State Management Mechanism</a>. As it turns out, handling cookies correctly is surprisingly complex… 😞 <a href="#fnref:rfc6265" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Load Testing Engine Evolution
https://stormforger.com/blog/load-testing-engine-evolution/
2020-06-15T23:00:00+00:00
2020-06-15T23:00:00+00:00
Sebastian Cohnen
<p><a href="/blog/load-testing-engine-evolution/"><img src="/blog/load-testing-engine-evolution/inline-right-big-bang-0d008f07.jpg" alt="Load testing engine evolution"></a>
StormForger's load testing engine has been based on Tsung since we started our SaaS offering in 2014. In 2019 we began to design a new engine which now has been rolled out to all of our customers. Today I'd like to shed some light where we are coming from and what our reasoning is for moving forward without Tsung.</p>
<p></p>
<p><a href="/blog/load-testing-engine-evolution/"><img src="/blog/load-testing-engine-evolution/inline-right-big-bang-0d008f07.jpg" alt="Load testing engine evolution" /></a>
StormForger's load testing engine has been based on Tsung since we started our SaaS offering in 2014. In 2019 we began to design a new engine which now has been rolled out to all of our customers. Today I'd like to shed some light where we are coming from and what our reasoning is for moving forward without Tsung.</p>
<p></p>
<p>I'd like to start with how we came to use Tsung<sup id="fnref:tsung-usage" role="doc-noteref"><a href="#fn:tsung-usage" class="footnote" rel="footnote">1</a></sup> as our basis to build what you now know as StormForger.</p>
<h2 id="the-origin">The Origin</h2>
<p>I have been a long-time user of <a href="https://github.com/processone/tsung">Tsung</a> since around 2009, almost 5 years before StormForger was founded. It started as the foundation of an evaluation framework for my master thesis on "Design Patterns for Scalable, Service-oriented Webarchitectures" and I continued to use it in my time as a freelance consultant.<sup id="fnref:adcloud" role="doc-noteref"><a href="#fn:adcloud" class="footnote" rel="footnote">2</a></sup></p>
<p><img src="/blog/load-testing-engine-evolution/tsung-logo-12b429be.png" alt="Tsung" /></p>
<p>Tsung was always fascinating to me mainly because of its use of <a href="https://www.youtube.com/watch?v=xrIjfIjssLE">Erlang</a> and the design approach. It is relatively old (first commit 2001<sup id="fnref:tsung-first-commit" role="doc-noteref"><a href="#fn:tsung-first-commit" class="footnote" rel="footnote">3</a></sup>) and often misunderstood in terms of its strengths and weaknesses. Articles trying to compare Tsung to other performance testing tools often fall short because of a lack of understanding how Tsung is meant to be used.<sup id="fnref:tool-comparison" role="doc-noteref"><a href="#fn:tool-comparison" class="footnote" rel="footnote">4</a></sup></p>
<p>Tsung is <strong>incredibly efficient</strong> and scales easily to hundreds of thousands of concurrently active clients on a moderately sized distributed load generator cluster. Because of Erlang's, or better BEAM's, soft-real time properties it works beautifully for measuring duration which allowed us to run <a href="/blog/load-testing-an-interactive-tv-show-with-over-1-million-users/">very large scale test early on</a>.<sup id="fnref:tsung-process-hibernation" role="doc-noteref"><a href="#fn:tsung-process-hibernation" class="footnote" rel="footnote">5</a></sup></p>
<p>But Tsung is not without its flaws. One major shortcoming of Tsung in context of using it in an automated fashion is its design to be used by an operator: Tsung has been designed to be used by a human, sitting in front of a machine (or being SSH'd into one), running a bunch of commands, looking at logs etc. Tsung does not provide nice, machine-readable error messages, some statistics are in a <a href="http://tsung.erlang-projects.org/user_manual/reports.html">strange format</a> and meant to be processed by a <a href="https://www.perl.org">Perl</a> script generating gnuplot programs<sup id="fnref:tsung-reports" role="doc-noteref"><a href="#fn:tsung-reports" class="footnote" rel="footnote">6</a></sup>. Over time we have found several improvements and workarounds for the fact that Tsung was never meant to be automated. Changing this fundamentally is very hard though.</p>
<p>Another source of problems for us was that Tsung is a multi-protocol<sup id="fnref:tsung-protocols" role="doc-noteref"><a href="#fn:tsung-protocols" class="footnote" rel="footnote">7</a></sup> load testing tool, almost more like a framework for network-based performance testing. StormForger on the other hand has always been focused on HTTP. This made it very hard sometimes to change things in Tsung because of its generic approach to many aspects. For example, supporting HTTP/2 would be a massive undertaking<sup id="fnref:tsung-http2" role="doc-noteref"><a href="#fn:tsung-http2" class="footnote" rel="footnote">8</a></sup> which we considered multiple times. In the end it's a good example that you cannot have it both ways: Highly generic and still strong in specialised scenarios.</p>
<p>We came to the conclusion that we need a purpose build engine, that is less generic and built specifically to our needs.</p>
<h2 id="the-goal">The Goal</h2>
<p>The goal with our new engine was to be specialised to what we need: focused on HTTP and most importantly built to be automated and integrated from the beginning.</p>
<p>We have a lot of interesting features in mind to kick off the next generation of our engine, but our first goal was to have new foundation to build upon. This was also important to us, because we have to keep in mind the many thousand test case definitions that our customers have written over the years. Since many of them run automatically we need to minimize the migration efforts our customers have to do.</p>
<p>We quickly came to the conclusion that we should aim for the following:</p>
<ol>
<li>Be as close as possible to a <strong>drop-in-replacement</strong> for the current feature set.</li>
<li>Support <strong>new need-to-have features right away</strong> if this is not in direct conflict with point 1.</li>
<li>Design and build for <strong>automation from the beginning</strong>, including live profiling in production, better monitoring and observability and simpler operations in general.</li>
<li>Layout a <strong>foundation for new features</strong> with possible breaking changes to existing test cases.</li>
</ol>
<p>So far we are quite pleased with the outcome: Only some undocumented, special features we built for our customers needed some adjustment. Without directly breaking compatibility we were still able to bring some <a href="https://docs.stormforger.com/reference/engine/">new features and better behaviour</a> right from the beginning to all our customers. For example this includes better debuggability for our customers, support for HTTP/2 and TLS1.3 and even faster "time to first request" when a test starts.</p>
<p>We also greatly improved our internal development and testing processes which already allowed us on several occasions to deliver new features rapidly.</p>
<h2 id="the-migration-path">The Migration Path</h2>
<p>Before we actively started over the migration of StormForger customers to our new engine, we made a lot of internal experiments and sanity checks in addition to our usual automated test suite. Since our <a href="https://docs.stormforger.com/guides/getting-started/">test case DSL</a> is a declarative description, we could also challenge our new engine with many thousands of existing test cases to see if we missed something – all without actually running tests against our customers's infrastructure.</p>
<p>The next step was migrating the first customers to our new engine. We started with this a few months ago by picking customers with whom we have shared Slack channels as part of their extended support package and offered them to give our new engine a test. We only switched over a couple of customers and observed their test runs very closely. Since many run their tests automated on a daily basis, we were able to gather lots of data, address smaller issues right away and pushed out an updated version rapidly.</p>
<p>Some weeks ago, we defaulted all new customers to our new engine while only a small fraction remained pinned to our old engine based on Tsung. This helped customers with a larger test code base and lots of active DevOps teams to get the required evaluation done and to ensure that everything is working as expected.</p>
<h2 id="the-road-ahead">The Road Ahead</h2>
<p>Our new <a href="https://docs.stormforger.com/reference/engine/">engine</a> solved some long standing design problem we had right away and enabled some features right from the start: Mainly HTTP/2 and multiple open connections per simulated client - which was simply not possible with Tsung before. We also improved a lot of our internal tooling and significantly improved the tools we can offer our customers to create and debug their test cases.</p>
<p>With our new engine and a new foundation in place, we are already working on new features related to capturing more metrics, better and deeper integration into our customers's development and QA processes as well as making our test case DSL even more expressive.</p>
<p>We will keep supporting our beloved, legacy engine for a few more weeks for the unlikely case that customers encounter an issue. After that we can tackle the next features we have in mind that are a breaking change to our old engine.</p>
<p>It is not without mixed feelings that we have to say: Farewell, dear Tsung. Thank you for many billion requests and years of great service. We'll miss you. 🤧.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:tsung-usage" role="doc-endnote">
<p>Although we never denied that we use Tsung, we did not communicate it publicly. Some folks are surprisingly good at looking for details like <a href="https://news.ycombinator.com/item?id=7921942">a commenter on Hacker News in 2014</a> shows: He pointed out, that parts of our logs looked like an Erlang process identifier and our keywords in our DSL have a lot of familiar terms to Tsung users. Crazy! <img src="/blog/load-testing-engine-evolution/hacker-news-erlang-pid-8e05ded2.png" alt="Hacker News" /> <a href="#fnref:tsung-usage" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:adcloud" role="doc-endnote">
<p>At one gig I had as a freelancer working on performance optimisations and scalable architectures I actually worked together with <a href="/team">Lars and Stephan</a> <a href="#fnref:adcloud" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:tsung-first-commit" role="doc-endnote">
<p>Tsung's first commit in May 2001: <a href="https://github.com/processone/tsung/commit/cf947d6fa1f1341b3253c6e5918379ecd2fdb30e">"New repository initialized by cvs2svn"</a> <a href="#fnref:tsung-first-commit" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:tool-comparison" role="doc-endnote">
<p>I won't call names, but usually comparisons are biased. Tsung seems to be the misunderstood underdog in most articles I've read so far and I can't remember a writeup where Tsung actually made it as "the winner" 🤔 <a href="#fnref:tool-comparison" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:tsung-process-hibernation" role="doc-endnote">
<p>Actually one of the reasons Tsung is so efficient with many ten-thousand of active users is because of <a href="http://www.erlang.org/doc/man/erlang.html#hibernate-3">Erlang's process hibernation feature</a> <a href="#fnref:tsung-process-hibernation" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:tsung-reports" role="doc-endnote">
<p><a href="http://tsung.erlang-projects.org/user_manual/reports.html#tsung-summary">http://tsung.erlang-projects.org/user_manual/reports.html#tsung-summary</a> <a href="#fnref:tsung-reports" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:tsung-protocols" role="doc-endnote">
<p>Supported protocols are HTTP, WebSockets, WebDAV, SOAP, PostgreSQL, MySQL, LDAP, MQTT, AMQP and Jabber/XMPP <a href="#fnref:tsung-protocols" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:tsung-http2" role="doc-endnote">
<p>I don't want to go into all the details why HTTP/2 would be quite hard to implement. Tsung comes from a time where there was a little 3rd party ecosystem available for Erlang. Everything is hand crafted and implemented in Tsung directly, down to the HTTP client and connection handling. While certainly possible it was not feasible with our resources. <a href="#fnref:tsung-http2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Don't give up yet… keep-alive!
https://stormforger.com/blog/dont-give-up-yet-keep-alive/
2020-06-04T23:00:00+00:00
2020-06-04T23:00:00+00:00
Sebastian Cohnen
<p><a href="/blog/dont-give-up-yet-keep-alive/"><img src="/blog/dont-give-up-yet-keep-alive/inline-left-telephone-exchange-15accccf.jpg" alt="Load testing engine evolution"></a>
There is one source of performance problems we've been encountering even before we started with StormForger and still see every time: Missing HTTP Keep-Alive. This article is about why this is still a problem and relevant in 2020 and why performance testing is an important tool to uncover such issues.</p>
<p></p>
<p><a href="/blog/dont-give-up-yet-keep-alive/"><img src="/blog/dont-give-up-yet-keep-alive/inline-left-telephone-exchange-15accccf.jpg" alt="Load testing engine evolution" /></a>
There is one source of performance problems we've been encountering even before we started with StormForger and still see every time: Missing HTTP Keep-Alive. This article is about why this is still a problem and relevant in 2020 and why performance testing is an important tool to uncover such issues.</p>
<p></p>
<p>HTTP is with its 24 years a well aged fellow among the web protocols.<sup id="fnref:http09" role="doc-noteref"><a href="#fn:http09" class="footnote" rel="footnote">1</a></sup> Today we are mostly using HTTP/1.1<sup id="fnref:http11_rfc" role="doc-noteref"><a href="#fn:http11_rfc" class="footnote" rel="footnote">2</a></sup> or <a href="https://tools.ietf.org/html/rfc7540">HTTP/2</a> and if you have fully embraced the new HTTP/2 world in your entire system this article is mostly an anecdote of past issues. But HTTP/1.1 is still alive and kicking for many systems. And even given its age, people are still forgetting about a very important feature that previous versions did not provide: Keep-Alive.<sup id="fnref:http10_keep_alive" role="doc-noteref"><a href="#fn:http10_keep_alive" class="footnote" rel="footnote">3</a></sup></p>
<p>To clarify, I'm not talking about TCP keep-alive (which is disabled by default). Also I'm not talking about other kinds of keep-alive mechanisms for other protocols which are equally important to keep an eye on. Today, we will focus on HTTP keep-alive.</p>
<h2 id="how-does-http-work">How does HTTP work?</h2>
<p>HTTP (at least prior to HTTP/2) is a very simple protocol. For a given request to fetch data from a server the following steps happen (simplified):</p>
<p><img src="/blog/dont-give-up-yet-keep-alive/tcp-tls-handshake-433c08e0.svg" alt="TLS Handshake!" /></p>
<ul>
<li>DNS lookup is made (not in picture),</li>
<li>a new TCP connection is established,</li>
<li>the TLS handshake is performed,</li>
<li>request headers and optional payload is sent,</li>
<li>the response is read and</li>
<li>the <strong>connection is closed</strong>.</li>
</ul>
<p>The last point is the topic of this article: Don't close the connection!</p>
<p>HTTP 1.1 learned to re-use an existing connection: If the response was read entirely, a new request could be sent using the existing connection. This happens automatically if both parties understand it. Unless the client sets the <code>Connection: close</code> request header or the server actively closes the connection it will be reused for subsequent requests. Sounds like a no-brainer, right?</p>
<h2 id="why-is-this-important-why-bother">Why is this important? Why bother?</h2>
<p>We seem to forget about the fact that there might be an issue with keep-alive. Almost everyone seems to be aware that his concept exists, but few are actively checking that everything is working as expected. You might be surprised how often keep-alive is not configured properly!</p>
<p>The other issue is: Developers and operations people <strong>heavily</strong> underestimate the impact of doing a DNS lookup, establishing a TCP connection and making a TLS handshake. Over and over again. For every single HTTP request. Every. Single. Time.</p>
<p>From our experience we can tell that the overhead will add up very quickly. And it does not make a big difference what kind of system you are building. Even for internal or even local systems there is usually not really anything to gain from closing the connection. You don't have to take our word for it – there are <a href="https://hpbn.co/http1x/#benefits-of-keepalive-connections">many resources</a> out there supporting this.</p>
<p>What we and our customers are observing when running tests with missing keep-alive is slower response times even for moderate load. If more and more requests take longer to process, more connections stay active so more resources are consumed and blocked. In many cases systems under tests do not recover until traffic stops.</p>
<p>Here is a quick example I've done a while back for a <a href="https://www.meetup.com/aws-cologne/events/260502656/">talk at the AWS User Group in Cologne</a>. I used a simple StormForger test case to give you an idea how the TCP reconnects impacts latency (find the test definition at the end of this article). The following image is a latency histogram over all requests made by this test (available in all StormForger reports):</p>
<p><img src="/blog/dont-give-up-yet-keep-alive/bimodal-dist-69204a66.png" alt="Bimodal Distribution" /></p>
<p>You might have already guessed it: Left is with keep-alive, right is without. Same target, same request, same response.</p>
<p>Yes, this is a simple and a bit artificial example, but not so far from many setups we see our customers are testing. We see a clear bimodal distribution: One maximum where new connections need to be established and the other when an existing connection is being used. The difference is rather significant.</p>
<p>The difference comes from multiple factors:</p>
<ul>
<li>only spend DNS, TCP and TLS once per peer (multiple times if you are using a pool of connections)</li>
<li>allocating a TCP socket is also not for free, especially when the system is under load</li>
<li>resources are finite and keeping sockets around can also quickly add up. Also look out for sockets in the <code>TIME_WAIT</code> state.</li>
<li>worst-case: You can also run out of <a href="https://en.wikipedia.org/wiki/Ephemeral_port">ephemeral ports</a>.</li>
</ul>
<p>If you want to learn more about TCP, sockets and <code>TIME_WAIT</code> and how to optimize your servers, check out this great article by Vincent Bernat: <a href="https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux">https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux</a>.</p>
<h2 id="keep-alive-and-current-architectural-approaches">Keep-Alive and Current Architectural Approaches</h2>
<p>The issue with keep-alive being overlooked is that the impact gets bigger considering some currently trending architectural approaches.</p>
<p>Take for example Server-less or Function-as-a-Service (FaaS)<sup id="fnref:nodejs_keep_alive" role="doc-noteref"><a href="#fn:nodejs_keep_alive" class="footnote" rel="footnote">4</a></sup>. With FaaS you need to be stateless, but an application is usually not really fully stateless. Most of the time you solve this by externalizing state to other components and services. And how do you access the state again? Quite often it is done via HTTP. You should also check out <a href="https://twitter.com/theburningmonk">Yan Cui's</a> article on <a href="https://theburningmonk.com/2019/02/lambda-optimization-tip-enable-http-keep-alive/">HTTP keep-alive as an optimization for AWS Lambda</a>.</p>
<p>This especially affects Microservices: HTTP is often selected as the communication protocol of choice.</p>
<p>Again and again we are witnesses when our customers uncover these problems using performance tests and have rather quick wins in terms of latency, stability and general efficiency.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Use HTTP keep-alive. Always.</p>
<p>More importantly don't just assume it is used, check it. It can easily be tested with curl via <code>curl -v http://example.com</code> and looking for <code>* Connection #0 to host example.com left intact</code> at the end of the output. Testing it on a larger scale and especially the impact is also done easily with a performance test using StormForger. Catching a misconfiguration or a unintended configuration change using automated performance testing is even better because you minimize the risk of the potential havoc.</p>
<h2 id="more-details">More Details</h2>
<p>I've been using a simple test case to showcase the impact of HTTP keep-alive. We have two scenarios, each weighted 50%. One session does 25 HTTP requests with keep-alive (which is the default with StormForger) and the other one does 25 HTTP requests without keep-alive.</p>
<p>Note that our <a href="https://github.com/stormforger/testapp">testapp</a> does HTTP keep-alive by default:</p>
<div class="highlight"><pre class="highlight javascript"><code><span class="nx">definition</span><span class="p">.</span><span class="nx">session</span><span class="p">(</span><span class="dl">"</span><span class="s2">keep-alive</span><span class="dl">"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">session</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Every clients gets a new environment, so the first</span>
<span class="c1">// request cannot reuse an existing connection.</span>
<span class="nx">context</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">"</span><span class="s2">http://testapp.loadtest.party/</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="na">tag</span><span class="p">:</span> <span class="dl">"</span><span class="s2">no-keep-alive</span><span class="dl">"</span><span class="p">,</span> <span class="p">});</span>
<span class="c1">// HTTP Keep-Alive is the default, so for all the following</span>
<span class="c1">// requests in this loop, we can reuse the connection.</span>
<span class="nx">session</span><span class="p">.</span><span class="nx">times</span><span class="p">(</span><span class="mi">26</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">context</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">context</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">"</span><span class="s2">http://testapp.loadtest.party/</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="na">tag</span><span class="p">:</span> <span class="dl">"</span><span class="s2">keep-alive</span><span class="dl">"</span> <span class="p">});</span>
<span class="nx">context</span><span class="p">.</span><span class="nx">waitExp</span><span class="p">(</span><span class="mf">0.5</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="nx">definition</span><span class="p">.</span><span class="nx">session</span><span class="p">(</span><span class="dl">"</span><span class="s2">no-keep-alive</span><span class="dl">"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">session</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Setting the "Connection: close" header, we signal our</span>
<span class="c1">// client to close the connection when the transfer has</span>
<span class="c1">// finished, regardless if the server offers to keep the</span>
<span class="c1">// connection intact.</span>
<span class="nx">session</span><span class="p">.</span><span class="nx">times</span><span class="p">(</span><span class="mi">25</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">context</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">context</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">"</span><span class="s2">http://testapp.loadtest.party/</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">tag</span><span class="p">:</span> <span class="dl">"</span><span class="s2">no-keep-alive</span><span class="dl">"</span><span class="p">,</span>
<span class="na">headers</span><span class="p">:</span> <span class="p">{</span> <span class="na">Connection</span><span class="p">:</span> <span class="dl">"</span><span class="s2">close</span><span class="dl">"</span><span class="p">,</span> <span class="p">},</span>
<span class="p">});</span>
<span class="nx">context</span><span class="p">.</span><span class="nx">waitExp</span><span class="p">(</span><span class="mf">0.5</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">});</span>
</code></pre></div>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:http09" role="doc-endnote">
<p>Actually HTTP is even older, but I'm referring to <a href="https://tools.ietf.org/html/rfc1945">RFC1945</a>, or HTTP V1.0. <a href="https://www.w3.org/Protocols/HTTP/AsImplemented.html">HTTP V0.9</a> actually dates back almost 30 years. <a href="#fnref:http09" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:http11_rfc" role="doc-endnote">
<p>HTTP 1.1 is actually a collection of RFCs: <a href="https://tools.ietf.org/html/rfc7230">RFC 7230, HTTP/1.1: Message Syntax and Routing</a>, <a href="https://tools.ietf.org/html/rfc7231">RFC 7231, HTTP/1.1: Semantics and Content</a>, <a href="https://tools.ietf.org/html/rfc7232">RFC 7232, HTTP/1.1: Conditional Requests</a>, <a href="https://tools.ietf.org/html/rfc7233">RFC 7233, HTTP/1.1: Range Requests</a>, <a href="https://tools.ietf.org/html/rfc7234">RFC 7234, HTTP/1.1: Caching</a>, <a href="https://tools.ietf.org/html/rfc7235">RFC 7235, HTTP/1.1: Authentication</a> <a href="#fnref:http11_rfc" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:http10_keep_alive" role="doc-endnote">
<p>Technically HTTP 1.0 could also support keep-alive but it was opt-in and not actually specified how this should work in detail. If the client wants a connection to be reused, one has to send <code>Connection: keep-alive</code> and check if the server responds with the same header. Only then (depending on the implementation) the connection was kept intact after a request. <a href="#fnref:http10_keep_alive" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:nodejs_keep_alive" role="doc-endnote">
<p>Node.js's HTTP client or better <a href="https://nodejs.org/api/http.html#http_class_http_agent">HTTP Agent</a> does not keep connections alive. You have to configure it explicitly, which is a bummer, because Node.js is a pretty popular technology for FaaS and Server-less applications. <a href="#fnref:nodejs_keep_alive" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Performance Testing at over 1 Million Requests per Second
https://stormforger.com/blog/perf-testing-over-1-million-requests/
2020-05-28T23:00:00+00:00
2020-05-28T23:00:00+00:00
Sebastian Cohnen
<p><a href="/blog/perf-testing-over-1-million-requests/"><img src="/blog/perf-testing-over-1-million-requests/inline-right-streets-bd98b56f.jpg" alt="Load testing engine evolution"></a>
At StormForger we regularly evaluate our load testing setup and infrastructure to make sure everything works as expected for our customers and to anticipate issues early on. Since we have been working on a revamp of our core load testing engine for the past weeks and months it was about time to scale our internal testing efforts up to eleven!</p>
<p></p>
<p><a href="/blog/perf-testing-over-1-million-requests/"><img src="/blog/perf-testing-over-1-million-requests/inline-right-streets-bd98b56f.jpg" alt="Load testing engine evolution" /></a>
At StormForger we regularly evaluate our load testing setup and infrastructure to make sure everything works as expected for our customers and to anticipate issues early on. Since we have been working on a revamp of our core load testing engine for the past weeks and months it was about time to scale our internal testing efforts up to eleven!</p>
<p></p>
<h2 id="goal">Goal</h2>
<p>We recently finished switching over our customers to our new load testing engine. While having done a lot of testing internally, we thought: Why not go a little over our requirements? No sooner said than done. Our goal? Make more then one million requests per second with at least one million active clients. 😱</p>
<p>To be honest: Most of our clients have less traffic they need to simulate. Some require a large amount of bandwidth, others have very little, but very valuable traffic. But we always need to be aware what our limits are so that our customers can rely on StormForger.</p>
<h2 id="premise">Premise</h2>
<p>The use case we selected as the basis for our experiments is based on a fictitious <a href="https://en.wikipedia.org/wiki/Hybrid_Broadcast_Broadband_TV">HbbTV</a> scenario were the requirement is to have a large number of viewers having their HbbTV enabled devices sending a regular heartbeat.</p>
<p>Although streaming is getting more and more popular, "normal" broadcast television is still VERY big around the world. HbbTV is a solution to bring more interactivity into the broadcasting industry. It is also used for analytics purposes which we want to use as an example in this article.</p>
<p>What we had in mind boils down to a very simple scenario per simulated client:</p>
<ul>
<li>randomly pick a station identifier from a data source</li>
<li>make a POST request acquiring a session token for the station being watched</li>
<li>enter a loop for several minutes</li>
<li>make a heartbeat HTTP request including the session token roughly every second</li>
</ul>
<p>By the way, some readers will remember that we have written about TV-related and <a href="https://stormforger.com/blog/load-testing-an-interactive-tv-show-with-over-1-million-users/">very large tests</a> before in the past. Compared to our adventures testing a very large scale interactive German TV show in 2014, this test will be way bigger!</p>
<h2 id="challenges-and-test-setup">Challenges and Test Setup</h2>
<p>The first problem we encountered was actually not our load generator setup itself. We have many years of experience in ad-hoc provisioning of cloud resources and managing them without any manual intervention. The problem was to set up a target that can actually handle the load. So how to test a very efficient, scalable load testing engine?</p>
<p>As you might know, we have our <a href="https://github.com/stormforger/testapp">open source testapp</a> publicly <a href="https://testapp.loadtest.party">available</a>. This is a simple Golang application and we are running it on a cheap yet beefy bare metal server. The box is quite powerful but not nearly enough to handle north of 1,000,000 requests per second with many more established TCP connections.</p>
<p>We are always looking at different cloud technologies mostly out of curiosity. One of the things that caught our interest already in the past was <a href="https://stormforger.com/blog/aws-fargate-network-performance/">AWS Fargate</a>, AWS's solution to run containers without having to manage servers or clusters. Since we build a <a href="https://hub.docker.com/r/stormforger/testapp">docker image for our testapp</a> already, we thought why not try to run and scale this on Fargate?</p>
<p>After some experiments on how efficient our testapp runs on Fargate we ended up with the following configuration:</p>
<ul>
<li>one AWS Network Load Balancer to have a single target to test</li>
<li>150 Fargate containers with 2 cores and 4GB RAM each running our testapp (we hit some problems in utilising more than 2 cores efficiently, but that's a story for another time).</li>
<li>test target and load generator cluster located in Dublin, Ireland</li>
</ul>
<p>To set up and manage our target on Fargate we used <a href="https://github.com/awslabs/fargatecli">fargatecli</a> which is a command line tool to setup Fargate tasks and services. Once we have requested an increase in the allowed Fargate tasks, provisioning our test target was very simple. First we created a new network load balancer (NLB) and then created a new Fargate service with 150 instances of our testapp.</p>
<h2 id="preparations">Preparations</h2>
<p>Before we actually ran the fully scaled test, we set ourselves some intermediary steps. The issue going from 0 to 100 usually is that problems quickly generate a lot of noise if you run at higher traffic scenarios. Generally it is a good idea to cover some basic ground first.</p>
<p>We defined three steps which is something we recommend our customers to do as well:</p>
<p><strong>0) Measure the base capacity of the target system</strong></p>
<p>First we made a series of tests against different combinations of CPU/memory configurations of our testapp deployed on a single AWS Fargate task to establish a baseline. Our goal was to get close to 80% CPU utilisation per container.</p>
<p>While our testapp usually runs on a beefy bare metal server we quickly realised that our app behaves quite a bit different on AWS Fargate: We get around 6.250 req/sec/core on our bare metal 8 cores box and only 3.250 req/sec/core on 2-core 4GB containers. 🤔 Since our goal is not to optimise for performance here and scaling Fargate is not an issue, we had to grit our teeth and carry on. 😁</p>
<p>We also noticed that the <a href="https://docs.amazonaws.cn/en_us/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_limits">default <code>nofile</code> limit in Fargate tasks is 2048/4096 (soft/hard)</a> which we had to increase as we were planning to have many thousand connections per container.</p>
<p><strong>1) Run at ~10% for 5 minutes</strong></p>
<p>The first step towards our goal went very smoothly. We mainly used this step to verify our results from our capacity testing step to improve our confidence for the next steps.</p>
<p><strong>2) Run at ~50% for 10 minutes</strong></p>
<p>Because we are not using AWS Fargate in our daily operation, we immediately hit the default account limit of available AWS Fargate Task instances. Before we could continue we had to request an increase with AWS Support. When this was approved we could run our test without further issues.</p>
<h2 id="fire-it-up">Fire it up!</h2>
<p>After passing our internal milestones to verify that our target would not be overloaded by accident and everything behaves the way we want it for our experiment, it was time for our first test run.</p>
<p>It took almost exactly 11 seconds for our load generator cluster to be provisioned after we hit the "Launch new Test Run" button. 17 seconds later we reached 1 million requests per second!</p>
<p><img src="/blog/perf-testing-over-1-million-requests/running-test-ff703fb1.png" alt="1M rps!" /></p>
<p>After that we have performed a series of tests at that scale to pinpoint possible optimisations in our test infrastructure and analytics components. It turns out that processing hundreds of millions of requests per test run takes a moment and generates quite a bit of data that need to be taken care of.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Using our new load generator engine (which is now enabled by default for all customers) scaling a StormForger performance test up to very large scenarios is the same effort from the user perspective compared to a very small scale test.</p>
<p>Setting up a test target that can handle over a million active clients and well over 1 million requests per second was a bit more involved. Our usual and recommended approach to make intermediate steps towards your testing goal has proven to be a good idea — especially because we saw significant differences in vertical scalability on Fargate for our testapp (investigation pending). Horizontal scaling on the other hand was very straight forward: Just crank up the number of tasks and you are good to go!</p>
<p>During our testing, we generated hundreds of millions requests which needed to be analysed. We confirmed some assumptions we had with our test analysis pipeline which we are planning to work on to further improve processing times as our goal is that nothing should take longer then 60 seconds. Once analysed we were glad to see that our reports and latency analysis tooling work as expected with sub-second delays.</p>
<p>Do you have questions or remarks regarding AWS Fargate, large scale testing or other performance topics? Just <a href="/support">drop us a line</a> :)</p>
<p><img src="/blog/perf-testing-over-1-million-requests/graphs-7cfd93f1.png" alt="Test Run Graphs" /></p>
<p><img src="/blog/perf-testing-over-1-million-requests/latency-dist-0a5a1266.png" alt="Latency Distribution Analysis" /></p>
<h2 id="some-more-details">Some More Details</h2>
<p>Here are some more details on the setup:</p>
<ul>
<li>we used <a href="https://github.com/awslabs/fargatecli">fargatecli</a> to setup the Network Load Balancer and Fargate tasks</li>
<li>we used 150 Fargate container, each with 2048 CPU units and 4096 MB memory</li>
<li>each container handled around 6.500 requests per second at 80% CPU utilisation</li>
<li>we used HTTP/2 and gzip compression for all requests</li>
</ul>
<p>The StormForger test case definition that was executed is pretty straight forward. Note that we artificially delay the responses of our testapp to get a bit more realistic "processing time" of the target.</p>
<div class="highlight"><pre class="highlight javascript"><code><span class="nx">definition</span><span class="p">.</span><span class="nx">setTarget</span><span class="p">(</span><span class="dl">"</span><span class="s2">http://testapp-lb-657638500f694428.elb.eu-west-1.amazonaws.com:8080</span><span class="dl">"</span><span class="p">);</span>
<span class="c1">// 1,000,000 active users, ~5min per User: 3333.33 arrivals/sec</span>
<span class="nx">definition</span><span class="p">.</span><span class="nx">setArrivalPhases</span><span class="p">([{</span> <span class="na">duration</span><span class="p">:</span> <span class="mi">10</span> <span class="o">*</span> <span class="mi">60</span><span class="p">,</span> <span class="na">rate</span><span class="p">:</span> <span class="mf">3333.33</span><span class="p">,</span> <span class="p">}]);</span>
<span class="nx">definition</span><span class="p">.</span><span class="nx">session</span><span class="p">(</span><span class="dl">"</span><span class="s2">up-to-eleven</span><span class="dl">"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">session</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">channels</span> <span class="o">=</span> <span class="nx">session</span><span class="p">.</span><span class="nx">ds</span><span class="p">.</span><span class="nx">loadStructured</span><span class="p">(</span><span class="dl">"</span><span class="s2">hbbtv_channels.csv</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">channel</span> <span class="o">=</span> <span class="nx">session</span><span class="p">.</span><span class="nx">ds</span><span class="p">.</span><span class="nx">pickFrom</span><span class="p">(</span><span class="nx">channels</span><span class="p">);</span>
<span class="c1">// Get a "watching token" for the selected channel</span>
<span class="nx">session</span><span class="p">.</span><span class="nx">post</span><span class="p">(</span><span class="dl">"</span><span class="s2">/random/get_token?delay=100&channel=:id</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">tag</span><span class="p">:</span> <span class="dl">"</span><span class="s2">get_token</span><span class="dl">"</span><span class="p">,</span>
<span class="na">params</span><span class="p">:</span> <span class="p">{</span>
<span class="na">id</span><span class="p">:</span> <span class="nx">channel</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">"</span><span class="s2">id</span><span class="dl">"</span><span class="p">),</span>
<span class="p">},</span>
<span class="na">extraction</span><span class="p">:</span> <span class="p">{</span>
<span class="na">jsonpath</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">token</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">$.token</span><span class="dl">"</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="nx">session</span><span class="p">.</span><span class="nx">assert</span><span class="p">(</span><span class="dl">"</span><span class="s2">token_received</span><span class="dl">"</span><span class="p">,</span> <span class="nx">session</span><span class="p">.</span><span class="nx">getVar</span><span class="p">(</span><span class="dl">"</span><span class="s2">token</span><span class="dl">"</span><span class="p">),</span> <span class="dl">"</span><span class="s2">!=</span><span class="dl">"</span><span class="p">,</span> <span class="dl">""</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">pings</span> <span class="o">=</span> <span class="nx">session</span><span class="p">.</span><span class="nx">ds</span><span class="p">.</span><span class="nx">generate</span><span class="p">(</span><span class="dl">"</span><span class="s2">random_number</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="na">range</span><span class="p">:</span> <span class="p">[</span><span class="mi">280</span><span class="p">,</span> <span class="mi">320</span><span class="p">]});</span>
<span class="nx">session</span><span class="p">.</span><span class="nx">times</span><span class="p">(</span><span class="nx">pings</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">ctx</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">"</span><span class="s2">/ping?delay=100&token=:token</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">tag</span><span class="p">:</span> <span class="dl">"</span><span class="s2">ping</span><span class="dl">"</span><span class="p">,</span>
<span class="na">params</span><span class="p">:</span> <span class="p">{</span>
<span class="na">id</span><span class="p">:</span> <span class="nx">session</span><span class="p">.</span><span class="nx">getVar</span><span class="p">(</span><span class="dl">"</span><span class="s2">token</span><span class="dl">"</span><span class="p">),</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="nx">ctx</span><span class="p">.</span><span class="nx">waitExp</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">});</span>
</code></pre></div>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
AWS Fargate Network Performance
https://stormforger.com/blog/aws-fargate-network-performance/
2018-06-03T23:00:00Z
2018-06-03T23:00:00Z
Sebastian Cohnen
<p><a href="/blog/aws-fargate-network-performance"><img src="/blog/aws-fargate-network-performance/aws_fargate_logo-3b0a0457.png" alt="AWS Fargate network Performance"></a></p>
<p>AWS Fargate is Amazons solution to run containers without managing servers or clusters. For many aspects AWS Fargate is similar to AWS Elastic Container Service but without having to deal with EC2 clusters. It is basically "server-less containers".</p>
<p>Let's take a first look at AWS Fargate's network performance!</p>
<p><strong>TLDR</strong>; Sustained network throughput is very stable, but not symmetric and does not really grow with assigned container resources.</p>
<p></p>
<p><a href="/blog/aws-fargate-network-performance"><img src="/blog/aws-fargate-network-performance/aws_fargate_logo-3b0a0457.png" alt="AWS Fargate network Performance" /></a></p>
<p>AWS Fargate is Amazons solution to run containers without managing servers or clusters. For many aspects AWS Fargate is similar to AWS Elastic Container Service but without having to deal with EC2 clusters. It is basically "server-less containers".</p>
<p>Let's take a first look at AWS Fargate's network performance!</p>
<p><strong>TLDR</strong>; Sustained network throughput is very stable, but not symmetric and does not really grow with assigned container resources.</p>
<p></p>
<p><a href="https://aws.amazon.com/fargate/">AWS Fargate</a> was <a href="https://aws.amazon.com/about-aws/whats-new/2018/04/aws-fargate-now-available-in-ohio--oregon--and-ireland-regions/">recently launched</a> in Dublin/Ireland (<code>eu-west-1</code>) which is a good opportunity to take a look at some performance characteristics of that service.</p>
<p>There are various articles on AWS EC2 network performance, sometimes between instances, sometimes to other AWS services like S3 (e.g. by <a href="https://cloudonaut.io/ec2-network-performance-cheat-sheet/">Andreas Wittig</a>). For EC2 we know those numbers very well and we regularly check them in order to recommend correct <a href="https://docs.stormforger.com/reference/test-cluster/">StormForger cluster sizes</a> to our customers. <strong>Known network performance is very important in order not to be mislead when doing performance testing.</strong> For EC2, raw network performance in terms of bandwidth depends on instance type and more importantly on instance size within one instance family. The general rule of thumb: The bigger the instance (and the more you pay), the better networking gets.</p>
<p>Back to AWS Fargate. We were wondering what kind of network performance can one expect from AWS Fargate? And how does the container's resource sizing (CPU and memory) relate to its network performance? These might be important parameters to know if you have a bandwidth dependent workload.</p>
<h2 id="preparation-and-test-setup">Preparation and Test Setup</h2>
<p>To asses network performance we are going to use <a href="https://iperf.fr/iperf-download.php">iPerf3</a>. To make the measurement not being limited on the iPerf server part, we are using beefy 72 core <code>c5.18xlarge</code> instances, which are advertised with up to 25Gibt/s network performance. During preparation we did sanity checks between two <code>c5.18xlarge</code> instances where we saw 22Gibt/s sustained throughput with peaks reaching close to 25Gibt/s. That should provide us with enough headroom for our experiments.</p>
<p><img src="/blog/aws-fargate-network-performance/cores-aec6cb1c.png" alt="MORE CORES" /></p>
<p>Our test target will be AWS Fargate containers launched in <code>eu-west-1</code>. Fargate allows five tiers of "CPU Units" with a range of memory (check <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html">AWS Fargate documentation</a>). Since we were interested in the relation of network performance and CPU/memory sizing, we decided to test 10 scenarios, the minimum and maximum memory values for each of the five CPU Unit tiers:</p>
<table>
<thead>
<tr>
<th>CPU Units</th>
<th>vCPU</th>
<th>Memory (MiB)</th>
<th>Price per hour (USD)</th>
</tr>
</thead>
<tbody>
<tr>
<td>256</td>
<td>0.25</td>
<td>512</td>
<td>0.019</td>
</tr>
<tr>
<td>256</td>
<td>0.25</td>
<td>2048</td>
<td>0.03805</td>
</tr>
<tr>
<td>512</td>
<td>0.5</td>
<td>1024</td>
<td>0.038</td>
</tr>
<tr>
<td>512</td>
<td>0.5</td>
<td>4096</td>
<td>0.0761</td>
</tr>
<tr>
<td>1024</td>
<td>1</td>
<td>2048</td>
<td>0.076</td>
</tr>
<tr>
<td>1024</td>
<td>1</td>
<td>8192</td>
<td>0.1522</td>
</tr>
<tr>
<td>2048</td>
<td>2</td>
<td>4096</td>
<td>0.152</td>
</tr>
<tr>
<td>2048</td>
<td>2</td>
<td>16384</td>
<td>0.3044</td>
</tr>
<tr>
<td>4096</td>
<td>4</td>
<td>8192</td>
<td>0.304</td>
</tr>
<tr>
<td>4096</td>
<td>4</td>
<td>30720</td>
<td>0.5834</td>
</tr>
</tbody>
</table>
<p>We also tested the same 10 configurations with iperf's <code>--reverse</code> option to verify symmetric network performance, resulting in a total of 20 scenarios.</p>
<p>All test scenarios are configured to omit the first 10 seconds of measurement to skip past <a href="https://en.wikipedia.org/wiki/TCP_congestion_control#Slow_start">TCP slow-start</a> window and also (initial) peak performance. Actual measurements are performed for 60 seconds using two parallel TCP streams. We are primarily interested in sustained network performance, ignoring shot throughput peaks.</p>
<h2 id="results">Results</h2>
<p>The results were quite interesting and quite a bit surprising to us. But without further ado, let's take a look at the results.</p>
<p>Here is a plot of all tested configurations with the average bandwidth measured over 60 seconds. "Fargate out" refers to traffic being send from AWS Fargate (iPerf server) to our EC2 instance (iPerf client) done via iPerf's <code>--reverse</code> flag. "Fargate in" is the other direction, without <code>--reverse</code>, sending data from EC2 to the Fargate container.</p>
<p><img src="/blog/aws-fargate-network-performance/bandwidth-18b948a1.svg" alt="AWS Fargate Network Performance" /></p>
<p>Here is the same result data:</p>
<table>
<thead>
<tr>
<th>CPU Units</th>
<th>vCPU</th>
<th>Memory (MB)</th>
<th>Outgoing (MBit/s)</th>
<th>Incoming (MBit/s)</th>
</tr>
</thead>
<tbody>
<tr>
<td>256</td>
<td>0.25</td>
<td>512</td>
<td>268</td>
<td>136</td>
</tr>
<tr>
<td>256</td>
<td>0.25</td>
<td>2048</td>
<td>268</td>
<td>136</td>
</tr>
<tr>
<td>512</td>
<td>0.5</td>
<td>1024</td>
<td>268</td>
<td>136</td>
</tr>
<tr>
<td>512</td>
<td>0.5</td>
<td>4096</td>
<td>625</td>
<td>319</td>
</tr>
<tr>
<td>1024</td>
<td>1</td>
<td>2048</td>
<td>268</td>
<td>137</td>
</tr>
<tr>
<td>1024</td>
<td>1</td>
<td>8192</td>
<td>455</td>
<td>454</td>
</tr>
<tr>
<td>2048</td>
<td>2</td>
<td>4096</td>
<td>3404</td>
<td>319</td>
</tr>
<tr>
<td>2048</td>
<td>2</td>
<td>16384</td>
<td>637</td>
<td>636</td>
</tr>
<tr>
<td>4096</td>
<td>4</td>
<td>8192</td>
<td>2753</td>
<td>456</td>
</tr>
<tr>
<td>4096</td>
<td>4</td>
<td>30720</td>
<td>637</td>
<td>636</td>
</tr>
</tbody>
</table>
<p>Although for most measurements the bandwidth is not super high, the measured bandwidth is very stable over time. For all tests there was an initial peak, but the sustained bandwidth was very solid with little to no variation. This is very nice as this makes it quite predictable.</p>
<p>In addition to the network throughput we also took a look at the containers CPU utilization (normalized to 100%). This draws a very interesting picture which could at least explain why network performance is not symmetric (in higher then out in all cases):</p>
<p><img src="/blog/aws-fargate-network-performance/cpu-714819a4.svg" alt="AWS Fargate Network Performance" /></p>
<h2 id="conclusions">Conclusions</h2>
<p>We are not surprised by the relatively low throughput of the smaller container configurations (this is basically very similar to EC2 network performance). Two aspects strikes us as very odd:</p>
<ol>
<li><strong>Network throughput on AWS Fargate does not seem to be symmetric.</strong> Often there is "just" a 2x difference, but it goes up to over 10x.</li>
<li><strong>Ingress performance of 2048 CPU / 4 GB memory configuration is way off at 3Gbits</strong> (same with 4096 CPU / 8 GB memory). And the throughput goes down again when we increased CPU/memory for the container.</li>
</ol>
<p>This is <strong>really</strong> strange observation. The biggest container you can get with 4096 CPU Units and 30GB memory has roughly the same network performance as the <strong>50% cheaper 2048 CPU/16GB option</strong>.</p>
<p>The initial assumption was that network performance correlates with allocated resources like with EC2. At first this seems to be the case, until the 2048 CPU Unit tier where things got really odd. Compared to pricing and network performance for EC2 the price per MBit is actually not that bad (at least for ingress).</p>
<p>For a follow-up one might look into more details like measuring all memory tiers (1GB increments currently), vary the amount of TCP streams in iPerf, etc. There might be configuration tweaking possible to increase network performance overall, but that does not explain the immensely (positive) performance outliers.</p>
<p>Do you have questions or remarks regarding AWS Fargate or other performance topics? Just <a href="/support">drop us a line</a> :)</p>
<h2 id="more-details">More Details</h2>
<p>In case you are interested in a bit more detail of the test setup, here you go!</p>
<p>We used an unofficial <a href="https://github.com/jpignata/fargate"><code>fargate</code> CLI tool</a> by John Pignata, which has a really nice and simple interface. We had to <a href="https://github.com/jpignata/fargate/pull/54">patch the tool</a> to support the newly available regions, including <code>eu-west-1</code> which we wanted to use for testing.</p>
<p>We used the following <code>Dockerfile</code> to set up the iPerf3 servers running in Fargate:</p>
<div class="highlight"><pre class="highlight docker"><code><span class="k">FROM</span><span class="s"> alpine:latest</span>
<span class="k">RUN </span>apk <span class="nt">--update</span> add iperf3 <span class="se">\
</span> <span class="o">&&</span> <span class="nb">rm</span> <span class="nt">-rf</span> /var/cache/apk/<span class="k">*</span>
<span class="k">ENTRYPOINT</span><span class="s"> ["iperf3"]</span>
<span class="k">CMD</span><span class="s"> ["--server", "--json", "--verbose"]</span>
</code></pre></div>
<p>Creating a AWS Fargate task based on this <code>Dockerfile</code> with our desired parameters was very straight forward. This command will build the container image, upload it to your private Docker registry and launch a Fargate task with the given resource configuration:</p>
<div class="highlight"><pre class="highlight console"><code><span class="gp">$</span><span class="w"> </span>bin/fargate task run iperf-test <span class="nt">--region</span><span class="o">=</span>eu-west-1 <span class="nt">--cpu</span> <span class="nv">$CPU_UNITS</span> <span class="nt">--memory</span> <span class="nv">$MEMORY_UNITS</span>
</code></pre></div>
<p>When the container is ready, you can start iPerf from the beefy EC2 instance like this:</p>
<div class="highlight"><pre class="highlight console"><code><span class="gp">$</span><span class="w"> </span>iperf3 <span class="se">\</span>
<span class="nt">--time</span> 60 <span class="se">\</span>
<span class="nt">--omit</span> 10 <span class="se">\</span>
<span class="nt">--parallel</span> 2 <span class="se">\</span>
<span class="nt">--reverse</span> <span class="se">\</span>
<span class="nt">--verbose</span> <span class="se">\</span>
<span class="nt">--json</span> <span class="se">\</span>
<span class="nt">--get-server-output</span> <span class="se">\</span>
<span class="nt">--client</span> <span class="nv">$IPERF_TARGET_CONTAINER_IP</span> <span class="o">></span> measurements/c<span class="nv">$CPU_UNITS</span><span class="nt">-m</span><span class="nv">$MEMORY_UNITS</span>.json
</code></pre></div>
<p>After each test, we wait 20 seconds before starting the next experiment. This is go give both systems a bit to cool down.</p>
<p>The iPerf3 client (EC2 instance) and all Fargate containers were started in the same VPC using the same Security Group within <code>eu-west-1</code> region. No other user processes were running on the EC2 instance, besides network monitoring tools for sanity checking.</p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Determine Your Performance Impact by Meltdown & Spectre
https://stormforger.com/blog/determine-performance-impact-by-meltdown-and-spectre/
2018-01-04T23:00:00+00:00
2018-01-04T23:00:00+00:00
Sebastian Cohnen
<p><a href="/blog/determine-performance-impact-by-meltdown-and-spectre"><img src="/blog/determine-performance-impact-by-meltdown-and-spectre/inline-left-meltdown_spectre_logo-5ff22a04.png" alt="Meltdown & Spectre"></a></p>
<p>Major security related issues were disclosed just a few days ago affecting CPUs across all vendors and architectures, including Intel and AMD. These vulnerabilities have become known as Meltdown and Spectre and are very severe.</p>
<p>Mitigations have been released for many systems and environment, but you should check if you are fully patched up <strong>before continue reading this article</strong>! The problem effects all systems, regardless of virtualization or not.</p>
<p></p>
<p><a href="/blog/determine-performance-impact-by-meltdown-and-spectre"><img src="/blog/determine-performance-impact-by-meltdown-and-spectre/inline-left-meltdown_spectre_logo-5ff22a04.png" alt="Meltdown & Spectre" /></a></p>
<p>Major security related issues were disclosed just a few days ago affecting CPUs across all vendors and architectures, including Intel and AMD. These vulnerabilities have become known as Meltdown and Spectre and are very severe.</p>
<p>Mitigations have been released for many systems and environment, but you should check if you are fully patched up <strong>before continue reading this article</strong>! The problem effects all systems, regardless of virtualization or not.</p>
<p></p>
<p>There is still a lot of speculation on possible performance impacts caused by mitigations of <a href="https://meltdownattack.com/">Meltdown and Spectre</a>. While AWS states that they <a href="https://aws.amazon.com/security/security-bulletins/AWS-2018-013/"><em>"have not observed meaningful performance impact for the overwhelming majority of EC2 workloads"</em></a>, other reports indicate quite an impact (e.g. reported on the PostreSQL mailing list).</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">PostgreSQL SELECT 1 with the KPTI workaround for Intel CPU vulnerability <a href="https://t.co/N9gSvML2Fo">https://t.co/N9gSvML2Fo</a><br /><br />Best case: 17% slowdown<br />Worst case: 23%</p>— The Register (@TheRegister) <a href="https://twitter.com/TheRegister/status/948342806367518720?ref_src=twsrc%5Etfw">January 2, 2018</a></blockquote>
<h2 id="performance-impacts-are-workload-related">Performance impacts are workload related</h2>
<p>The security problem is related to the isolation of user and kernel processes so mitigations try to attack there. The performance degradation happens because the user process has to ask the kernel for many tasks for example IO-related operations like disk access or networking (<a href="https://en.wikipedia.org/wiki/System_call">system calls</a>). This probably explains why a pure database workload is more impacted than a typical web application that does much more non-IO business logic.</p>
<h2 id="determine-how-your-performance-is-impacted">Determine how your performance is impacted</h2>
<p>While not protecting against these issue is not an option, you might want to know what the performance and thus your business impact is.</p>
<p>Ideally you already have knowledge of the performance characteristics of your system. In this case you can compare pre and post patch behavior and look for potential issues, like increased resource utilization or latencies. Hint: If you are looking for a nice overview for Linux performance analysis, check out <a href="https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55">Linux Performance Analysis in 60,000 Milliseconds</a> by Netflix.</p>
<p>In any case, the only way to reliably determine that your business won't be affected by the yet unknown performance penalty is to do <a href="/blog/types-of-performance-testing/">performance tests</a>.</p>
<h3 id="how-to-get-started">How to get started?</h3>
<p>Performance testing is hard and you need to invest some time. However to get the first impression just create a test case for your web application or HTTP API.</p>
<p class="text-center">
<a href="https://app.stormforger.com" class="Button-cta">
<strong>Start to run tests with 300 Clients for free!</strong>
</a>
</p>
<p>If you have any questions just <a href="/support">drop a line</a> – we're happy to help!</p>
<h2 id="read-further">Read Further</h2>
<ul>
<li><a href="https://googleprojectzero.blogspot.de/2018/01/reading-privileged-memory-with-side.html">Google Project Zero: Reading privileged memory with a side-channel</a></li>
<li><a href="https://newsroom.intel.com/news/intel-responds-to-security-research-findings/">Official statement by Intel Corporation</a></li>
<li><a href="https://danielmiessler.com/blog/simple-explanation-difference-meltdown-spectre/">A Simple Explanation of the Differences Between Meltdown and Spectre</a></li>
<li><a href="https://www.theregister.co.uk/2018/01/02/intel_cpu_design_flaw/">Kernel-memory-leaking Intel processor design flaw forces Linux, Windows redesign</a></li>
</ul>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<hr />
<h5 id="disclaimer">Disclaimer</h5>
<p>Performance testing is not security or penetration testing!</p>
<p>The field of operation of StormForger is performance testing and <strong>not any type of security testing or security auditing</strong>, please refer to experts like cure53 for this.</p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Advices on performance from AWS' "Well-Architected Framework"
https://stormforger.com/blog/advices-on-performance-from-aws-well-architected-framework/
2017-03-21T23:00:00+00:00
2017-03-21T23:00:00+00:00
Denise Schynol
<p><a href="/blog/advices-on-performance-from-aws-well-architected-framework/"><img src="/blog/advices-on-performance-from-aws-well-architected-framework/inline-left-pillars-cloud-583732d5.jpg" alt="Cloud Pillars"></a></p>
<p>“AWS Well-Architected Framework” is a guideline for review and improvement on
cloud-based architectures. Amazon Web Services recommends “general design
principles” and describes best practice examples.</p>
<p>In this blog posting we sum up some advices on performance testing.</p>
<p></p>
<p><a href="/blog/advices-on-performance-from-aws-well-architected-framework/"><img src="/blog/advices-on-performance-from-aws-well-architected-framework/inline-left-pillars-cloud-583732d5.jpg" alt="Cloud Pillars" /></a></p>
<p>“AWS Well-Architected Framework” is a guideline for review and improvement on
cloud-based architectures. Amazon Web Services recommends “general design
principles” and describes best practice examples.</p>
<p>In this blog posting we sum up some advices on performance testing.</p>
<p></p>
<p>First let's start with an explanation of the general design principles suggested
by AWS followed by a closer look at some of the so-called “five pillars” of the
<a href="https://aws.amazon.com/de/blogs/aws/well-architected-working-backward-to-play-it-forward/">AWS Well-Architected
Framework</a> (p. 2).
Mentioning “five pillars”: In a former blog post from October 2015 the
author <a href="https://aws.amazon.com/de/blogs/aws/are-you-well-architected/">Jeff Barr counts only four
pillars</a>. So, the
here not mentioned pillar “operational excellence” seems to have gained more
importance over the time.</p>
<h2 id="security-reliability-performance-efficiency-cost-optimization--operational-excellence--five-pillars-of-the-aws-well-architected-framework">Security, Reliability, Performance Efficiency, Cost Optimization & Operational Excellence – Five Pillars of the AWS Well-Architected Framework</h2>
<p>In their definition of a Well-Architected Framework AWS speak about how they
help customers to make “architectural trade offs as your designs evolve” and
about the effects and learnings on performance after deploying into live
environment (p. 1). Based on this they created the AWS Well-Architected
Framework – a “set of questions you can use to evaluate how well an architecture
is aligned to AWS best practices” (p. 2). Five topics (“pillars”) are defined:
<strong>security</strong>, <strong>reliability</strong>, <strong>performance efficiency</strong>, <strong>cost
optimization</strong>, and <strong>operational excellence<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup></strong> (p. 2). AWS describes that
customers have to “make trade-offs between pillars based upon your business
context” (p. 2) when they improve their architecture. For example</p>
<blockquote>
<p>the optimization of performance efficiency has crucial effects on your cost
efficiency.</p>
</blockquote>
<p>So, it's recommended to do performance analysis and optimization of performance
bottlenecks on a regular basis to gain effect on e.g. cost efficiency.</p>
<p><img src="/blog/advices-on-performance-from-aws-well-architected-framework/5-pillars-aws-well-architected-framework-9f598f65.svg" alt="5 Pillars of the AWS Well Architected
Framework" /></p>
<h2 id="general-design-principles">General Design Principles</h2>
<p>The definition of the AWS Well-Architected
Framework is followed by a bunch of general design principles “to facilitate
good design in the cloud” (p. 2). We pitch on three points concerning load and
performance testing. The first one deals with testing systems at production
scale:</p>
<blockquote>
<p>In the cloud, <strong>you can create a production-scale test environment on
demand</strong>, complete your testing, and then decommission the resources. Because
you only pay for the test environment when it is running, you can simulate
your live environment for a fraction of the cost of testing on premises. (p.
3)</p>
</blockquote>
<p>AWS make it easy and affordable for you to test in the cloud. You can run your
tests without a huge increase of costs.</p>
<p>Another point is AWS’s suggestion to do improvements through “game days”:</p>
<blockquote>
<p>Test how your architecture and processes perform by regularly scheduling game
days to simulate events in production. This will help you understand where
improvements can be made and can help develop organizational experience in
dealing with events. (p. 3)</p>
</blockquote>
<p>An event might be the transmission of a newsletter, any other promotion or a
business related event like the launch of a new website or product. In this case
you can make use of spike testing which is comparable to a load or stress test.
If you are curious about the different types of testing check out our <a href="/blog/types-of-performance-testing/">blog post about types of performance testing</a>. One more appreciable
point deals with allowing for evolutionary architectures:</p>
<blockquote>
<p>(…) In the cloud, the capability to automate and test on demand lowers the
risk of impact from design changes. This allows systems to evolve over time so
that businesses can take advantage of innovations as a standard practice. (p.
3)</p>
</blockquote>
<p>It is common to introduce new features or a new technical product to your
architecture, test it in an automated, cloud based environment and learn about
the performance impact of the introduction. Especially test automation is one of
StormForger’s major topics as we deliver all the tools you need to do continuous
load testing in the cloud.</p>
<h2 id="operational-excellence">Operational Excellence</h2>
<p>Another pillar that is interesting when you consider
load testing:</p>
<blockquote>
<p>"The Operational Excellence pillar includes operational practices and procedures used to manage production workloads. This includes how planned changes are executed, as well as responses to unexpected operational events.” (p. 33).</p>
</blockquote>
<p>As mentioned before in this article spike testing or stress testing
are powerful ways to analyze the effects that planned or unplanned events may
cause. AWS advise their customers to document, test, and regularly review “[all]
processes and procedures of operational excellence” (p. 33).</p>
<h2 id="reliability">Reliability</h2>
<p>As part of the reliability pillar the AWS white paper mentions
“Change Management” and asks “How does your system adapt to changes in demand?”
(p. 49). The listing for best practices contains two suggestions: Automated
scaling and load testing. While Amazon provides a multitude of automatically
scalable services (like Amazon S3, Amazon CloudFront or AWS Elastic Beanstalk)
the choice of a load and performance testing tool remains your own decision.
StormForger is a both low barrier and comprehensive tool for continuous
performance testing with additional support for agile and DevOps teams. With our
JavaScript DSL to describe your test cases and our comprehensive reporting we
provide you everything to get started early and enable you to test as often as
needed to fulfill your requirements and learn about your systems behavior.
<a href="/how-to-start/">Learn more about how StormForger works</a>.</p>
<h2 id="performance">Performance</h2>
<p>The white paper actually speaks about the “Performance Efficiency”
pillar which is a bit redundant, as performance actually means resource
efficiency.</p>
<p>On page 20 the white paper asks the question: “How do you select the best
performing architecture?”. AWS offers various kind of support and services to
help answering these questions and help you to establish a good working
architecture. But there is one thing to keep in mind: “data obtained through
(…) load testing will be required to optimize your architecture” (p. 20). Of
course, we totally agree: You just can’t avoid load and performance testing!</p>
<p>Depending on your workload various resource types and sizes can differ to fit
your performance requirements. With a defined test case it is easy to rerun the
same test case against different infrastructure setups and gather needed data
and learnings.</p>
<blockquote>
<p>Deploy the latest version of your system on AWS using different resource types
and sizes, use monitoring to capture performance metrics, and then make a
selection based on a calculation of performance/cost. (p. 53)</p>
</blockquote>
<p>What AWS is suggesting here is what we call Configuration Testing. Configuration
Testing does not only cover your software configuration, but also your entire
environment. Rerunning of performance tests is part of a review routine to
“ensure that you continue to have the most appropriate resource type” (p. 56).
Approaches change, new technologies develop – use these progressions to refine
your architecture and improve its performance.</p>
<p>Load testing is also necessary in terms of using trade-offs to improve your
performance<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>: “Trade-offs can increase the complexity of your architecture,
and require load testing to ensure that a measurable benefit is obtained.” (p.
25)</p>
<p>It all boils down to: Performance Testing is not a one hit wonder. Continuity is
key to success and you should be setup for doing performance tests whenever
needed. In general you should start to test early and do it on a regular basis.
If possible integrate performance analysis in your development and testing
processes, ideally alongside your functional tests in your continuous
integration systems.</p>
<h2 id="conclusion">Conclusion</h2>
<p>AWS released a fine and valuable white paper worth reading, a
practical guideline to design and steadily improve a well-architected framework.
They offer a lot of useful services you might check out. Most of the targeted
problems we see in the wild on a regular basis and speak out the same
recommendations.</p>
<p>We always recommend to start early with small, easy and understandable test case
scenarios. It is easy to setup your first load test with StormForger for free.
<a href="https://app.stormforger.com/signup">Sign up</a>, <a href="https://docs.stormforger.com/guides/onboarding/">learn more in our documentation</a> and get in touch
with us to <a href="mailto:support@stormforger.com">get a personal on boarding</a>.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="https://aws.amazon.com/de/blogs/aws/are-you-well-architected/">In a former blog post from October 2015 the author Jeff Barr counts only four pillars</a>. So, the here not mentioned pillar “operational excellence” seems to have gained more importance over the time. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Focused on space-time trade-offs for performance efficiency. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Performance Testing – Pre and Post Cloud?
https://stormforger.com/blog/performance-testing-pre-and-post-cloud/
2016-08-19T10:00:00+00:00
2016-08-19T10:00:00+00:00
Sebastian Cohnen
<p><a href="/blog/performance-testing-pre-and-post-cloud/"><img src="/blog/performance-testing-pre-and-post-cloud/inline-left-sebastian-cohnen-performance-testing-cloud-architecture-2f1b52e5.jpg" alt="Cloud"></a></p>
<p>The last part of our blog posting series "Why Load and Performance Testing in the Cloud?" will give an answer to the question "what is the difference in all this to pre-cloud times?". The article is based upon a talk I gave at the <a href="https://aws.amazon.com/de/start-ups/loft/de-loft/">AWS PopUp Loft</a> in Berlin.</p>
<p></p>
<p><a href="/blog/performance-testing-pre-and-post-cloud/"><img src="/blog/performance-testing-pre-and-post-cloud/inline-left-sebastian-cohnen-performance-testing-cloud-architecture-2f1b52e5.jpg" alt="Cloud" /></a></p>
<p>The last part of our blog posting series "Why Load and Performance Testing in the Cloud?" will give an answer to the question "what is the difference in all this to pre-cloud times?". The article is based upon a talk I gave at the <a href="https://aws.amazon.com/de/start-ups/loft/de-loft/">AWS PopUp Loft</a> in Berlin.</p>
<p></p>
<h2 id="difference-to-pre-cloud">Difference to pre-cloud?</h2>
<p>After my overview on the <a href="/blog/types-of-performance-testing/">types of testing</a> I'd like to raise the question: Is this actually any different from what it used to be pre-cloud?</p>
<p>I would clearly have to answer: Yes and no. The testing needs and methods have not changed a lot. The needs have maybe increased a bit because our environments are tending to be more complex overall.</p>
<p>The main difference is though that multiple, scalable, performance testing environments used to be very expensive and very hard to manage. Not many could actually do that for real projects.</p>
<p>This is where the automation capability of the cloud can really make a difference. If you have your entire infrastructure, services, servers and code automated, spinning up testing environments suddenly becomes viable. You can create entire infrastructures and environments fully automated, run a series of performance tests, gather all relevant data and shut it down again within a couple of hours and for very little money.</p>
<p>One important thing to keep in mind, though, when aiming for such a testing environment: Do not forget about state. With state I am referring to databases, caches that are warm, etc. Having a reproducible environment, including test data is one of the bigger challenges. Frankly, I haven't seen a good and sound solution for that yet.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In a nut shell, you can state:</p>
<ul>
<li>Scaling resources is not scaling application</li>
<li>Understanding is very important to make your system efficient and scalable</li>
<li>Complexity is still around and will become apparent again in non-trivial ways when it comes to performance</li>
<li>You should utilize the cloud to make testing simpler</li>
</ul>
<p>Infrastructure and software architecture in general is nowadays ever changing and evolving. Getting solid data on how your system is performing is not only relevant for your application in particular but an important piece when it comes to do Continuous Architecture. Your architecture is ever evolving to run tests and validate it!</p>
<p>Want to know more? Switch to the previous articles:</p>
<ul>
<li><a href="/blog/why-load-performance-testing-the-cloud/">Why Load and Performance Testing in the Cloud?</a></li>
<li><a href="/blog/types-of-performance-testing/">Types of Performance Testing</a></li>
</ul>
<h2 id="slides">Slides</h2>
<script async="" class="speakerdeck-embed" data-id="1bf9d5cc7fff40c7a9f48feb428c343a" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Types of Performance Testing
https://stormforger.com/blog/types-of-performance-testing/
2016-07-08T15:00:00+00:00
2016-07-08T15:00:00+00:00
Sebastian Cohnen
<p><a href="/blog/types-of-performance-testing"><img src="/blog/types-of-performance-testing/inline-right-sebastian-cohnen-testing-types-26e39795.jpg" alt="Cloud"></a></p>
<p>The second part of our blog posting series shows an overview on the different types of performance testing. Learn more about load testing, scalability testing, stress, spike and soak testing, configuration testing as well as availability and resilience testing. The article is based upon a talk I gave at the <a href="https://aws.amazon.com/de/start-ups/loft/de-loft/">AWS PopUp Loft</a>, <a href="https://devopsconference.de/">DevOpsCon 2016</a> and other occasions.</p>
<p></p>
<p><a href="/blog/types-of-performance-testing"><img src="/blog/types-of-performance-testing/inline-right-sebastian-cohnen-testing-types-26e39795.jpg" alt="Cloud" /></a></p>
<p>The second part of our blog posting series shows an overview on the different types of performance testing. Learn more about load testing, scalability testing, stress, spike and soak testing, configuration testing as well as availability and resilience testing. The article is based upon a talk I gave at the <a href="https://aws.amazon.com/de/start-ups/loft/de-loft/">AWS PopUp Loft</a>, <a href="https://devopsconference.de/">DevOpsCon 2016</a> and other occasions.</p>
<p></p>
<h2 id="types-of-testing">Types of Testing</h2>
<p>In the <a href="/blog/why-load-performance-testing-the-cloud/">last blog post</a> I wrote about performance, scalability and the importance of performance testing in the cloud era. So, now, let us take a closer look at the performance testing methods.</p>
<p>It is needless to say that some of the types of testing are not really special when it comes to a cloud environment, but others are especially interesting.</p>
<h3 id="load-testing">Load Testing</h3>
<p>Load testing is sort of the simplest form of performance testing. You induce a normal or expected workload to a system under test and observe it. You can use load tests to determine general system behavior, latency and throughput. In general load tests are used to verify your quality criteria.</p>
<h3 id="stress-testing">Stress Testing</h3>
<p>Stress testing is basically a load test, but we are applying a higher-than-expected workload and see how the system behaves under serious stress and when exceeding the design limits. You want to learn when your system breaks and how it starts to fail when being in a serious traffic situation.</p>
<p>A typical approach is to steadily increase the load to see where the system under test begins to violate its non-functional requirements. You can use this "tipping point" to describe the capacity of the given system, like "we can handle 1000 concurrent users per application server before we start to violate our quality requirements".</p>
<h3 id="scalability-testing">Scalability Testing</h3>
<p>With scalability testing you are changing the perspective to answer the question: How effective can I grow? You can run a series of stress tests and gather data on how effective you really are.</p>
<p>Using stress tests in a series where you steadily increase the system's ressources, you can easily tell if your system can translate this into additional capacity.</p>
<p><img src="/blog/types-of-performance-testing/scalability-analysis-fb53aecd.png" alt="Cloud" /></p>
<p>Knowing how well and how far your system will scale by adding resources, you can now make an informed decision: You might not need to do anything right now or you need to take action (e.g. remove bottlenecks). Or you simply add more resources to your problem to mitigate a scaling issue for the time being. Suffice to say, that this is a basic requirement for capacity planning and to do cost estimation.</p>
<h3 id="spike-testing">Spike Testing</h3>
<p>Spike testing can be used to determine how well your system can cope with sudden traffic spikes. It is comparable to a load or stress test, but modeled as a sudden burst of traffic. It can be a good preparation for a planned marketing campaign or an unplanned event like being featured on Reddit or Hacker News. Spike Testing can tell you if you are making good use of the elasticity of the cloud when faced with these kinds of events.</p>
<h3 id="soak-testing">Soak Testing</h3>
<p>Soak testing, again, is basically a load test where you hold the load over longer periods of time to look for long term effects, like memory leaks, disk space filling up, etc. The duration of a soak test depends on your situtation. Usually a soak test runs for several hours.</p>
<h3 id="configuration-testing">Configuration Testing</h3>
<p>While load, stress, spike and soak testing are not particular special when it comes to the cloud, the next testing method is one of the most interesting ones: Configuration Testing.</p>
<p>The perspective shifts now to looking at the changes in performance if the configuration is modified. The change might be positive, but also negative in case you want to optimize for costs (remember: Performance is resource usage per unit of work, money is also a resource). The point is that you know and can quantize the change.</p>
<p>Configuration can be almost anything here: your environment, services that you are using, dependencies of your software — all can be seen as configuration.</p>
<p>Configuration testing is, for obvious reasons, a very important tool in order to learn about the impact of a system's environment to its performance. It is always a series of test runs where you compare and analyze the impact of multiple configurations.</p>
<h4 id="configuration-options">Configuration Options</h4>
<p>Now, it starts to get really interesting: We have a lot of configuration options to choose from when it comes to the cloud. These options include:</p>
<ul>
<li>Instance types selection (for EC2, RDS, EC, ES)</li>
<li>Auto scaling configuration (Scaling Policies, Instance Launch Time, Scaling Lifecycle)</li>
<li>Throughput provisioning (EBS IOPS, DynamoDB throughput, Kinesis bandwidth)</li>
<li>Service usage optimization (ELB pre-warming, Index Usages)</li>
</ul>
<p>Finally, you have approached the parts that you have more or less fully under your control, like the operating system, network stack and other kernel settings, software/web server/app server configuration, dependencies, etc. etc.</p>
<p>To emphasize this once more: The aim is to look for change in performance. So, you can use configuration testing to optimize for costs as well, while you know what kind of trade-off you are taking.</p>
<h3 id="availability--resilience-testing">Availability & Resilience Testing</h3>
<p>The last type of performance testing I will introduce is availability & resilience testing. I will only briefly touch this, because this is probably enough for one single blog article. The idea is to look at certain processes and behavior under load and check if you have covered this.</p>
<ul>
<li>What about deployments? Possible even with DB migrations under load?</li>
<li>Ever changing, ever evolving infrastructure? Automatic scaling environments?</li>
<li>Failure scenarios and failover mechanisms?</li>
</ul>
<h4 id="principles-of-chaos-engineering">Principles of Chaos Engineering</h4>
<p>The idea to give this its own testing category is inspired by <a href="http://principlesofchaos.org">Principles of Chaos Engineering</a>. You most probably have checked for those points in some form of manual or automated functional testing, but did you also check for them under load? While you aren't all Netflix and can't do that in production, at least think about to do that in a test environment using artificial traffic.</p>
<p>Want to know more? Take a look at the previous article:</p>
<ul>
<li><a href="/blog/why-load-performance-testing-the-cloud/">Why Load and Performance Testing in the Cloud?</a></li>
</ul>
<p>And here you find the third part of our blog posting series:</p>
<ul>
<li><a href="/blog/performance-testing-pre-and-post-cloud/">Performance Testing – Pre and Post Cloud?</a></li>
</ul>
<h2 id="slides">Slides</h2>
<p>Find the slides of the DevOpsCon Berlin 2016 talk [DE] here:</p>
<script async="" class="speakerdeck-embed" data-id="8ca1161b9cc24ece87d5c29809c0a659" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Why Load & Performance Testing in the Cloud?
https://stormforger.com/blog/why-load-performance-testing-the-cloud/
2015-11-14T09:00:00+00:00
2015-11-14T09:00:00+00:00
Sebastian Cohnen
<!-- [![Sebastian Cohnen](/blog/2015-11-14-why-load-performance-testing-the-cloud/Sebastian-Cohnen_Why-load-and-performance-testing-the-cloud.jpg)]() -->
<p><a href="/blog/why-load-performance-testing-the-cloud"><img src="/blog/why-load-performance-testing-the-cloud/inline-left-header-stormforger-load-test-api-http-8e3841a8.jpg" alt="Cloud"></a></p>
<p>At the <a href="https://aws.amazon.com/de/start-ups/loft/de-loft/">AWS PopUp Loft</a> in Berlin I gave a talk on why load and performance testing is still (or especially) relevant in the era of cloud infrastructure. I expanded the talk for AWS Summit Berlin 2016 and again at <a href="https://devopsconference.de">DevOpsCon 2016</a>. Here in our blog you will find it in three crunchy bites, beginning with performance and scalability and the importance of performance testing in the cloud era. I will give an overview on the different types of testing and explain the differences between pre and post cloud times.</p>
<p></p>
<!-- [![Sebastian Cohnen](/blog/2015-11-14-why-load-performance-testing-the-cloud/Sebastian-Cohnen_Why-load-and-performance-testing-the-cloud.jpg)]() -->
<p><a href="/blog/why-load-performance-testing-the-cloud"><img src="/blog/why-load-performance-testing-the-cloud/inline-left-header-stormforger-load-test-api-http-8e3841a8.jpg" alt="Cloud" /></a></p>
<p>At the <a href="https://aws.amazon.com/de/start-ups/loft/de-loft/">AWS PopUp Loft</a> in Berlin I gave a talk on why load and performance testing is still (or especially) relevant in the era of cloud infrastructure. I expanded the talk for AWS Summit Berlin 2016 and again at <a href="https://devopsconference.de">DevOpsCon 2016</a>. Here in our blog you will find it in three crunchy bites, beginning with performance and scalability and the importance of performance testing in the cloud era. I will give an overview on the different types of testing and explain the differences between pre and post cloud times.</p>
<p></p>
<h2 id="abstract">Abstract</h2>
<p>The Cloud™ is infinite and scalable. Period.
Why is it important to test for performance and scalability characteristics of a cloud-based system? Won't AWS scale for me as long as I can afford it to?
Yes, but… AWS only operates and scales resources. They won't automatically make your system fast, stable and — more importantly — scalable. Performance testing is crucial to understand your system and architecture design and your cloud hosting environment.</p>
<h2 id="performance">Performance</h2>
<p>I state: Load and performance testing is still a thing in the cloud. To prove that, I will briefly cover some basics, starting with performance.</p>
<p>Performance is often used in a not very well defined way. The most abstract description of performance I can think of is the ability of a system to fulfill a task within defined dimensions. It is a measurement for efficiency! The most often used dimension is time and a unit of work could be a request or transaction. This would then describe the performance of a system in terms of request latency for example.</p>
<p>You can extend this definition to describe the efficiency of a system or server to get the performance statement "1 instance can handle 250 rps at a 99th percentile of 250ms"
.</p>
<h2 id="scalability">Scalability</h2>
<p>Next up: Scalability. That is a term often used next to or even interchangeably with performance even though they are not as much related as many think. While performance is a measurement for efficiency, scalability on the other hand describes how effectively you can translate additional resources into additional capacity.</p>
<p>This distinction is crucial because it can help you want to design a scalable system which you don't want to fall for performance optimizations when you are actually aiming for scalability.</p>
<p>An example for an ideally scaling system would be: If you tenfold the resources and have tenfold the capacity and you can hold the linear scaling, then you have done a terrific job!</p>
<p>But you have to keep in mind, that one does not imply the other. You can have an inefficient system that scales very well. Think of a video encoding service that can potentially be scaled very well simply by adding servers. But you can also have a highly efficient, optimized system that might not be scalable at all, maybe because the communication overhead between multiple nodes would make that impossible.</p>
<p><a href="https://twitter.com/jboner">Jonas Bonér</a> (Founder & CTO, Lightbend) has a nice <a href="http://www.slideshare.net/jboner/scalability-availability-stability-patterns/15-How_do_I_know_if">presentation</a> on that topic and he asks two good questions:</p>
<blockquote>
<p>How do I know if I have a <strong>performance problem</strong>? If your system is slow for a single user! <br />
How do I know if I have a <strong>scalability problem</strong>? If your system is fast for a single user, but slow under heavy load!</p>
</blockquote>
<h2 id="load-and-performance-testing">Load and Performance Testing</h2>
<p>So what is performance testing? The Wikipedia article on software performance testing gives a nice definition:</p>
<blockquote>
<p>In software engineering, performance testing is a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload. — Wikipedia on <a href="https://en.wikipedia.org/wiki/Software_performance_testing">Software Performance Testing</a></p>
</blockquote>
<p>In other words: Performance testing is a family of non-functional testing methods, which all:</p>
<ul>
<li>induce a well defined workload</li>
<li>in order to observe the system under test</li>
<li>in order to verify and understand its performance characteristics</li>
</ul>
<p>There are a bunch of testing methods that can be seen as types of performance testing. The main difference is that they have different goals and testing perspectives. I will have a look at all the types in part 2.</p>
<h2 id="so-why-performance-testing-in-the-cloud">So Why Performance Testing In The Cloud?</h2>
<p>I think it is set that the cloud is here to stay. Some of the most interesting key points that the cloud offers us are: IaaS, PaaS and other-things-as-a-Service. Combined with APIs, a high degree of automation and resources that are available on-demand we get a cost effective and scalable environment that was not available in this dimension before.</p>
<p>It is obvious that you should care about performance. But some might ask: Why care about performance testing in the cloud? You can always take out your credit card and add more resources to the problem. But scaling resources is not the same as scaling your application. You have to design your system very carefully to make that statement true as there are many pitfalls to avoid.</p>
<p>A simple — possibly even stupid — example would be that you have your ELB set up (which scales automatically), and your application servers are scaled automatically by the AWS Auto Scaler. But you forgot to take care of scaling your persistence layer as well.</p>
<p><a href=""><img src="/blog/why-load-performance-testing-the-cloud/cloud-scaling-elb-asg-rds-19e0d128.png" alt="Cloud" /></a></p>
<p>The reason why you still have to think a lot even in a managed environment like AWS is understanding.</p>
<h2 id="know-your-application-architecture">Know your application architecture!</h2>
<p>You need to understand the performance related characteristics and implications of your application architecture — and it doesn't really matter if it is a distributed, microservice, or monolithic style system. You are also running your system in a complex environment, in this case the cloud. And you might utilize other third party services that also have an impact on your systems performance.</p>
<p>The reason why you have to have a good understanding on what is going on when it comes to performance is because complexity has not simply vanished. It is a lot easier these days to get started to and run systems based on managed cloud resources and services. But the underlying complexity has merely moved somewhere else (to be someone else's problem).</p>
<p>The issue is, that complexity can have a non-trivial impact on performance which can make it very hard to reason about a system if you have to handle it as a black box.</p>
<p>Read more in part 2 and part 3 of our blog posting series:</p>
<ul>
<li>
<p><a href="/blog/types-of-performance-testing/">Types of Performance Testing</a></p>
</li>
<li>
<p><a href="/blog/performance-testing-pre-and-post-cloud/">Performance Testing – Pre and Post Cloud?</a></p>
</li>
</ul>
<h2 id="slides">Slides</h2>
<script async="" class="speakerdeck-embed" data-id="1bf9d5cc7fff40c7a9f48feb428c343a" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Performance Testing an E-commerce Platform
https://stormforger.com/blog/performance-testing-e-commerce-platform/
2015-08-13T09:00:00+00:00
2015-08-13T09:00:00+00:00
Sebastian Cohnen
<p><a href="/blog/performance-testing-e-commerce-platform/"><img src="/blog/performance-testing-e-commerce-platform/inline-left-galeria-kaufhof-e552ea29.png" alt="Galeria Kaufhof"></a>
In June 2015 our customer GALERIA Kaufhof relaunched<br>their E-commerce platform
galeria-kaufhof.de.</p>
<p>Several teams worked for about a year on this greenfields project aimed at
building a new foundation for their customer shopping experience.</p>
<p>Prior to the official relaunch we conducted comprehensive performance and load
tests for quality assurance, configuration testing and capacity planning.</p>
<p></p>
<p><a href="/blog/performance-testing-e-commerce-platform/"><img src="/blog/performance-testing-e-commerce-platform/inline-left-galeria-kaufhof-e552ea29.png" alt="Galeria Kaufhof" /></a>
In June 2015 our customer GALERIA Kaufhof relaunched<br />their E-commerce platform
galeria-kaufhof.de.</p>
<p>Several teams worked for about a year on this greenfields project aimed at
building a new foundation for their customer shopping experience.</p>
<p>Prior to the official relaunch we conducted comprehensive performance and load
tests for quality assurance, configuration testing and capacity planning.</p>
<p></p>
<h2 id="new-architecture">New Architecture</h2>
<p><a href="https://inoio.de">inoio</a>, which is one of GALERIA Kaufhofs contractors, wrote a
nice (German) <a href="https://inoio.de/blog/2014/09/20/technologie-sprung-bei-galeria-kaufhof/">blog
post</a>
(<a href="https://translate.google.com/translate?hl=en&sl=de&u=https://inoio.de/blog/2014/09/20/technologie-sprung-bei-galeria-kaufhof/">translation</a>)</p>
<p>on the general architecture of this endeavor.</p>
<p>The goal of the rebuild was to get rid of the monolithic system and introduce a
new scalable,
<a href="http://en.wikipedia.org/wiki/Shared_nothing_architecture">shared-nothing</a>,
<a href="https://www.innoq.com/de/links/self-contained-systems-infodeck/">self contained
systems</a>
architecture to be ready for future features with a reduced time-to-market.
Everything from the operational environment up to the user interface and user
experience was redesigned and build from scratch.</p>
<p>Kaufhofs team and architecture is divided along business domain areas into scrum
teams:</p>
<p><img src="/blog/performance-testing-e-commerce-platform/inline-right-overview-77c049aa.png" alt="Overview" /></p>
<ul>
<li><strong>Front-end Integration</strong>: Integrating the systems to a coherent website
experience</li>
<li><strong>Explore</strong>: Controlling teasers</li>
<li><strong>Search</strong>: Product search and navigation</li>
<li><strong>Evaluate</strong>: Details on products</li>
<li><strong>Order</strong>: Order process</li>
<li><strong>Control</strong>: Customer account handling</li>
<li><strong>Foundation Systems</strong>: Horizontal services like media & asset delivery,
feature toggles etc.</li>
<li><strong>Platform Engineering</strong>: The machine room. Covering tools, deployment and
platform operations.</li>
</ul>
<p>Systems contained in these domain areas, also called "verticals", are designed
to</p>
<ul>
<li>not share any code</li>
<li>be loosely coupled and</li>
<li>technologically independent (they currently have Java/Scala and Ruby on Rails
teams).</li>
</ul>
<p>The only thing these self contained systems share, is the platform they operate
on. Each of these systems has the authority about its front-end, business logic
down to data storage.</p>
<h2 id="testing-approach">Testing Approach</h2>
<p>We joined the effort backed by the platform engineering team (PENG for short).
PENG provided insights like log and monitoring data while we conducted our
testing and analysis.</p>
<p>Our general approach is to assess systems in a bottom-up approach. For this
project we ended up with three test categories:</p>
<ol>
<li>
<p><strong>Service Tests</strong>: We first identified performance critical components of
each vertical and developed a test case scenario for each. The purpose of these
fine-grained test cases is to get a starting point and baseline for further
testing. These isolated tests are also used to artificially stress code paths
when troubleshooting performance related issues.</p>
</li>
<li>
<p><strong>Vertical Tests</strong>: The next step is to compile these isolated tests into
vertical or service-wide tests that will cover multiple endpoints and features
of a given service. These tests are still isolated on a vertical level so that
each team can run those tests without effecting other systems.</p>
</li>
<li>
<p><strong>Combined Tests</strong>: In the last step more complete tests are compiled. These
tests are modeled after reference data from the existing shop system and will
reuse scenarios developed in previous tests. By design these tests will cover
almost all verticals and are aimed at identifying previously undiscovered
performance critical dependencies. The goal is to get a broader view at the
system and it is the first time a more user-centric workload is modeled.</p>
</li>
</ol>
<p>There were also a couple of tests to establish a baseline in network performance
and latency. As well as a set of specialized tests to stress the shop's content
delivery and caching architecture.</p>
<h2 id="infrastructure-provider-evaluation">Infrastructure Provider Evaluation</h2>
<p>Using test cases from our first step as reference, we evaluated different
<a href="https://www.openstack.org/">OpenStack</a>-based providers and their <em>flavors</em> of
compute instances and other configuration aspects. All providers were located in
Europe, but we still had to verify that the bandwidth and base latency between
the individual providers and our load generators in Frankfurt (AWS
<em>eu-central-1</em>) and Dublin (AWS <em>eu-west-1</em>) are well known and understood.</p>
<p>The platform engineering team build a operating environment where each team can
deploy their application to. They make this possible by utilizing OpenStack APIs
and automate the entire provisioning processes using
Puppet. This made it very simple
to bootstrap and manage different target environments and e.g. make
configuration related changes.</p>
<p>For our customer we ended up in comparing multiple providers with traffic
originating from Frankfurt and Dublin.</p>
<blockquote>
<p><em>"The flexibility of StormForger enabled our platform engineering team to run
large load tests targeting different environments and data centers with ease."</em></p>
<p>— <em>Torsten Hamper, Head of System Engineering eShop Systems, GALERIA
Kaufhof GmbH</em></p>
</blockquote>
<h2 id="conclusion">Conclusion</h2>
<p>GALERIA Kaufhof succeeded with their relaunch project and managed to create a
modern, well designed and scalable E-commerce platform.</p>
<p>Evaluating different target environment is as important as testing system and
application level configurations. Understanding the basic performance
characteristics is also crucial for capacity planing and resource estimation
e.g. when you have to ensure the system is ready for big marketing events and
traffic spikes.</p>
<p>In case you speak German and would like read more about the ongoing development
check out GALERIA Kaufhofs blog at
<a href="https://galeria-kaufhof.github.io/">galeria-kaufhof.github.io</a>.</p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Building a Startup with NoSQL
https://stormforger.com/blog/building-a-startup-with-nosql/
2014-12-10T10:00:00+00:00
2014-12-10T10:00:00+00:00
Sebastian Cohnen
<p>At NoSQL Matters 2014 conference in Barcelona I was privileged to give a talk on how we at StormForger go about screening and selecting NoSQL and other technologies; with which to build our products. Although NoSQL is not necessarily the primary focus here at StormForger, we still use a number of these exciting systems and I would like to share our thoughts.</p>
<p><img src="/blog/building-a-startup-with-nosql/tisba-at-nosql-matters-bcn-3036df2a.jpg" alt="Sebastian at NoSQL Matters"></p>
<p></p>
<p>At NoSQL Matters 2014 conference in Barcelona I was privileged to give a talk on how we at StormForger go about screening and selecting NoSQL and other technologies; with which to build our products. Although NoSQL is not necessarily the primary focus here at StormForger, we still use a number of these exciting systems and I would like to share our thoughts.</p>
<p><img src="/blog/building-a-startup-with-nosql/tisba-at-nosql-matters-bcn-3036df2a.jpg" alt="Sebastian at NoSQL Matters" /></p>
<p></p>
<h2 id="startups-and-nosql">Startups and NoSQL</h2>
<p>Startups are agile & open minded, they often consist of small teams and have to be kind of pragmatic for that reason. I won't go into defining NoSQL, I'd like to refer to the <a href="https://en.wikipedia.org/wiki/NoSQL">Wikipedia definition</a> for that. Important is that NoSQL is (or at least should be) selected for two reasons: ease of use or you have a <strong>very</strong> special problem to solve.</p>
<p>I'd argue that most of the time, you don't have a uber-special problem to solve. You are probably not dealing with big data (TB+ of data, or many billions of items), you don't have to guarantee like five nines availability and don't have to scale right away to millions of requests per second…</p>
<p>If you want to be lean and agile, you should focus on the ease of use aspects (development & operation). NoSQL is about modeling data in other means than tabular relations, so maybe your data model fits to a document store, or column store, or key-value store. If that is the case, it might be "easier" to just use a NoSQL database and not try to fiddle around with mapping your data structures to <a href="http://www.postgresql.org/">PostgreSQL</a> or <a href="http://www.mysql.com">MySQL</a>.</p>
<p>But ease of use does not stop with how you model your data. Keep in mind that you still need integration into your environments, languages and frameworks, you need good tooling for your operation needs, general maturity and reliability is also important and last but not least, you need some kind of support — community or commercial.</p>
<h2 id="polyglot-persistence-at-stormforger">Polyglot Persistence at StormForger</h2>
<p>We at StormForger — besides being a startup — have different kinds of needs for data persistence. Following the <a href="http://martinfowler.com/bliki/PolyglotPersistence.html">polyglot persistence</a> approach, we have found different tools for each job.</p>
<p>Here are some examples for NoSQL usage at StormForger. We use…</p>
<ul>
<li>InfluxDB for time series data,</li>
<li><a href="http://redis.io">Redis</a> for all our caching needs, and</li>
<li><a href="http://www.elasticsearch.org">Elasticsearch</a> for log aggregation & analysis.</li>
</ul>
<p>There is no one size fits all solution for us — and most probably not for you either!</p>
<h2 id="being-lean-and-pragmatic">Being lean and pragmatic</h2>
<p>We have one more use case for data which is not really a good fit for tabular relations: Our test case definitions consist of highly structured and complex data structures.</p>
<p>Although we have begun to evaluate solutions for this need as well, we saw that we can be pragmatic about that for now. Our current solution? <strong>Serialize the JSON and just stuff it into MySQL</strong>. Although we might be in need for a more sophisticated solution in the near future, we don't need it right now to build our MVP and test our first assumptions about what the customer actually wants.</p>
<p>Especially with limited resources that you have in a startup context it's very important to take a step back every time you encounter a new interesting technology. I myself often also fall for the fancy new stuff, but as a startup we want to get one thing right: <strong>be lean and test if the product we envision is actually what the customer wants.</strong></p>
<h2 id="conclusion">Conclusion</h2>
<p>It can be perfectly fine <strong>not</strong> to have the optimal technical solution upfront. Maybe it's fine to serialize your structured data like we do, or maybe you can just use PostgreSQL's <a href="http://www.postgresql.org/docs/9.4/static/hstore.html">hstore</a> or the upcoming <a href="http://www.postgresql.org/docs/9.4/static/datatype-json.html">jsonb fields</a>.</p>
<p>If your product is indeed fundamentally based on dealing with e.g. highly structured data directly, take a look at document stores with powerful query capabilities, like e.g. <a href="https://www.arangodb.com/">ArangoDB</a>. Or you have to crunch terabytes of data with tools from the <a href="https://hadoop.apache.org/">Hadoop</a> eco system. In all other cases: Be agile, think lean and focus on validating your ideas!</p>
<p><em>Besides embedded here, you can find the slides on <a href="https://speakerdeck.com/tisba/building-a-startup-with-nosql">Speaker Deck</a>.</em></p>
<script async="" class="speakerdeck-embed" data-id="a53838005b730132958c5acb03821d2a" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
2nd Screen Mobile App: Backend API Load Testing
https://stormforger.com/blog/2nd-screen-mobile-app-backend-api-load-testing/
2014-11-12T08:57:00+00:00
2014-11-12T08:57:00+00:00
Sebastian Cohnen
<p>Having had the opportunity to support the <a href="https://www.grandcentrix.net/">grandcentrix</a> team in May to <a href="/blog/load-testing-an-interactive-tv-show-with-over-1-million-users/">help Quizduell im Ersten</a>, I was happy to once again be called to assess another interactive TV show with a 2nd screen app.</p>
<p><a href="/blog/2nd-screen-mobile-app-backend-api-load-testing/"><img src="/blog/2nd-screen-mobile-app-backend-api-load-testing/inline-right-logo_rtli-f63e5400.png" alt="RTL interactive"></a>
For <a href="http://www.rtl-interactive.de/">RTL interactive</a> I tested a sophisticated backend for an upcoming casting show. The backend was accessed through their <em>RTL Inside</em> app. This opportunity presented me with a very interesting scenario and in this post I would like to outline how their system was designed and how I was able to test the performance and scalability of the architecture.</p>
<p></p>
<p>Having had the opportunity to support the <a href="https://www.grandcentrix.net/">grandcentrix</a> team in May to <a href="/blog/load-testing-an-interactive-tv-show-with-over-1-million-users/">help Quizduell im Ersten</a>, I was happy to once again be called to assess another interactive TV show with a 2nd screen app.</p>
<p><a href="/blog/2nd-screen-mobile-app-backend-api-load-testing/"><img src="/blog/2nd-screen-mobile-app-backend-api-load-testing/inline-right-logo_rtli-f63e5400.png" alt="RTL interactive" /></a>
For <a href="http://www.rtl-interactive.de/">RTL interactive</a> I tested a sophisticated backend for an upcoming casting show. The backend was accessed through their <em>RTL Inside</em> app. This opportunity presented me with a very interesting scenario and in this post I would like to outline how their system was designed and how I was able to test the performance and scalability of the architecture.</p>
<p></p>
<blockquote>
<p><img src="/blog/2nd-screen-mobile-app-backend-api-load-testing/jerome_patt_rtl_interactive-30a6ab31.png" alt="Jérôme Patt, Project Manager at RTL interactive" />
<em>"We were actually quite relaxed during the show premiere. Everything worked
flawlessly and our complex workflows performed pretty well. Sebastian provided a
great deal of help with his professional guidance and the extensive load tests!
This brought us very close to an optimal situation, where the going live is not
the first-time peak-traffic for the architecture we designed. Load testing from
a professional third party is a major success factor for a software project and
should be on everyone's top priority list."</em></p>
<p>— <em>Jérôme Patt, Project Manager at <a href="http://www.rtl-interactive.de/">RTL interactive</a></em></p>
</blockquote>
<p>Please note, that for obvious reasons, I'm not allowed to mention any further specific details. Suffice to say RTL interactive (RTLi for short) was pleased with the findings and all the systems tested performed well in production. Please direct your inquires regarding RTL interactive, RTL, or the mentioned show to RTL's <a href="http://www.rtl-interactive.de/cms/presse/kontakt.php">press office</a>. If you have a similar project or any questions regarding performance, load or scalability testing, please feel free <a href="mailto:support@stormforger.com">to get in contact</a> with me!</p>
<h2 id="the-show">The Show</h2>
<p>The show format is that of a casting show (otherwise known as a <a href="https://en.wikipedia.org/wiki/Talent_show">talent show</a>). Several acts are introduced to the audience and they are set to perform behind a giant set of tv-screens.</p>
<p>The unique feature of this casting show is, that the viewers at home are also the jury. While a show act performs on stage, they (the audience at home) can decide with their vote whether they like the particular act or not. The big display wall fills up with pictures of viewers who voted in favor of this act. While the voting progresses the audience's appreciation is shown and once the act reaches 75% before the performance is over, the wall is lifted and the act may continue to the next round.</p>
<p>Before viewers can participate, they have to download and install the <em>RTL Inside</em> app and register to be a show juror. Optionally they can then choose to</p>
<ul>
<li>upload a photo,</li>
<li>use their Facebook picture or</li>
<li>don't want to have a picture at all.</li>
</ul>
<p>For every show act there are two interaction steps for the viewers at home:</p>
<ol>
<li>check-in</li>
<li>vote</li>
</ol>
<ul>
<li>
<p><strong>Check-In</strong>: In the check-in phase viewers have to signal if they are going to vote for the next act or not. The user check-in is required and is then used to calculate the total percentage value. When the check-in phase is over, and the act starts to perform, the voting phase begins.</p>
</li>
<li>
<p><strong>Vote</strong>: During the voting phase, the viewer uses the <em>RTL inside</em> app to signal if he or she likes the current act or not.</p>
</li>
</ul>
<p><a href="http://www.rtl.de"><img src="/blog/2nd-screen-mobile-app-backend-api-load-testing/inline-right-logo_rtl-5f1fb72b.png" alt="RTL" /></a>
Needless to say that an interactive show format like this running on <a href="http://rtl.de">RTL</a> has strict quality and performance requirements. Beside the potential reach of such a show, the synchronized check-in and voting phases were expected to produce a significant number of requests at their peak.</p>
<blockquote>
<p><em>About 2,000,000 jurors signed up using the "RTL inside" app, resulting in 7,400,000 positives votes and 9,900,00 check-ins.</em></p>
<p>— RTL press release, <em>roughly translated from German</em>.</p>
</blockquote>
<h2 id="background--architecture">Background & Architecture</h2>
<p>The backend systems were developed in-house at RTL interactive. I had the opportunity to work very closely with the development and operation teams and was impressed by the chosen architectural approaches.</p>
<p><a href="http://aws.amazon.com"><img src="/blog/2nd-screen-mobile-app-backend-api-load-testing/inline-right-logo-aws-8407fa01.png" alt="Amazon Web Services" /></a></p>
<p>RTLi decided to run the show's backend systems on <a href="https://aws.amazon.com">Amazon Web Services</a> (AWS). They committed to an interesting hybrid approach using many AWS managed services together with custom services build on top of <a href="https://aws.amazon.com/ec2/">Amazon ECs</a>.</p>
<p>The <em>RTL Inside</em> app had a special area created for the show where an in-app browser is used to make up the user interface. The app communicates directly (authenticated via <a href="http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html">AWS Signature Version 4</a>) with various AWS services, such as: <a href="https://aws.amazon.com/s3/">Amazon S3</a> and <a href="https://aws.amazon.com/sqs/">Amazon SQS</a>. This allows for the system to scale almost automatically without having to deal with <a href="https://aws.amazon.com/autoscaling/">auto scaling</a> nor anything other complicated moving parts. Other components were handled by services running on Amazon ECs and 3rd party content delivery networks.</p>
<h2 id="performance-load--scalability-testing">Performance, Load & Scalability Testing</h2>
<p>When launching a high profile interactive TV show format like this, you have to test relentlessly and extensively before you go live. Despite the fact that the RTLi team put great effort into making the optimal use of the scaling properties of AWS, dynamic performance and scalability testing is always mandatory. Results and findings of our load testing profile were also used as a basis for the capacity planning process.</p>
<p>The system itself was very nice to test, since the majority of interactions happened through AWS APIs. The architecture makes heavy use of Amazon SQS internally which results in a highly decoupled environment. As a result, most parts could be tested one after another in isolation which is always a great testing property — especially when you aim for very high throughputs and service quality. Another great aspect is that while the team was investigating a finding from previous test runs, the other system components could be tested independently, thus cutting down on time invested.</p>
<h3 id="modelling-test-cases">Modelling Test Cases</h3>
<p>Several test scenarios where modelled: From simple cases to test single components to complete and comprehensive test cases which interact with all APIs like a user would do through the app: sign up, photo upload, state polling, check-in, vote.</p>
<p>Since almost all test cases had to interact with AWS, I had to implement the <a href="http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html">AWS Signature Version 4</a> calculation algorithm in an efficient as possible manner. Since the signature is based on the request (e.g. payload), pre-computation was not an option, as every user has its own unique API keys, authentication and device tokens. Efficiency was important because we want to test with <strong>a lot</strong> of users all of which generating a good number of concurrent requests!</p>
<p>Some services were developed and operated by RTLi using Amazon ECs and <a href="https://aws.amazon.com/elasticloadbalancing/">Amazon Elastic Load Balancing</a>. Needless to say that they had to be thoroughly tested as well. General service quality, performance and scalability were the primary testing goals, besides system stability in edge case and desaster scenarios. Having a solid basis for capacity estimation and understanding the scaling properties of those systems and services was very important too.</p>
<h3 id="end-to-end-test">End-to-End Test</h3>
<p>Despite the testability of system components in isolation, we also performed <strong>extensive end-to-end tests</strong> to assess the entire process chain, spanning from processes running on Amazon ECs, background workers consuming Amazon SQS messages down to data feeds for the TV studio and other administrative dashboards reading data from various metric systems and <a href="https://aws.amazon.com/dynamodb/">Amazon DynamoDB</a>. In addition to client facing APIs there were several background systems involved e.g. to process votes and to aggregate and analyze data.</p>
<p>The extensive end-to-end tests served several goals:</p>
<ul>
<li>testing component integration under load</li>
<li>gathering data for capacity planning</li>
<li>performing scalability analysis for services running on Amazon ECs</li>
<li>assessing the system stability under stress and peak workloads</li>
<li>ensure that service quality requirements are fulfilled</li>
</ul>
<p>On a related note: AWS also always recommends testing architectures build with AWS services in a proof of concept approach. As always, be aware to give your provider a heads up when you run large scale load tests! :) For AWS this could mean to let them <a href="https://aws.amazon.com/articles/1636185810492479#pre-warming">pre-warm Amazon Elastic Load Balancers</a>, change the <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html">partitioning of Amazon S3 buckets</a> for very high throughput or increasing all kinds of <a href="http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html">account limits</a>. In case you are in doubt, reach out to <a href="https://aws.amazon.com/premiumsupport/">AWS Support</a>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We had a number of important findings which could all be addressed prior to the first show and I was told that all systems worked flawlessly in production. Together with the excellent AWS Enterprise Support, the RTLi team and I were able to pinpoint unexpected effects and conduct root cause analysis for strange latency impacts and service behavior we were seeing during tests.</p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Coming up: A whole week on performance and infrastructure in Barcelona, Spain - we are in!
https://stormforger.com/blog/coming-up-a-whole-week-on-performance-and-infrastructure-in-barcelona-spain-we-are-in/
2014-11-08T17:07:00+00:00
2014-11-08T17:07:00+00:00
Lars Wolff
<p><img src="/blog/coming-up-a-whole-week-on-performance-and-infrastructure-in-barcelona-spain-we-are-in/2014-11-07-barcelona-webperformancedays-velocity-nosqlmatters-0ee58424.png" alt="One Week Performance and Infrastructure with O'REILLY Velocity, WebPerformanceDays and NoSQL Matters in Barcelona!"></p>
<p>From November 17th to November 22nd the conferences O'REILLY® Velocity EU 2014, WebPerfDays and NoSQL
Matters will take place in Barcelona, Spain.</p>
<p></p>
<p><img src="/blog/coming-up-a-whole-week-on-performance-and-infrastructure-in-barcelona-spain-we-are-in/2014-11-07-barcelona-webperformancedays-velocity-nosqlmatters-0ee58424.png" alt="One Week Performance and Infrastructure with O'REILLY Velocity, WebPerformanceDays and NoSQL Matters in Barcelona!" /></p>
<p>From November 17th to November 22nd the conferences O'REILLY® Velocity EU 2014, WebPerfDays and NoSQL
Matters will take place in Barcelona, Spain.</p>
<p></p>
<p>In the week starting on November 17th 2014 the
O’REILLY® Velocity EU 2014 takes place in Barcelona, Spain. Sebastian (<a href="https://twitter.com/tisba">@tisba</a>)</p>
<p>In the week starting on November 17th 2014 the O’REILLY® Velocity EU 2014 takes place in Barcelona, Spain. Sebastian (<a href="https://twitter.com/tisba">@tisba</a>) and me (<a href="https://twitter.com/larsvegas">@larsvegas</a>) are looking forward
to attending this conference. I’ve been at O’REILLY® Velocity EU in London back</p>
<p>in 2010 and really loved the conference. I am curious how it will be this time.</p>
<p>On Thursday, November 20th 2014 the
<a href="http://www.webperfdays.org/events/2014-barcelona/index.html">WebPerfDays</a>
follow as the next event. WebPerfDays, for those who have never heard about it, is an
<a href="http://en.wikipedia.org/wiki/Unconference">unconference</a>
for the Web Performance Community. Sebastian will do an ignite talk on "Load Testing with over 1.000.000 Users", while I will do an ignite talk on "Continuous Performance and Load Testing". I will offer an open space session on this topic as well. Come and join us!</p>
<p>Concluding the week of conferences with trainings and talks on NoSQL is the
NoSQL Matters on Friday and Saturday.
Sebastian will be giving a talk on how to build a startup with NoSQL where he will give you some insight on how we at StormForger are using NoSQL technology and how you can too!</p>
<p>Let’s have a good time - drop us a line or give us a shot on twitter:
<a href="https://twitter.com/tisba">@tisba</a>
or
<a href="https://twitter.com/larsvegas">@larsvegas</a></p>
<p>See you around! :)</p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Amazon Web Services launched EU (Frankfurt) Region in Germany
https://stormforger.com/blog/amazon-web-services-launches-eu-frankfurt-region-in-germany-load-testing/
2014-10-23T15:28:00+00:00
2014-10-23T15:28:00+00:00
Lars Wolff
<p><img src="/blog/amazon-web-services-launches-eu-frankfurt-region-in-germany-load-testing/lets-make-the-web-a-faster-place-8821a033.png" alt="Lets make the web a faster place!"></p>
<p>Amazon Web Services launched a new Region in Frankfurt am Main /
Germany. We offer you a discount on Load and Performance Testing your
application in the EU (Frankfurt) Region.</p>
<p></p>
<p><img src="/blog/amazon-web-services-launches-eu-frankfurt-region-in-germany-load-testing/lets-make-the-web-a-faster-place-8821a033.png" alt="Lets make the web a faster place!" /></p>
<p>Amazon Web Services launched a new Region in Frankfurt am Main /
Germany. We offer you a discount on Load and Performance Testing your
application in the EU (Frankfurt) Region.</p>
<p></p>
<p>The new Region<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> supports most of the services as @jeffbar announced on the Amazon
Web Services Blog.</p>
<blockquote class="twitter-tweet" lang="en"><p>Now Open - <a href="https://twitter.com/hashtag/AWS?src=hash">#AWS</a> Germany (Frankfurt)
Region - EC2, DynamoDB, S3, and Much More - <a href="http://t.co/MauSKIm0Nq">http://t.co/MauSKIm0Nq</a> <a href="http://t.co/MmZow0DLLR">pic.twitter.com/MmZow0DLLR</a></p>— Jeff
Barr (@jeffbarr) <a href="https://twitter.com/jeffbarr/status/525272946021916674">October 23,
2014</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p><br /></p>
<p>To be honest, I personally don’t know Frankfurt that much. Like probably for most Germans Frankfurt is about:</p>
<ul>
<li>the Frankfurt Stock Exchange<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup></li>
<li>the Frankfurt Airport<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup></li>
<li>and food like Frankfurter Würstchen<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote" rel="footnote">4</a></sup> or Grüne Soße<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote" rel="footnote">5</a></sup></li>
</ul>
<p>And of course, Frankfurt has DE-CIX<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote" rel="footnote">6</a></sup>, the
largest internet exchange point worldwide in peak terms.</p>
<p><strong>Finally, Frankfurt now has an Amazon Web Service Region. Nice!<br />
Welcome your data in Germany!</strong></p>
<p>Since we want to make the web a faster place, we offer:</p>
<div class="alert alert-info" role="alert">
<p class="lead text-center text-error">
<strong>15% discount*</strong>
</p>
<p class="lead text-center text-error">
for full service and consulting on Load and Performance Testing
<br />
your HTTP API in
the new Amazon Web Services EU (Frankfurt) Region.
</p>
<p class="lead text-center text-error">
<a class="Button-cta" href="mailto:support+aws-eu-fra@stormforger.com"><strong>Request a concrete quote!</strong></a>
</p>
</div>
<p>Happy Load Testing :)</p>
<p class="smallsmall text-muted">
* This offer is limited available and valid until 31st Dec 2014. Project example 3 day Load Testing - First day: definition of non functional requirements, second and third day:
load testing and analysis, assessment and recommendation.
</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="http://aws.amazon.com/de/blogs/aws/aws-region-germany/">Amazon Web Services launched a new Region in Frankfurt am Main / Germany</a> <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p><a href="https://www.deutsche-boerse.com/dbg-en/our-company/frankfurt-stock-exchange">Frankfurt Stock Exchange</a> <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p><a href="https://www.frankfurt-airport.com/en.html">Frankfurt Airport</a> <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p><a href="http://en.wikipedia.org/wiki/Frankfurter_W%C3%BCrstchen">Frankfurter Würstchen</a> <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p><a href="http://en.wikipedia.org/wiki/Green_sauce#German_Gr.C3.BCne_So.C3.9Fe">Frankfurter Grüne Soße</a> <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p><a href="http://en.wikipedia.org/wiki/DE-CIX">DE-CIX</a> <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Getting Started with Time Series Data
https://stormforger.com/blog/getting-started-with-time-series-data/
2014-09-08T12:08:00+00:00
2014-09-08T12:08:00+00:00
Sebastian Cohnen
<p><img src="/blog/getting-started-with-time-series-data/nosql_01-c585fb75.jpg" alt="NoSQL Matters Dublin 2014">
<small class="text-muted">NoSQL Matters Dublin 2014 ⋅ © Dr. Celler Cologne Lectures GmbH & Co. KG</small></p>
<p>Last week I gave a talk at NoSQL Matters Dublin 2014 on "Getting Started with Time Series Data".</p>
<p>In this presentation I gave a quick introduction into time series data and databases. In particular I presented InfluxDB's Query Language and how you can organize, down-sample and aggregate data when it arrives at InfluxDB using their <a href="https://www.influxdata.com/blog/continuous-queries-in-influxdb-part-i/">Continuous Query</a> feature. You can find the slides over at <a href="https://speakerdeck.com/tisba/getting-started-with-time-series-data">Speaker Deck</a> or embedded at the end of this post.</p>
<blockquote class="twitter-tweet" lang="en">
<p>I blame <a href="https://twitter.com/tisba">@tisba</a> who's infected me with InfluxDB, but I must confess <a href="https://twitter.com/dweet_io">@dweet_io</a> + <a href="https://twitter.com/InfluxDB">@InfluxDB</a> == fun
Will keep you posted …
<a href="https://twitter.com/hashtag/IoT?src=hash">#IoT</a> <a href="https://twitter.com/hashtag/timeseries?src=hash">#timeseries</a></p>— Michael Hausenblas (@mhausenblas) <a href="https://twitter.com/mhausenblas/status/508325184529784832">September 6, 2014</a>
</blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p></p>
<p><img src="/blog/getting-started-with-time-series-data/nosql_01-c585fb75.jpg" alt="NoSQL Matters Dublin 2014" />
<small class="text-muted">NoSQL Matters Dublin 2014 ⋅ © Dr. Celler Cologne Lectures GmbH & Co. KG</small></p>
<p>Last week I gave a talk at NoSQL Matters Dublin 2014 on "Getting Started with Time Series Data".</p>
<p>In this presentation I gave a quick introduction into time series data and databases. In particular I presented InfluxDB's Query Language and how you can organize, down-sample and aggregate data when it arrives at InfluxDB using their <a href="https://www.influxdata.com/blog/continuous-queries-in-influxdb-part-i/">Continuous Query</a> feature. You can find the slides over at <a href="https://speakerdeck.com/tisba/getting-started-with-time-series-data">Speaker Deck</a> or embedded at the end of this post.</p>
<blockquote class="twitter-tweet" lang="en"><p>I blame <a href="https://twitter.com/tisba">@tisba</a> who's infected me with InfluxDB, but I must confess <a href="https://twitter.com/dweet_io">@dweet_io</a> + <a href="https://twitter.com/InfluxDB">@InfluxDB</a> == fun Will keep you posted … <a href="https://twitter.com/hashtag/IoT?src=hash">#IoT</a> <a href="https://twitter.com/hashtag/timeseries?src=hash">#timeseries</a></p>— Michael Hausenblas (@mhausenblas) <a href="https://twitter.com/mhausenblas/status/508325184529784832">September 6, 2014</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p></p>
<p><a href="https://twitter.com/mhausenblas">Michael Hausenblas</a> immediately got <a href="https://twitter.com/mhausenblas/status/508325184529784832">inspired</a> and blogged about using InfluxDB to get sensor data from https://dweet.io (Internet of Things) and visualize it using <a href="http://grafana.org/">Grafana</a>. Michael's <a href="https://medium.com/large-scale-data-processing/time-series-databases-fb4618201867">post</a> is a must read if you are interested in getting to know InfluxDB.</p>
<p><img src="/blog/getting-started-with-time-series-data/nosql_02-94fb5d96.jpg" alt="NoSQL Matters Dublin 2014" />
<small class="text-muted">NoSQL Matters Dublin 2014 ⋅ © Dr. Celler Cologne Lectures GmbH & Co. KG</small></p>
<script async="" class="speakerdeck-embed" data-id="3c0dfc80167a0132e24f3ade9a061f96" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>
<p><img src="/blog/getting-started-with-time-series-data/nosql_03-3b577af3.jpg" alt="NoSQL Matters Dublin 2014" />
<small class="text-muted">NoSQL Matters Dublin 2014 ⋅ © Dr. Celler Cologne Lectures GmbH & Co. KG</small></p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
StormForger on Hacker News
https://stormforger.com/blog/stormforger-on-hacker-news/
2014-07-02T13:22:00Z
2014-07-02T13:22:00Z
Sebastian Cohnen
<p>You might have noticed that StormForger was <a href="https://news.ycombinator.com/item?id=7920930">trending on Hacker News</a> on Friday, 20th June. For quite some time we were listed 3rd place. Thanks a lot for voting and your interest!</p>
<blockquote class="twitter-tweet" lang="en">
<p>Stormforger – Cloud-based Load Testing as a Service <a href="https://t.co/i1wHk3yMDQ">https://t.co/i1wHk3yMDQ</a></p>— Hacker News (@newsycombinator) <a href="https://twitter.com/newsycombinator/statuses/480033402822094848">June 20, 2014</a>
</blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p></p>
<p>You might have noticed that StormForger was <a href="https://news.ycombinator.com/item?id=7920930">trending on Hacker News</a> on Friday, 20th June. For quite some time we were listed 3rd place. Thanks a lot for voting and your interest!</p>
<blockquote class="twitter-tweet" lang="en"><p>Stormforger – Cloud-based Load Testing as a Service <a href="https://t.co/i1wHk3yMDQ">https://t.co/i1wHk3yMDQ</a></p>— Hacker News (@newsycombinator) <a href="https://twitter.com/newsycombinator/statuses/480033402822094848">June 20, 2014</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p></p>
<p><img src="/blog/stormforger-on-hacker-news/Screen_Shot_2014_06_20_at_18_19_36-1de54c89.png" alt="StormForger on Hacker News" /></p>
<p>It was quite a surprise when I started to note StormForger was mentioned more and more on twitter and it took a while to find out what was going on: Someone posted a link to our landing page!</p>
<blockquote class="twitter-tweet" lang="en"><p><a href="https://t.co/0mTXufY40Q">https://t.co/0mTXufY40Q</a> <a href="https://twitter.com/StormForgerApp">@StormForgerApp</a>, folks from Cologne are gaining traction with their wonderful project. <a href="https://twitter.com/hashtag/loadtesting?src=hash">#loadtesting</a> <a href="https://twitter.com/hashtag/testing?src=hash">#testing</a></p>— ziya aktas (@zaktas) <a href="https://twitter.com/zaktas/statuses/480662209006407680">June 22, 2014</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>We got tons of valuable feedback, requests for beta access and quite a few sign ups for our newsletter. If you want beta access, <a href="https://app.stormforger.com/users/sign_up">sign up</a>!</p>
<p>Beside the event on Hacker News a lot of stuff is going on and things are rapidly moving forward. Exciting times! :)</p>
<p><em>PS, a little side note: Although the landing page was hit by quite some traffic spikes on Friday evening, the server was very relaxed. The reason is pretty simple: static content + nginx! ;-)</em></p>
<p><img src="/blog/stormforger-on-hacker-news/Screen_Shot_2014_06_20_at_18_21_41-c798b6c0.png" alt="Side note" /></p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Load Testing an interactive TV Show with over 1 Million Users
https://stormforger.com/blog/load-testing-an-interactive-tv-show-with-over-1-million-users/
2014-05-27T11:30:00+00:00
2014-05-27T11:30:00+00:00
Sebastian Cohnen
<p>On May 12th the new and innovative show <a href="http://www.daserste.de/unterhaltung/quiz-show/quizduell/index.html">Quizduell im Ersten</a> started on German national television (<a href="https://en.wikipedia.org/wiki/ARD_(broadcaster)">ARD</a>). The show is an adaption of the very popular mobile game <a href="http://www.quizduell-game.de/">Quizduell</a> which has a user-based of over 30 million users, so the TV show was expected to be very popular.</p>
<p>I had previously <a href="/blog/quizduell-last-und-testing/">blogged</a> about the specific challenges faced when load testing such an application. This blog prompted the makers of Quizduell im Ersten to approach me to assist in dealing with the specific problems. This post is a summary and translation of both articles.</p>
<p><a href="/team">I</a> was approached by <a href="https://www.grandcentrix.net/">grandcentrix</a> as an external load testing service provider. Grandcentrix had published a German FAQ which is no longer available) on their company blog about what the challenges were and what went wrong. Most information in this article is based on these FAQs. Please note, that for obvious reasons, I'm not allowed to explicitly mention any specific detail. Suffice to say they were pleased with the results.</p>
<p></p>
<p>On May 12th the new and innovative show <a href="http://www.daserste.de/unterhaltung/quiz-show/quizduell/index.html">Quizduell im Ersten</a> started on German national television (<a href="https://en.wikipedia.org/wiki/ARD_(broadcaster)">ARD</a>). The show is an adaption of the very popular mobile game <a href="http://www.quizduell-game.de/">Quizduell</a> which has a user-based of over 30 million users, so the TV show was expected to be very popular.</p>
<p>I had previously <a href="/blog/quizduell-last-und-testing/">blogged</a> about the specific challenges faced when load testing such an application. This blog prompted the makers of Quizduell im Ersten to approach me to assist in dealing with the specific problems. This post is a summary and translation of both articles.</p>
<p><a href="/team">I</a> was approached by <a href="https://www.grandcentrix.net/">grandcentrix</a> as an external load testing service provider. Grandcentrix had published a German FAQ which is no longer available) on their company blog about what the challenges were and what went wrong. Most information in this article is based on these FAQs. Please note, that for obvious reasons, I'm not allowed to explicitly mention any specific detail. Suffice to say they were pleased with the results.</p>
<p></p>
<blockquote>
<p>"Sebastian was of tremendous help to set up and conduct the required large scale load tests. To be able to watch our Mobile Mass Response platform under load, was more than helpful in those stressful days […]"</p>
<p>— <em>Ralf Rottmann (<a href="https://twitter.com/ralf">@ralf</a>), Managing Partner at <a href="https://grandcentrix.net/">grandcentrix GmbH</a></em></p>
</blockquote>
<h3 id="flashback">Flashback</h3>
<p>I won't go into detail on the game mechanics of Quizduell im Ersten (rather check that out for yourselves). One of the big technical challenges was the achievement of the so called "TV synchronicity": Enabling an interactive, real time game experience with the TV audience via the Quizduell App at home. Depending on the game state, every 1 to 10 seconds the App is communicating at least once with the backend API. This means that (at peak) the requests per second correspond with the total number of users playing the game.</p>
<p>In the above mentioned FAQs, grandcentrix explains where the essential problems were. Parts of the complexity and the requirements were removed and I was asked to conduct load tests to back the refactoring efforts. In addition to the originally tested 85,000 requests per second, we hit the system with over 330,000 requests per second which correlates to about <strong>1 million Quizduell players</strong>.</p>
<p>I worked very closely with the grandcentrix team on quality assurance and we conducted lots of rather large scale stress tests. In this article I'd like to outline a few more details about what we did.</p>
<h3 id="test-setup">Test Setup</h3>
<p>The test cluster consisted of up to 50 AWS EC2 instances with 800 cores, 1.5 TB RAM and with well over 50GBits bandwidth. The setup was chosen to rule out any effects due to overloading the testing system.</p>
<p>The test case was modeled in a way, that test clients actually play Quizduell. A simulated API client, reacts to different game states, respects instructions to polling intervals, chooses game categories and answers to questions – the latter however not always correctly :)</p>
<p>After the test was modeled, I took care of provisioning the test systems, conducted and monitored each test execution. The grandcentrix team could therefore focus entirely on analyzing internal metrics and logs while a bot automatically reported the current test state (number of current users, current request rates, bandwidth, latency statistics, etc.) to a <a href="https://slack.com/">Slack chat</a> room. After each test execution, all relevant metrics and charts were generated and thoroughly analyzed and interpreted by the team.</p>
<h3 id="challenges">Challenges</h3>
<p>One of the bigger challenges was, that the Quizduell API is running on <a href="https://cloud.google.com/products/app-engine/">Google App Engine</a>. A deeper look on the runtime environment was therefore not possible and we had to rely on the Google support team which was outstanding.</p>
<ul>
<li>
<p><strong>DOS protection</strong>: Since the test does not have 1,000,000 servers and IP addresses, the Google DOS protection was quite a problem for some time. Action from the Google support team was required to permanently unban the load generators from the DOS protection.</p>
</li>
<li>
<p><strong>Google Magic</strong>: There are a lot of tuning knobs, which control the (scaling) behavior of App Engine and some of those are only changeable by Google itself. We had to carefully adjust those parameters to further optimize response times and stability of the API.</p>
</li>
<li>
<p><strong>Network</strong>: There are strange things happening at higher request rates and the resulting network bandwidth. Strange effects on TCP connect timings, flow control, routing and other phenomena had to be traced back to their root causes, understood and, if possible, eliminated.</p>
</li>
</ul>
<h3 id="results">Results</h3>
<p>Conducting comprehensive load tests prior to relaunching the app-enabled show had shown that grandcentrix's product "Mobile Mass Response" (not available anymore) platform is capable of handling the expected load. The latest shows have proven that most of the initial performance problems could be resolved.</p>
<p>Here are a few numbers: While testing, my setup did a total of <strong>1,213,583,187 requests</strong> in over 50 load test runs to the Quizduell system and a total of about <strong>2.21 TB</strong> of data was moved. The error rate was at about 0.000000216% (1 error every 4,624,616 requests).</p>
<p><strong>Followup</strong></p>
<p>Last Friday (23.05.) we conducted another round of large tests resulting in another <strong>800+ million requests</strong> and close to another TB of data transfer. We were able to identify and remove the remaining performance issues.</p>
<blockquote>
<p>"We’d like to thank <a href="https://twitter.com/tisba">@tisba</a> for supporting the making of <a href="https://twitter.com/search?q=%23Quizduell&src=hash">#Quizduell</a> im Ersten with the best load testing service ever! Lovin’ it!"</p>
<p><em>grandcentrix (@grandcentrix)</em> <a href="https://twitter.com/grandcentrix/statuses/470096959567839232">May 24, 2014</a></p>
</blockquote>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Quizduell im Lasttest
https://stormforger.com/blog/quizduell-im-lasttest/
2014-05-22T19:01:00+00:00
2014-05-22T19:01:00+00:00
Sebastian Cohnen
<p><em>Dear English reader, <strike>I'll translate this post later</strike> take a look at the <a href="/blog/load-testing-an-interactive-tv-show-with-over-1-million-users/">translated post</a>. <strong>tl;dr</strong>: I had the opportunity to load test the API backend for a big German TV show („Quizduell im Ersten“) by simulating up to 1 million users.</em></p>
<p>Nach dem Start des <a href="http://www.daserste.de/unterhaltung/quiz-show/quizduell/index.html">Quizduell im Ersten</a> mit einigen <a href="/blog/quizduell-last-und-testing/">Pannen</a>, wurde <a href="/team">ich</a> als externer Lasttester von grandcentrix angefragt. <strike>Mittlerweile hat [grandcentrix](https://grandcentrix.net) auf dem Unternehmensblog ein FAQ veröffentlicht</strike> (<em>nicht mehr verfügbar</em>), in dem vor allem technische Fragen angesprochen und erklärt werden. Die meisten Informationen in diesem Beitrag stammen aus diesen FAQs. Ich bitte um Verständnis, dass ich keine weiteren Details kommunizieren kann.</p>
<p></p>
<p><em>Dear English reader, <strike>I'll translate this post later</strike> take a look at the <a href="/blog/load-testing-an-interactive-tv-show-with-over-1-million-users/">translated post</a>. <strong>tl;dr</strong>: I had the opportunity to load test the API backend for a big German TV show („Quizduell im Ersten“) by simulating up to 1 million users.</em></p>
<p>Nach dem Start des <a href="http://www.daserste.de/unterhaltung/quiz-show/quizduell/index.html">Quizduell im Ersten</a> mit einigen <a href="/blog/quizduell-last-und-testing/">Pannen</a>, wurde <a href="/team">ich</a> als externer Lasttester von grandcentrix angefragt. <strike>Mittlerweile hat [grandcentrix](https://grandcentrix.net) auf dem Unternehmensblog ein FAQ veröffentlicht</strike> (<em>nicht mehr verfügbar</em>), in dem vor allem technische Fragen angesprochen und erklärt werden. Die meisten Informationen in diesem Beitrag stammen aus diesen FAQs. Ich bitte um Verständnis, dass ich keine weiteren Details kommunizieren kann.</p>
<p></p>
<h3 id="rckblick">Rückblick</h3>
<p>Ohne auf Details der Spielmechanik des Quizduell im Ersten einzugehen, war eine der großen technischen Herausforderungen die Erreichung der sogenannten „TV Synchronität“: Es soll ein interaktives, Echtzeit-ähnliches Spielen mit dem Publikum über die Quizduell App ermöglicht werden. Dazu wird, je nach Spielzustand, alle 1 bis 10 Sekunden mindestens ein Mal mit der API kommuniziert. Das bedeutet, dass bei Spitzenlast die Anzahl der Spieler der Anzahl der API Anfragen pro Sekunde entspricht.</p>
<p>In den FAQs zum Quizduell im Ersten stellte grandcentrix dar, worin die maßgeblichen Probleme beim Sendungsstart bestanden. Die Komplexität und die Anforderungen an das System wurden zurückgebaut und gleichzeitig wurden externe Lasttests durchgeführt, um den Umbau abzusichern. Über die ursprüngliche getestete Spitzenlast von 85.000 Anfragen pro Sekunde hinaus wurde das neue System mit meiner Hilfe auf über 330.000 Anfragen pro Sekunde getestet, das entspricht etwa <strong>1 Million Quizduell Spielern</strong>.</p>
<p>Ich habe die letzten Tage intensiv mit dem grandcentrix Team an der Qualitätssicherung gearbeitet und viele Belastungstests durchgeführt. Ein paar weitere Details möchte ich hier darlegen.</p>
<h3 id="testsetup">Testsetup</h3>
<p>Der Testcluster bestand aus bis zu 50 AWS EC2 Instanzen mit 800 CPU Cores, 1,5 TB RAM und über 50GBits Bandbreite. Das Setup wurde so gewählt, um Effekte durch Überlastung in jedem Fall auszuschließen.</p>
<p>Der Testcase wurde so modelliert, dass er das Quizduell spielen kann. Ein simulierter Client reagiert auf Zustandsänderungen im Spiel, beachtet Anweisungen zum Abfrageintervall, wählt Kategorien und beantwortet sogar Fragen – wenn letzteres auch nicht immer richtig :)</p>
<p>Nachdem der Test modelliert war, übernahm ich die Provisionierung der Testsysteme und die Durchführung und Überwachung des jeweiligen Testablaufes. Das grandcentrix Team konnte sich somit voll und ganz auf die Analyse der internen Metriken und Logs konzentrieren, während ein Bot automatisch den aktuellen Stand (Anzahl Benutzer, aktuelle Request Raten, Bandbreite, Latenzen etc.) in einen <a href="https://slack.com/">Slack Chat Room</a> berichtete. Abschließend wurden relevante Metriken und Charts erzeugt, die vom Team ausgiebig analysiert und interpertiert wurden.</p>
<h3 id="herausforderungen">Herausforderungen</h3>
<p>Die Herausforderungen beim Testen bestanden vor allem darin, dass die Quizduell API auf <a href="https://cloud.google.com/products/app-engine/">Google App Engine</a> lief und somit ein detailierter Einblick in die Laufzeitumgebung erschwert wurde.</p>
<ul>
<li>
<p><strong>DOS Protection</strong>: Bedingt dadurch, dass der Test nicht von 1.000.000 Rechnern und von selbiger Anzahl IPs kommt, führte dazu, dass sich immer wieder die DOS Protection von Google aktivierte. Hier war stets das Eingreifen des Google Support Teams erforderlich.</p>
</li>
<li>
<p><strong>Google Magic</strong>: Es gibt eine Reihe von Parametern, die das (Skalierungs-)Verhalten von App Engine steuern – einige dieser Parameter kann nur Google selber ändern. Hier galt es die Einstellungen weiter zu justieren, um die Antwortzeiten zu optimieren.</p>
</li>
<li>
<p><strong>Netzwerk</strong>: Bei den hohen Request-Raten und der damit verbundenen Netzwerkbandbreite treten oft seltsame Effekte auf. Unerklärliche Anstiege beim TCP Verbindungsaufbau, <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Flow_control">Flow Control</a> und viele andere Phänomene gilt es so gut wie möglich auf eine Ursache einzugrenzen, zu erklären und wenn möglich zu beseitigen.</p>
</li>
</ul>
<h3 id="ergebnisse">Ergebnisse</h3>
<p>Im Wesentlichen haben wir durch die umfangreichen Lasttests im Vorfeld zum Neustart der Sendung mit App bestätigen können, dass die Mobile Mass Response Plattform in der Lage ist das Quizduell im Ersten handhaben zu können. Die vergangenen Sendungen haben ebenfalls bestätigt, dass die meisten initialen Probleme mit der Performance beseitigt werden konnten.</p>
<blockquote>
<p>„Sebastian war eine große Erleichterung beim Aufsetzen und Durchführen der erforderlichen Lasttests, während der funktionalen Anpassungen. Das Verhalten unserer Mobile Mass Response Plattform unter Last beobachten zu können, war überaus hilfreich in der stressigen Zeit der letzten Tage […]“</p>
</blockquote>
<blockquote>
<p><em>Ralf Rottmann (<a href="https://twitter.com/ralf">@ralf</a>), grandcentrix GmbH</em></p>
</blockquote>
<p>Hier noch ein paar Zahlen: Wir haben insgesamt <strong>1.213.583.187 Anfragen</strong> in über 50 Lasttests an das Quizduell System gemacht und dabei rund <strong>2,21 TB</strong> an Daten bewegt. Die Fehlerrate lag bei knapp 0,000000216% (oder 1 Fehler auf 4.624.616 Anfragen).</p>
<p><strong>Nachtrag:</strong> Am vergangenen Freitag haben wir ein paar weitere große Tests mit weiteren <strong>800+ Millionen Anfragen</strong> und fast einem weiteren Terrabyte an Datentransfer durchgeführt. Wir konnten erfolgreich die letzten Probleme identifizieren und beseitigen.</p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Quizduell, Last und Testing
https://stormforger.com/blog/quizduell-last-und-testing/
2014-05-13T15:15:00+00:00
2014-05-13T15:15:00+00:00
Sebastian Cohnen
<p><em>Dear English reader. This post is about an event happened yesterday evening. tl;dr: A new interactive TV show had technical issues sustaining the load.</em></p>
<p><a href="http://www.daserste.de/unterhaltung/quiz-show/quizduell/index.html">Das Quizduell</a> (eine Adaption von der <a href="http://www.quizduell-game.de/">Quizduell</a> App) mit Jörg Pilawa startete mit technischen Pannen und es folgte stundenlange Häme auf Twitter und Facebook. Die Online-Ausgaben diverser Medien berichteten teilweise noch während des Geschehens.</p>
<p></p>
<p><em>Dear English reader. This post is about an event happened yesterday evening. tl;dr: A new interactive TV show had technical issues sustaining the load.</em></p>
<p><a href="http://www.daserste.de/unterhaltung/quiz-show/quizduell/index.html">Das Quizduell</a> (eine Adaption von der <a href="http://www.quizduell-game.de/">Quizduell</a> App) mit Jörg Pilawa startete mit technischen Pannen und es folgte stundenlange Häme auf Twitter und Facebook. Die Online-Ausgaben diverser Medien berichteten teilweise noch während des Geschehens.</p>
<p></p>
<h3 id="was-war-geschehen">Was war geschehen?</h3>
<p>Jörg Pilawa sagte während der Sendung, dass die Server überlastet seien und später, dass ein einzelner User über 15.000 Server blockieren würde. Andere Medien (<a href="http://www.welt.de/vermischtes/weltgeschehen/article127930874/Quizduell-Premiere-von-Hacker-attackiert.html">Welt</a>, <a href="http://www.focus.de/kultur/kino_tv/blamage-fuer-ard-premieren-panne-hacker-legen-quizduell-server-lahm_id_3838448.html">Focus</a>, <a href="http://www.sueddeutsche.de/medien/quizduell-in-der-ard-server-gehackt-team-deutschland-darf-nicht-mitspielen-1.1960320">Süddeutsche</a>, <a href="http://www.sueddeutsche.de/medien/quizduell-premiere-in-der-ard-wenn-du-einen-hintern-in-der-hose-hast-dann-melde-dich-1.1960203">Süddeutsche</a>, <a href="http://www.spiegel.de/kultur/tv/quizduell-start-hacker-legen-server-bei-ard-show-lahm-a-969029.html">Spiegel</a>) berichteten sofort über einen Hackerangriff, wovon allerdings in der <a href="http://daserste.ndr.de/quizduell/quizduell133.html">Pressemeldung</a> vom NDR gestern Abend nichts mehr zu lesen ist. Auch auf Twitter wird schlicht ein Softwarefehler und/oder der Benutzeransturm als Vermutung propagiert.</p>
<h3 id="technische-hintergrnde">Technische Hintergründe</h3>
<p>Das Mobile Mass Response System läuft nach Informationen von grandcentrix (Betreiber und Hersteller der App und des Backends) auf Google Infrastruktur (also entweder <a href="https://cloud.google.com/products/compute-engine/">Google Compute Engine</a> oder <a href="https://cloud.google.com/products/app-engine/">Google App Engine</a>). Die Aussage, dass mindestens 15.000 Server zum Einsatz gekommen sind, halte ich allerdings für eine Fehlinformation. Denn bei <a href="http://www.sueddeutsche.de/medien/quizduell-in-der-ard-server-gehackt-team-deutschland-darf-nicht-mitspielen-1.1960320">173.000 Anmeldungen</a> im Vorfeld würde das etwa 11,5 Benutzer pro Server bedeuten. Ich vermute, es sind Prozesse auf App Engine gemeint.</p>
<p>Weitere technische Details sind öffentlich leider nicht zu finden. Eventuell ist auf der Interactive Cologne mehr über die Architektur, den Vorfall und deren Hintergründe zu erfahren. Über ein <a href="https://twitter.com/ottopoellath/status/465957498248495104">Post-Mortem</a> oder einen „Lessons Learned“ Vortrag würde ich micht sehr freuen.</p>
<h3 id="lsst-sich-so-etwas-verhindern">Lässt sich so etwas verhindern?</h3>
<p>grandcentrix hat mit dem Voting-Systems des Eurovision Song Contest bewiesen, dass sie in der Lage sind, solch anspruchsvolle Architekturen zu entwickeln und zu betreiben. Lassen sich solche Zwischenfälle überhaupt verhindern?</p>
<p>Ob sich prinzipiell solche Ausfälle wie beim Quizduell verhindern lassen, ist zumindest fraglich. Es ist eine gute Mischung aus Know-How und vielleicht einem externen Auditing notwendig um Risiken zu minimieren. Trotzdem bergen solche Events enormes Potential für Probleme, da im Vorfeld nicht klar ist, mit welchen Mengen an Anfragen zu rechnen ist, ob es enorme Lastspitzen geben wird und wie sich das System unter solchen Voraussetzungen verhält.</p>
<p>Daher ist es unerlässlich, umfangreiche, nicht-funktionale Lasttests durchzuführen. Dabei sind Tests – vor allem in dieser Größenordnung – ein nicht triviales Unterfangen, selbst wenn die API auf den ersten Blick vermutlich relativ einfach erscheinen mag. Viele Testframeworks scheitern schon daran, viele hunderttausende Benutzer zuverlässig über längere Zeiträume zu simulieren; der Test und die Lastgeneratoren selber muss überwacht werden, um Überlastung zu verhindern? Wie testet man API Clients mit langsamer Verbindung? Wie bildet man dynamisches Verhalten ab? Sind die Testszenarien realistisch genug? usw.</p>
<p>Ursächlich für solche Probleme unter Last sind neben der Anwendungssoftware selbst, oft Probleme mit der (Cloud-)Infrastruktur, Lastverteilern, System- oder Netzwerkkonfigurationen der Application Server. Ein reiner Audit auf Basis von Quellcode oder Modell-basierte Analysen sind daher nicht ausreichend. Besonders problematisch ist der Betrieb solcher Systeme basierend auf Infrastruktur wie dem von Google betriebenen <a href="https://cloud.google.com/products/app-engine/">Google App Engine</a>, da hier der Nutzer wenig unmittelbare Kontrolle und Einfluss auf die Umgebung nehmen kann.</p>
<h3 id="fazit">Fazit</h3>
<p>Technisch skalierbare und Cloud-basierte Architekturen sind nicht notwendigerweise ein Garant für einen reibungslosen Ablauf. Vor allem bei Events wie dem Eurovision Song Contest oder dem Quizduell ist mit enormen Lastspitzen, „seltsamem“ Traffic oder „Angriffen“ zu rechnen. Ohne Lasttests, die den gesamten Stack (vom Loadbalancer bis zum Application Server) testen, kann auf Grund der Komplexität der eingesetzten Systeme keine verlässliche Aussage über das Verhalten eines Systems getroffen werden.</p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Using InfluxDB at StormForger
https://stormforger.com/blog/using-influxdb-at-stormforger/
2014-05-08T17:43:00+00:00
2014-05-08T17:43:00+00:00
Sebastian Cohnen
<p>Yesterday I gave a talk on InfluxDB at the <a href="http://www.nosql-cologne.org">NoSQL user group Cologne</a>. I gave a general introduction to time series data, InfluxDB and how we will use InfluxDB at StormForger to handle huge quantities of real-time processed metrics.</p>
<p></p>
<p>Yesterday I gave a talk on InfluxDB at the <a href="http://www.nosql-cologne.org">NoSQL user group Cologne</a>. I gave a general introduction to time series data, InfluxDB and how we will use InfluxDB at StormForger to handle huge quantities of real-time processed metrics.</p>
<p></p>
<p>Find the (German) slides here at <a href="https://speakerdeck.com/tisba/2014">Speaker Deck</a> or here:</p>
<script async="" class="speakerdeck-embed" data-id="295dc280b8c80131e4474251e58a135f" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<p><img src="/blog/using-influxdb-at-stormforger/influxdb-40b0f132.png" alt="InfluxDB" /></p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>
Introducing StormForger
https://stormforger.com/blog/introducing-stormforger/
2014-02-06T19:55:00Z
2014-02-06T19:55:00Z
Sebastian Cohnen
<p>Hi, my name is Sebastian (<a href="https://twitter.com/tisba">@tisba</a> on Twitter) and I'm working on a project called <a href="/">StormForger</a>. In this blog I'll keep you posted on API, performance and load testing related topics.</p>
<p>StormForger is a Load Testing as a Service platform, targeting HTTP-APIs. By providing a comprehensive and expressive DSL to specify your test cases and a high degree of automation, I try to enable people to do <strong>Continuous Load Testing</strong>.</p>
<p>To tell you in more detail what StormForger is about, I just recently published a high-level introduction video about it. If you are interested in testing your APIs, just get in contact. I'd love to have a chat with you on that topic, tell you more on the project or even setup a demo specifically to test for your APIs.</p>
<p></p>
<p>Hi, my name is Sebastian (<a href="https://twitter.com/tisba">@tisba</a> on Twitter) and I'm working on a project called <a href="/">StormForger</a>. In this blog I'll keep you posted on API, performance and load testing related topics.</p>
<p>StormForger is a Load Testing as a Service platform, targeting HTTP-APIs. By providing a comprehensive and expressive DSL to specify your test cases and a high degree of automation, I try to enable people to do <strong>Continuous Load Testing</strong>.</p>
<p>To tell you in more detail what StormForger is about, I just recently published a high-level introduction video about it. If you are interested in testing your APIs, just get in contact. I'd love to have a chat with you on that topic, tell you more on the project or even setup a demo specifically to test for your APIs.</p>
<p></p>
<div class="centered">
<div class="Youtube" data-embed="lhFbeXbiais">
<div class="PlayButton"></div>
</div>
</div>
<hr />
<p>Here is a transcript of the video:</p>
<p>Hi, my name is Sebastian and today I would like to introduce StormForger – a next generation, comprehensive API load testing solution. But before I start with StormForger, let me give you some context.</p>
<p>HTTP-based APIs can be found everywhere. Examples for APIs are…</p>
<ul>
<li>APIs to web applications</li>
<li>backends for mobile applications and online games</li>
<li>single page web applications</li>
<li>tracking & monitoring systems</li>
</ul>
<p>and so on. Applications that offer such APIs can be complex systems with extensive software stacks and their performance can be hard to predict.</p>
<p>While you most likely have test suites in place, that cover functional requirements, you might lack good answers to questions like:</p>
<ul>
<li>How fast is my system?</li>
<li>How many requests can my architecture handle?</li>
<li>Can my application scale?</li>
</ul>
<p>Load testing is a way of providing data to answer those questions.</p>
<p>It is, simply put, the process of putting a specific amount of traffic on a target system while observing the targets behavior.</p>
<p>For example, you might be interested in the response time, resource usage or error rates of your application under a specific amount of pressure.</p>
<p>Let's have a very brief look at what you have to do in order to run a load test. You have to…</p>
<ul>
<li>set test goals</li>
<li>plan tests</li>
<li>provision test resources</li>
<li>setup and deploy the test case</li>
<li>execute the test</li>
<li>monitor the running test</li>
<li>tear down the test environment</li>
<li>gather and analyze log and other sensor data</li>
<li>evaluate & compare results to previous test runs</li>
</ul>
<p>In short: Load testing can be a time consuming and challenging task. And it isn’t easy to get started with. But performance can be crucial to success, so you want to perform load tests on a regular basis. So you should ask yourself the question, why not run load tests like you do Continuous Integration?</p>
<p>The solution to target this in a fast and easy way is StormForger.</p>
<p>StormForger is a load testing as a service platform targeting HTTP APIs. It tries to simplify and automate as much as possible in regard to setting up, executing and analyzing API load tests. More importantly, StormForger tries to enable the user to conduct affordable comprehensive and reproducible load tests on a regular basis.</p>
<p>I think that getting started with load testing should be a no-brainer. My vision is to enable people to run load tests continuously: Continuous Load Testing so to speak. I also want to provide you with ongoing visibility into performance characteristics.</p>
<p>To achieve this, StormForger will provide you with the following:</p>
<ul>
<li>easy to understand test case language to specify your tests</li>
<li>management of test resources (setup, deployment, execution & monitoring)</li>
<li>detailed analytics of results</li>
<li>tools and metrics to compare multiple test runs over time</li>
<li>comprehensive HTTP API to automate and integrate StormForger into your environment</li>
</ul>
<p>StormForger is in open beta right now. If you want to try it out, get in contact.</p>
<p>Thanks for watching and happy load testing!</p>
<hr>
<p>
Do you want to learn more about StormForger - Performance Testing as a Service?
<a href="https://stormforger.com">Sign up for free!</a>
</p>