Load testing REST APIs

Your REST API project is in need of a load test, to confirm that it satisfies the clients needs. So, what tools do you use? This post is a round up of the ones I used, and my impressions.

When creating load tests for REST APIs, you're doing a lot of what we do when we test web pages. You have an URL, some parameters to it, you're using http - but usually you have some things that are different. Many APIs use http headers for transmitting their authentication bits: tokens, usernames, API version, API key, various info that's not passed in by URL. Some APIs may not have these requirements, and can be tested using standard web page testing tools that are not capable of setting header values.

Many APIs also require a login to allow calling the different resources, and your test tool would need to perform the login and pass the returned token along with every subsequent request.
The http status codes are also used differently in an API, when browsing the web a 404 is an error, whereas in an API context it could just mean that no matching record was found for your search, so it would be nice if the load test didn't report errors in these expected cases.
All the tools I've looked at are free, or at least have a free version with some limitations. I also looked at WCFStorm.Rest, but at the time it did not support load testing - I see now it does.

WCAT

I've done some load testing of web pages in the past, and WCAT did the job just fine. If you're not familiar with it, it's the tool the IIS guys use to benchmark IIS. It uses scripts to generate http requests to a web server. While having a bad rep for its learning curve (i.e. "brick wall"), it is very light-weight, free and absolutely bloat-free. It also has a nifty feature of doing scripted remote install of the clients, so that may save you some time.

I used the google chrome plugin postman to create my requests - but you can use anything that calls your API, or replay your logs - and had fiddler running in the background, capturing what was happening. Then, with the magic of fiddler wcat plugin, I was able to generate a WCAT script file, which can be run by the WCAT clients to create the actual load.

If you have a generated load, for instance the logs of your API in production, you can replay them through fiddler and the plugin, and have a real-world and perfectly repeatable load test for your API.

I had no problem generating the scripts for doing simple load tests like a GET to a single resource, or recording a specific set of operations on different static resources. The thing with WCAT is that it's not easily scriptable.

What you have recorded in your file is what it does, every time. In terms of reusing the data from response #1 in request #2, it's just not easily doable. The script will have recorded some value and put them in the header, but it's just a static value and WCAT will attempt to use it every time. And when your API changes (as it will), your wcat scripts need upgrading. In most cases, that will require a large portion of manual labour.

However, if you need to do testing locally, and you can hard code values (you can provide a file with input data, like usernames) or just test parts that don't rely on re-using data from requests, it's fine. It wouldn't be my first recommendation if you've never used it before, though.

Blitz.io

Blitz.io is a cloud based solution, while it's basically a cURL front-end. The cool thing is that it lets you specify the number of clients, and in which region of the world they should come from. There's no learning curve, there's a "command line" on the web site where you enter your query, the most basic one being just your root url.
First things first: To avoid (D)DoS attacks from their service, blitz.io requires you to setup a given url that returns a given number. We setup this in the loadbalancer for our project, which is a good option if you're doing a self-hosted project where no file can be added to your server (and you don't want that strange GUID in your routes).

They have an API for scripting your scenarios, so you can write them and keep them locally, and trigger the execution of the test. This comes in handy if you want to do automated performance tests, I guess. I wrote my scripts using node.js, since blitz.io provides a node module for this. It's also available on github.

The usage of this service is not free, of course. Everyone is given a free quota of 10 credits, which means 10 runs of 1000 users for one minute. If you increase the number of users, or the time period, you will use more credits. For my project, 1000 users was fine. But be prepared to buy some more credits, or you'll have to wait until next month to get 10 new ones.

The results are presented, as shown in the image above, in a nice user-friendly way. I've selected the highlights tab, which gives you, well, just that. Most times, the highlights will tell you what you wanted to know, and even your least technical proficient client will understand the essence of the text that follows.
It differs from WCAT in actually executing the tests over the internet, and that can reveal some weaknesses in your environment. It's also easier for you to set up a lot of users, if you don't have any machines available for WCAT. It is scriptable, which is a great plus, but it lacks the possiblity of extracting data from the responses and injecting them in subsequent requests. Last I heard, they had it in their backlog to implement this.

SoapUI + LoadUI

This software from Smartbear is a quite heavy install, but it does a nice job. I've used SoapUI previously for functional tests on web services, but I've not used it for load testing. Basically you set up a SOAP UI project and point it to your API. Then you create requests for each object. You have to edit the content-type if you want to use json instead of xml.
If you have already been using SoapUI for functional tests on your services or API, you can go right ahead and use those tests for a load test, with no work needed. Just skip the next paragraph.

Then you create tests for your endpoints and their routes. It is a bit of a cumbersome process. That's subject for a different blog post, but there's a tutorial here. What's nice about this is that you can script things in the steps, and you can create "transfer value"-steps where you extract data from one response and use it as input in a new request. Which made it possible to do a login, then get the token from the response and use it in the rest of the scenario. This was the only tool I was able to do this with!

Finally, you can create a test suite of your tests, and simply right click it to choose "Run in LoadUI". The UI in LoadUI is a bit different from what you'd expect, it looks almost like one of the old synthesizers in the 80s, but it works. In comparison with WCAT and Blitz.io, it particularly stands out in that it has a graphical UI, and that it lets the user adjust the load in real-time, and also monitor the statistics real-time. You can adjust the number of virtual users, as well as the distribution of scenarios, delays between steps, load increments and so on. Of all of the tools I considered, LoadUI was the only one with the real-time adjustment option. How useful it is, I'll leave up to you to decide.

There's also an option in SoapUI to run load tests internally, this option works well enough, but it doesn't provide the extra functionality given by LoadUI. If you just need to specify the number of virtual users and the steps they carry out, this tool may give you what you need.

Unfortunately, I did not feel that LoadUI was perfect either. Especially when running larger tests, most times my computer stopped responding, or the tests did not terminate as I wanted them to. Typically, if I crossed a limit in the number of users specified, the UI would respond more and more slowly, but when I tried to stop the test, it did not respond. After a while, I figured out if I could wait long enough, the results would eventually appear, but 30 minutes is an awkward long time. The best way I found to mitigate this was to run the setup the LoadUI scenario from the graphical UI, and then run the scenario from command-line. This effectively removes the real-time option, but that option may be more relevant when setting up the scenarios, than when actually performing the tests.

The breakdown

Happy load testing.