Performance Testing, Part 2: WebSockets

In part one of this article we've discussed how to implement a simple stress test application in Java and targeting the native SFS2X socket protocol.
In this second part we'll take a look at porting the same application to JavaScript which uses the WebSocket protocol and run it in the browser.
Finally, in part three of this series, we will discuss several testing tips, best practices and how to avoid potential pitfalls.

Testing Strategy

As outlined in the previous article we need to build two components for our tests:

  • The SFS2X client with our testing logic
  • The Test Replicator, which is a small program that spawns hundreds of clients which connect and talk to the server

Building the WebSocket-based test in JavaScript

We will begin by building the Test Replicator class:

			var WSClient = {};
			
			WSClient.TestReplicator = function(params)
			{
				this.clients = [];
			
				this.generationSpeed = params.generationSpeed;
				this.totalCCU = params.totalCCU;
				this.clientClass = params.clientClass;
			}
			
			WSClient.TestReplicator.prototype.handleDisconnection = function(client)
			{
				// remove from client array
				var index = this.clients.indexOf(client);
			
				if (index > -1)
					this.clients.splice(index, 1);
			
				if (this.clients.length == 0)
				{
					console.info("===== TEST COMPLETE =====");
				}
			}
			
			WSClient.TestReplicator.prototype.generateClients = function()
			{
				if (this.clients.length >= this.totalCCU)
				{
					console.log("-- All clients generated: " + this.clients.length + " --")
					return;
				}
			
				else
				{
					var newClient = new this.clientClass();
					this.clients.push(newClient);
			
					newClient.setReplicator(this);
					newClient.startUp();
			
					// Call itself until done
					setTimeout( () => { this.generateClients(); }, this.generationSpeed );
				}
			}

The class requires a configuration object passed in the constructor to setup the test, with the following properties:

  • generationSpeed: the ms. interval between each client generation
  • totalCCU: the total amount of clients to be created
  • clientClass: the name of the SFS2X client class

Building the JavaScript client

As seen in part one of the tutorial we're going to build a simple client (based on the SFS2X JavaScript API) that performs the following operations:

  • Connect to the target server
  • Login as an anonymous User
  • Join a Room
  • Send a number of public messages at regular intervals
  • Disconnect

Below is the relative implementation:

			WSClient.SimpleTestClient = function() 
			{
				this.MAX_MESSAGES = 5;
				this.MESSAGE_INTERVAL = 2000; // milliseconds
				this.TEST_MESSAGE = "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua";
			
				this.config = {};
				this.config.host = "127.0.0.1";
				this.config.port = 8080;
				this.config.zone = "BasicExamples";
				this.config.debug = false;
				this.config.useSSL = false;
				
				this.sfs = new SFS2X.SmartFox(this.config)
			
				// Add event listeners
				this.sfs.addEventListener(SFS2X.SFSEvent.CONNECTION, this.onConnection, this);
				this.sfs.addEventListener(SFS2X.SFSEvent.CONNECTION_LOST, this.onConnectionLost, this);
				this.sfs.addEventListener(SFS2X.SFSEvent.LOGIN, this.onLogin, this);
				this.sfs.addEventListener(SFS2X.SFSEvent.ROOM_JOIN, this.onRoomJoin, this);
			
				this.messageCount = 0;
			}
			
			WSClient.SimpleTestClient.prototype = new WSClient.BaseClient();
			
			WSClient.SimpleTestClient.prototype.startUp = function (argument) 
			{
				this.sfs.connect();
			}
			
			WSClient.SimpleTestClient.prototype.onConnection = function(event)
			{
				if (event.success)
				{
					console.log("Connected to SmartFoxServer 2X!");
					this.sfs.send(new SFS2X.LoginRequest(""));
				}
				else
					console.log("Connection failed");
			}
			
			WSClient.SimpleTestClient.prototype.onConnectionLost = function(event)
			{
				console.log("Connection lost: " + this.sfs.mySelf.name);
				this.shutDown();
			}
			
			WSClient.SimpleTestClient.prototype.onLogin = function(event)
			{
				console.log("Logged in as: " + this.sfs.mySelf.name);
				this.sfs.send(new SFS2X.JoinRoomRequest("The Lobby"));
			}
			
			WSClient.SimpleTestClient.prototype.onRoomJoin = function(event)
			{
				console.log("Joined in Room: " + event.room);
				this.runMessageLoop();
			}
			
			WSClient.SimpleTestClient.prototype.runMessageLoop = function()
			{
				if (this.messageCount >= this.MAX_MESSAGES)
				{
					console.log("All messages sent");
					this.sfs.disconnect();
				}
			
				else
				{
					this.sfs.send(new SFS2X.PublicMessageRequest(this.TEST_MESSAGE));
					this.messageCount++;
			
					setTimeout(() => { this.runMessageLoop() }, this.MESSAGE_INTERVAL);
				}
			}

If you are familiar with the basics of the SmartFox client API this code shouldn't require much explanation. If not, we recommend following the official documentation here.

One interesting aspect to point out is that every client we want to run with this tool is required to extend the BaseClient class, which looks like this:

			WSClient.BaseClient = function() 
			{
				this.replicator = null;
			};
			
			WSClient.BaseClient.prototype.startUp = function() {}
			
			WSClient.BaseClient.prototype.shutDown = function() 
			{
				this.replicator.handleDisconnection(this)
			}
			
			WSClient.BaseClient.prototype.setReplicator = function(replicator) 
			{
				this.replicator = replicator;
			}

Extending the base class is done in our test client at this line:
WSClient.SimpleTestClient.prototype = new WSClient.BaseClient();

There are only two methods that we need to override:

  • startUp() is our entry point of the test client. This is invoked when the class is instantiated and it's usually the method where we want to setup and launch the SFS2X connection. Make sure to override this one.
  • shutDown() is called by our client when the test is complete or a disconnection occurs. There is no need to override this method unless we also modify the TestReplicator class.

Putting it all together

In order to satisfy the necessary dependencies we will load the base class, the SFS2X API and the rest of the code via an HTML file.

			<html>
			<head>
				<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
				<title>:: Test Runner ::</title>
				
				<script type="text/javascript" src="sfs2x-api-1.7.17.js"></script>
				<script type="text/javascript" src="WSTestLib.js"></script>
				<script type="text/javascript" src="SimpleTestClient.js"></script>
				<script type="text/javascript">		
					function main () 
					{
						var testParams = 
						{
							generationSpeed: 100,
							totalCCU: 10,
							clientClass: WSClient.SimpleTestClient
						};
			
						var testReplicator = new WSClient.TestReplicator(testParams);
						testReplicator.generateClients();
					}
				</script>
			
			</head>
			
			<body onload="main()">
				<h2>:: JS | Stress Test Tool ::</h2>
				<hr>
			</body>
			
			</html>

We have also added a small main() function where we set up the basic parameters to start the test. The method is called as soon as the page is loaded, via the onload event.

Running the test

Now that we've completed the overview we can see how to run the application in the browser.

For starters we need to keep in mind that most browsers are limited to around 200-250 connections per single web-page, and running multiple pages/tabs is not an option. This is because only the page/tab that is running in foreground gets all the resources, while the others won't get enough CPU cycles and slow down the test.

After having tested several popular browsers (Chrome, Firefox, Edge, Safari) we have concluded that Mozilla Firefox is the best choice for this stress test application. The main reason is that Firefox allows to reconfigure the maximum number of open WebSockets, although there're still some limitations (more on this below).

To access the Firefox configuration type about:config in the Firefox search bar, then search for 'socket' in the new search field that appears in the page.

Firefox setting

With this setting you will likely be able to reach around 980-990 connections, after which the client will likely fail to connect more sockets. The reason for this is not entirely clear, but it's a reasonable amount for a test and it can be improved by adding more machines running in parallel.

Finally we can run the test application by dragging the main WSTestRun.html file onto a new browser window. You can also open the developer's console if you want to see the test activity (using CTRL+SHIFT+K on Windows or CMD+SHIFT+K on Mac).

Downloads

You can download the the JS template code discussed so far below:

Additional resources

This article is followed up two more tutorials that we recommend: