Performance Testing

Stress-testing the client/server performance of your game is crucial to a successful launch as you can preemptively discover issues that would otherwise take time to manifest and disrupt the player's experience.
In particular stress tests can help you with:

  • Predicting the scalability of the server.
  • Discovering potential client and server-side issues.
  • Highlighting problems in your Extensions that may not be evident when testing locally.

In this three part tutorial we are going to build a template application based on the client SFS2X API, that will help you stress test your online games. To achieve this we'll be using:

  • The Java API for socket-based clients (covered in this article).
  • The JavaScript API for WebSocket-based clients (covered in part two).

Finally, part three of this series, we will offer different tips about how to create a proper test and avoid potential pitfalls.

Testing strategy

In order to generate multi-user traffic towards our SFS2X server we will need two components:

  • An SFS2X client with the our testing logic.
  • A Test Replicator, which is a small program that spawns hundreds of clients that connect and talk to the server.

Building the socket-based test in Java

We are going to start by building the Test Replicator first:

			public class StressTestReplicator
			{
				private final List<BaseStressClient> clients;
				private final ScheduledThreadPoolExecutor generator;
				
				private String clientClassName;		
				private int generationSpeed = 100; 	
				private int totalCCU = 10;			
				
				private Class<?> clientClass;
				private ScheduledFuture<?> generationTask;
				
				public StressTestReplicator(Properties config)
				{
					clients = new LinkedList<>();
					generator = new ScheduledThreadPoolExecutor(1);
					
					clientClassName = config.getProperty("clientClassName");
					
					try { generationSpeed = Integer.parseInt(config.getProperty("generationSpeed")); } catch (NumberFormatException e ) {};
					try { totalCCU = Integer.parseInt(config.getProperty("totalCCU")); } catch (NumberFormatException e ) {};
						
					System.out.printf("%s, %s, %s\n", clientClassName, generationSpeed, totalCCU);
					
					try
					{
						// Load main client class
						clientClass = Class.forName(clientClassName);
						
						// Prepare generation
						generationTask = generator.scheduleAtFixedRate(new GeneratorRunner(), 0, generationSpeed, TimeUnit.MILLISECONDS);
					}
					catch (ClassNotFoundException e) 
					{ 
						System.out.println("Specified Client class: " + clientClassName + " not found! Quitting.");
					}
				}
				
				void handleClientDisconnect(BaseStressClient client)
				{
					synchronized (clients)
					{
						clients.remove(client);
					}
			
					if (clients.size() == 0)
					{
						System.out.println("===== TEST COMPLETE =====");
						System.exit(0);
					}
				}
				
				public static void main(String[] args) throws Exception
				{
					String defaultCfg = args.length > 0 ? args[0] : "config.properties";
					
					Properties props = new Properties();
					props.load(new FileInputStream(defaultCfg));
					
					new StressTestReplicator(props);
				}
					
				private class GeneratorRunner implements Runnable
				{
					@Override
					public void run()
					{
						try
						{
							if (clients.size() < totalCCU)
								startupNewClient();
							else
								generationTask.cancel(true);
						}
						catch (Exception e)
						{
							System.out.println("ERROR Generating client: " + e.getMessage());
						}
					}
					
					private void startupNewClient() throws Exception
					{
						BaseStressClient client = (BaseStressClient) clientClass.newInstance();
						
						synchronized (clients)
						{
							clients.add(client);
						}
						
						client.setShell(StressTestReplicator.this);
						
						client.startUp();
					}
				}
			}

This is a relatively simple class that reads the test configuration from a properties file (config.properties) and starts generating new clients until a certain number of CCU is reached.

The generation process is delegated to a dedicated Scheduler whose Task is handled by the GeneratorRunner class.

The external config.properties file looks like this:

			clientClassName=sfs2x.example.stresstest.SimpleChatClient
			generationSpeed=100
			totalCCU=10
  • clientClassName: the name of the class to be used as the client logic
  • totalCCU: the total number of clients for the test
  • generationSpeed: the interval between each generated client, expressed in milliseconds

Building the Java client

Now it's time to build a simple Java client based on the SFS2X Java API, that performs the following operations:

  • Connect to the target server
  • Login as an anonymous User
  • Join a Room
  • Send a number of public messages at regular intervals
  • Disconnect

Below is the relative implementation:

			public class SimpleChatClient extends BaseStressClient
			{
				// A scheduler for sending messages shared among all client bots.
				private static ScheduledExecutorService sched = new ScheduledThreadPoolExecutor(1);
				private static final int TOT_PUB_MESSAGES = 50;
				private static final String PUB_MESSAGE = "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua";
				
				private SmartFox sfs;
				private ConfigData cfg;
				private IEventListener evtListener;
				private ScheduledFuture<?> publicMessageTask; 
				private int pubMessageCount = 0;
				
				@Override
				public void startUp()
				{
					sfs = new SmartFox();
					cfg = new ConfigData();
					evtListener = new SFSEventListener();
					
					cfg.setHost("localhost");
					cfg.setPort(9933);
					cfg.setZone("BasicExamples");
					
					sfs.addEventListener(SFSEvent.CONNECTION, evtListener);
					sfs.addEventListener(SFSEvent.CONNECTION_LOST, evtListener);
					sfs.addEventListener(SFSEvent.LOGIN, evtListener);
					sfs.addEventListener(SFSEvent.ROOM_JOIN, evtListener);
					sfs.addEventListener(SFSEvent.PUBLIC_MESSAGE, evtListener);
					
					sfs.connect(cfg);
				}
				
				
				public class SFSEventListener implements IEventListener
				{
					@Override
					public void dispatch(BaseEvent evt) throws SFSException
					{
						String type = evt.getType();
						Map<String, Object> params = evt.getArguments();
						
						if (type.equals(SFSEvent.CONNECTION))
						{
							boolean success = (Boolean) params.get("success");
							
							if (success)
								sfs.send(new LoginRequest(""));
							else
							{
								System.err.println("Connection failed");
								cleanUp();
							}
						}
						
						else if (type.equals(SFSEvent.CONNECTION_LOST))
						{
							System.out.println("Client disconnected. ");
							cleanUp();
						}
						
						else if (type.equals(SFSEvent.LOGIN))
						{
							// Join room
							sfs.send(new JoinRoomRequest("The Lobby"));
						}
						
						else if (type.equals(SFSEvent.ROOM_JOIN))
						{
							publicMessageTask = sched.scheduleAtFixedRate(new Runnable()
							{
								@Override
								public void run()
								{
									if (pubMessageCount < TOT_PUB_MESSAGES)
									{
										sfs.send(new PublicMessageRequest(PUB_MESSAGE));
										pubMessageCount++;
									}
									else
									{
										// End of test
										sfs.disconnect();
									}
									
								}
							}, 0, 2, TimeUnit.SECONDS);
						}
						
					}
				}
				
				private void cleanUp()
				{
					// Remove listeners
					sfs.removeAllEventListeners();
					
					// Stop task
					if (publicMessageTask != null)
						publicMessageTask.cancel(true);
					
					// Signal end of session
					onShutDown();
				}
			}

We need to make sure that our custom client class extends the BaseClient, which looks like this: 

			public abstract class BaseStressClient
			{
				private StressTestReplicator shell;
				
				public abstract void startUp();
				
				public void setShell(StressTestReplicator shell)
				{
					this.shell = shell;
				}
				
				protected void onShutDown()
				{
					shell.handleClientDisconnect(this);
				}
			}

There are only two methods that we can override:

  • startUp() is our entry point of the test client. This is invoked when the class is instantiated and it's usually the method where we want to setup and launch the SFS2X connection. Make sure to override this one.

  • shutDown() is called by our client when the test is complete or a disconnection occurs. There is no need to override this method unless we also modify the TestReplicator class.

Launching the first test

You can run the test directly in your IDE or you can use the scripts provided in the project folder:

  • Windows: double click the run.bat script (or launch it from a command prompt Window)
  • Linux/macOS: open the terminal, change directory to the project folder and execute ./run.sh

Before we start let's double check that we have correctly configured the host and port parameters for the connection and set the totalCCU property in config.properties to 1.

This way we can check that everything is in order and that the client is working correctly. Once this is done we can proceed with configuring the tool for a larger test.

Downloads

You can download the the Java template code discussed so far below:

The zip contains an Eclipse project that can be directly imported into the IDE. If you're using a different environment you can create a new project and import the sources and relative jar dependencies.

Additional resources

This article is followed up two more tutorials that we recommend: