Cluster Monitor module
The AdminTool's Cluster Monitor module shows the fundamental status indicators that administrators should keep under control when monitoring the nodes of a cluster. It also provides a few controls to manually intervene on the nodes orchestration. A separate tab shows the server-side events logged by the cluster's Conductor component as it adds and removes the nodes.
This module is available when connected to a Lobby Node only.
Cluster status
The Cluster Status tab is the core of this administration module. It provides a global view on the cluster condition, the list of Game Nodes currently running in the cluster and a number of charts showing the trend of the nodes main indicators.
Global status
The top section of the dashboard shows the Lobby status icon and the global indicators for the cluster, as described below. If a single Lobby is currently running in the cluster, its status is always active (green status icon). Other Lobby Nodes launched after the first one are always inactive, as they don't participate in the cluster. More information on the active/inactive state of the Lobby is available in the Lobby Node migration document.
Please note that an inactive Lobby doesn't show the actual cluster status described in this document, because no Game Nodes are connected to it.
- Uptime
The timer shows how long the cluster has been running since it was launched. - Total Nodes
The total number of Game Nodes currently running in the cluster alongside the Lobby. Nodes are divided into active and inactive. A node is flagged as inactive if the health check executed by the Lobby fails for a protracted period of time (usually due to high latency in Lobby ↔ Game Node communication). - Total CCU
The total number of users currently connected to the Lobby and Game Nodes. - Total Rooms
The total number of Rooms currently existing on the Game Nodes.
Nodes status
Below the global indicators, a table lists all the Game Nodes, grouped by active or inactive status, showing the main server details and indicators.
- Node
The IP address or custom domain name of the Game Node. You can click the toolbox icon to access the Game Node's AdminTool. - CCU
The number of users currently connected to the Game Node. - CPU
The current usage, in percentage, of the available CPU cores. It tells you how busy is the server at the moment: the server starts to get very busy when many concurrent requests are sent and it has to quickly respond to all of them. - Memory
The current memory usage in the Java Virtual Machine, divided into used memory and allocated memory. - Network
The incoming and outgoing data transfer rates of the Game Node. - Threads
The number of threads currently running. This value can be useful to check if your custom Extension is generating an uncontrolled number of threads causing a degradation in performance. - Rooms
The number of Rooms currently existing on the Game Node. - Joined
The elapsed time since the Game Node joined the cluster. - Health
Together with the next two, this indicator provides information on the state of the Game Node in the cluster. If an high latency in Lobby ↔ Game Node communication is measured for some time, the Game Node is considered to be unhealthy and the green light turns to red. In case of prolonged unhealthy state, the node becomes inactive. - Recyclable (R)
The node left the cluster due to a scale-down event triggered by the Conductor component of the Orchestration system, but it can join the cluster again if a scale-up condition arises within a certain amount of time. - Terminable (T)
The node left the cluster due to a scale-down event. The time threshold to recycle this Game Node has passed and it is now scheduled for termination.
The nodes list is updated every 5 seconds, but you can force the update by clicking on the Refresh button below the table's bottom right corner.
The Scale-up button allows triggering the Conductor to add a number of new Game Nodes to the cluster, independently from the Orchestration automated process. Enter the number of nodes to be launched in the modal panel that shows up when clicking the button, then confirm.
The Deactivate nodes button makes it possible to set the selected nodes as inactive, excluding them from the Load Balancing algorithm. More information on the active/inactive state of the Game Nodes is available in the Game Nodes migration document.
Charts
Select one or more nodes using the checkboxes in the first column of the list to display the diagrams showing the evolution over time of the nodes' main indicators: concurrent users, CPU, used memory, data transfer rate (both in and out) and threads.
The displayed time span can be selected using the Range dropdown. Move the mouse over the series to read the point values, or click a chart to enlarge it.
Charts are refreshed every minute. Small charts always show 30 data points; large charts show between 30 and 360 points, depending on the displayed time span. In any case data is sampled accordingly.
Orchestration Log
The Orchestration Log tab displays the latest events logged by the Conductor component during the automated process od adding and removing Game Nodes.
In particular this view is useful, in conjunction with the Scale-up button discussed before, to check the status of the action when the cluster scaling is triggered manually.