Dedicated Game Server

This page/guide was originally published in 2009 and has not been updated since then.  It remains here as an archive.  Any links on the page may or may not work.  

As with all applications, where someone has found a way to sell something, someone else has found a way to do something for free (or relatively free).

Most gaming communities rent their servers from a company which specializes in game servers.  While most of these companies run a good program, they can get to be pretty expensive.  Obviously, you’re paying for support, knowledge, and high availability.  When the server crashes, you want to blame someone else to fix it.  Not have to pull an all-nighter to get it running.  Most companies charge anywhere between $0.99 and $5.00 per player slot to run a server (depends on it’s usage, etc.).  As true with anywhere else, you get what you pay for.  The cheaper ones will tend to stack many server instances on a single piece of hardware, limit bandwidth, limit FPS (which determines shot accuracy), etc.

The smaller providers tend to rent dedicated servers from Colocrossing.com – which houses data centers across the country (world, I believe), and offers nothing but high end bandwidth and solid hardware.  Many gaming communities choose after awhile to go the dedicated server route for its ease of use in adding and deleting game servers, as well as utilizing the reduced demands of a community versus the entire playing field.

I’ll be walking you through the setup of a dedicated game server from start to finish.  This is not for those unfamiliar with Linux. While there are many companies out there who will gladly rent you a server with Windows on it, you pay a premium for this.  In my opinion, it’s not worth it, when you’re running a server that can cost between $100 and $300 per month. Typically, the premium for Windows (and Cpanel installations) is an additional $25 per month or more.

Let’s talk hardware to begin with.  You’ll want a provider with a datacenter in a major city – Chicago, New York, Dallas, Atlanta, Washington DC, or San Jose.  Colocrossing covers those.  I personally also recommend SingleHop, as they have a Chicago and Dallas datacenter.  You can get away with any others, but realize that there’s extra routing involved to get there.  In games, especially First Person Shooters (FPS), latency makes a difference.  You’ll also want to get something with a decent amount of bandwidth and a good (100MB) connection.  3TB should be good to run 3-4 game servers per month (assuming an average of 15-25 slots during “prime time”.  As with a desktop computer, upgrade memory wherever possible.

As with my home server, I prefer CentOS 64bit.  You can use any other flavors, and adjust the commands as accordingly.  Of course, Debian-based flavors don’t offer the package installer that CentOS offers.

Initial Setup

You’ll be starting with a “naked” server – basically it’s got the base system and SSH installed.  (Remember, Linux as a server is better in the command-line environment anyways.)  While most of your work will be done from root, never run the game servers under root.  (Security breach.)  To optimize performance, you won’t be running any extraneous processes like web server, MySQL, etc.  Just the gameserver instances, SSH, and FTP (if you so choose).  Try to avoid getting a CPanel installation, as that requires additional services.

yum -y groupinstall Base, Development Libraries, Development Tools, Administration Tools, FTP Server, Server Configuration Tools, System Tools, Editors, Yum Utilities
yum -y install screen, nano
yum -y update
yum -y upgrade

Most dedicated server providers will use a custom installation/format that puts the operating system files on a small partition, while the larger partition is left aside for your “var/www/” folder.  (Remember, most who rent dedicated servers are webmasters.).  You’ll need to create a symbolic link first.

mkdir /var/home
ln -s /var/home /home

Now, create your users so that you can run game servers without being root.  There’s some debate here as to running as a single user for all game servers or a game-server per user type of setup.  Personally, I prefer to run all game-servers under one userid.

useradd gameserver
password gameserver  (at this point, you'll be prompted to create a password)

You can continue with the next steps as root, but I prefer to go into the user account and work.  You can either log out, and log back in as the user, or run using the “sudo” command.

su gameserver
cd /home/gameserver
mkdir srcds
cd srcds
wget http://www.steampowered.com/download/hldsupdatetool.bin
ln -s /usr/bin/gunzip /usr/bin/uncompress
chmod +x hldsupdatetool.bin
./hldsupdatetool.bin
./steam
./steam -command update -game "Counter-Strike Source" -dir .

That’s a period at the end of that last line.  It’s basically telling the software to install all the game files in the local directory, creating folders “cstrike”, “hl2”, “bin”.  While this is fine if you’re running your own server at home, it’s not good if you’ll be running multiple servers with different settings.  It’s best to change that “.” to a folder name of your choosing, such as “css-server”.  If you want to install other games, visit http://developer.valvesoftware.com/wiki/Version_Game_Server.  You’ll be replacing the “Counter-Strike Source” with one of those options.  By the way, this install takes awhile (20-30 minutes).  If you know of a shortcut way to do this through the “screen” utility, try it out (especially if you want to do multiple installs at once).  A word of note – some of these games automatically create a “housing” folder, so you’re better off using the “.”

Once you’re installed, now it’s time to fire up the game server to make sure it works.  You’ll first need a server.cfg in your cstrike/cfg folder.  There are plenty of utilities and examples out there.

./srcds_run -console -game cstrike +map de_dust2 +ip xxx.xxx.xxx.xxx -port 27015 +maxplayers 24 -autoupdate -tickrate 100 -fps_max 0

The ip switch is if you have multiple ip’s to your box.  You want to try and limit the server usage to 1 or 2 instances per ip address.  Typically, ports 27015 and 27016 are good.  I’ve played on some at 27017 and 27018, but it’s not “standard”.  Certainly feel free to change your max_players.  The last switch “fps_max 0” tells the system to run at the maximum fps available.  Under ideal circumstances on CSS, we’re shooting for 1000 fps (or just under). Typing “stats” should show you what the server fps is.

Without closing out of the console (and assuming everything booted correctly), attempt to connect to the server from the game.  If the server’s running, but you can’t see it, make sure you set the ip correctly, that you’re not running a firewall (blocking ports), and try changing the ip switch to “-ip”.  If you’re trying this from a home server, make sure you use your internal network ip (such as 192.168.x.x).  You’d connect by trying the external ip (found at whatismyip.net).

If you’re ready to move on at this point (you connected ok, and know what the FPS of the server is), go ahead and leave your game, and CTRL-C to quit the server.  We’re going to need to make a file so that we can start the game server and have it run in the background (so you can exit the console).

cd /home/gameserver (don't forget to go to the folder you created if necessary)
nano server.sh
#!bin/sh
echo "Starting CS:Source Server"
sleep 1
screen -dms css-server -A ./srcds_run -console -game cstrike +map de_dust2 +ip xxx.xxx.xxx.xxx -port 27015 +maxplayers 24 -autoupdate -tickrate 100 -fps_max 0 -pidfile srcds-1.pid
save the file
chmod +x server.sh

Now, attempt to run the server.sh file (./server.sh) and reconnect to the server again.  Everything should be good to run full-time now.  You may want to kill the process until you’re ready to use it full time though.

kill srcds-1.pid

At this point, you at least have the game server running.  Now, you’ll want to tweak it to run at it’s most optimum.  This is where it starts getting tricky.  If you are NOT comfortable with linux at all, I suggest you stop here.

Server Tweaking

As I said earlier, the better FPS you have in First Person Shooters, the more “true” the game experience is.  If you started the server and didn’t achieve near 1000 fps, it’s time to do some tweaking.  Most server providers have a special setup of the OS’es (remember – setup for e-commerce applications), and as such, they run a high-latency kernel – which means that it’s not geared for performance, but stability.  This isn’t a good thing for game servers.  If you’re running a home server, chances are you have the “stock” kernel, which *should* allow you to get to 1000 fps.  I know that the newer versions of Ubuntu have a stock kernel that achieves the 1000 mark “out of the box” through the kernel timing.

yum -y update kernel

Chances are, this won’t get you a much better kernel speed timer.  But, it never hurts.  Restart the entire server, then start the game server and check the fps stats again.   If it doesn’t help.  Skip down to the section on Kernel Tweaking.  It’s REALLY advanced stuff.    For now, we’ll move on to some of the other steps.

crontab -e
*/5 * * * * renice -20 'cat /home/gameserver/srcds-1.pid' >/dev/null 2>&1
service crond restart

This is basically telling the system to check every 5 minutes to put the game server process into the 1st priority slot, using the most resources.

nano idler.c
int main()  {
  while(1);
}

gcc idler.c -o idler

nice ./idler

This is a little program to basically run in the background and help remove small variations (jitters) in the server fps.  Before running it, try heading out to http://www.fpsmeter.org and running a scan of your server while running to see how it looks.  If you get a lot of jittering, try running the idler after about 5-10 minutes.

Kernel Tweaking

If the yum kernel update didn’t help anything – guess what?  It’s time to compile a new kernel.  This is by far, the most dangerous thing you can do.  If you don’t compile it correctly, you run the risk of the system not booting up, and losing all your data.  Do this at your own risk!  Of course, we’ll put some safeguards in to make sure that we can get back.  But they don’t always work.

cd /usr/src
wget http://kernel.org/pub/linux/kernel/projects/rt/patch-2.6.26.8-rt16.gz
wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.26.8.tar.gz
tar zxf linux-2.6.26.8.tar.gz
cd linux-2.6.26.8
zcat ../patch-2.6.26.8-rt16.gz | patch -p1
cp -r /boot/config-'uname -r' .config
make menuconfig

You have two options when copying the config file (2nd to last step).  The first is listed, the alternate is to begin typing the “config” and then TAB to get the options.  I suggest (especially on dedicated servers you rent) using the earliest file there.  This comes in handy when you want to do multiple kernel compiles. When the menuconfig loads up, follow the steps below:

  • Processor type and features:
    • Disable Tickless System (Dynamic Ticks)
    • Enable High Resolution Timer Support
    • Select your processor under Processor family
    • Change Preemtion Mode to Complete Preemption (Real-Time)
    • Enable Enable priority boosting of RCU read-side critical sections (ignore if not present)
    • Disable Enable tracing for RCU – currently stats in debugfs (ignore if not present)
    • Enable Machine Check Exception and select Intel or AMD depending on your CPU
    • Change Timer frequency to 1000 HZ
  • General setup:
    • RCU Subsystem (ignore if not present)
      • Enable RCU Implementation (Preemptible RCU)
      • Disable Enable tracing for RCU
  • Power management and ACPI options
    • Enable Power Management support
    • Disable Power Management Debug Support
    • Disable Suspend to RAM and standby
    • Disable Hibernation (aka ‘suspend to disk’)
    • Enable ACPI (Advanced Configuration and Power Interface) Support
    • Disable CPU Frequency scaling
    • Disable CPU idle PM support
  • Networking support
    • Networking options
      • Enable Packet socket: mmapped IO
      • Optionally disable Network packet filtering framework (Netfilter) (Warning: this will disable your firewall!)
      • Disable QoS and/or fair queueing (Unless you need and use it…)
  • Device Drivers
    • Disable Watchdog Timer Support
    • Enable Real Time Clock
      • Enable PC-style ‘CMOS’
  • Kernel hacking
    • Disable everything
make
make modules
make modules_install
make install
grub
savedefault --default=0 --once
quit
restart

The line beginning with “grub” is your boot loading system.  We’re telling the system that we want to try out this new kernel configuration just once, then at the next reboot (or a failed boot), it will fall back to the original kernel.  If the new kernel works after reboot and it achieves the desired effect, then we’ll keep it with the following:

nano /boot/grub/grub.conf

Change the “default=1” to “default=0”.

For this kernel setup, we’re going to use a bit of a different rescheduler to check the system and make sure things are the highest priority.

nano /usr/local/sbin/resched.sh
#!/bin/sh

PIDS=`ps ax | grep sirq-hrtimer | grep -v grep | sed -e "s/^ *//" -e "s/ .*$//"`
for p in $PIDS; do
  chrt -f -p 99 $p
done

PIDS=`ps ax | grep sirq-timer | grep -v grep | sed -e "s/^ *//" -e "s/ .*$//"`
for p in $PIDS; do
 chrt -f -p 51 $p
done

PIDS=`pidof srcds_i686`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof srcds_i486`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof srcds_amd`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof hlds_i686`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof hlds_i486`
for p in $PIDS; do
  chrt -f -p 98 $p
done

PIDS=`pidof hlds_amd`
for p in $PIDS; do
  chrt -f -p 98 $p
done

chmod 755 /usr/local/sbin/resched.sh

nano /etc/crontab
*/5 * * * *  root  /usr/local/sbin/resched.sh > /dev/null 2>&1

service crond restart

Playing Around
Unfortunately, there’s no “one single way” to optimize performance of a server.  It involves a lot of testing, trial and error.  Each kernel recompile can take up to 30 minutes.  Every time you test, you’ll want to run both an empty and “load” fps test (using the fpsmeter.org utility).

I will suggest that you try different kernel patches as well, and also try different timer settings.  Especially if you don’t need to run a full 1000fps.  Try changing the fps_max to 600 (yields 500 max), which is usually considered a “decent” server.  With respect to trying different kernel patches, I’ve heard of others having decent success using the Zen-source patchset.  You’ll need to remove that rescheduler from the cron daemon if you do, though.Don’t forget to occasionally run a “top” command to see how your CPU usage is faring.  And don’t forget to reboot the server every now and then once you’re up and running.

Acknowledgments

Most of this tutorial has been prepared using guides from http://www.srcds.com and http://wiki.fragaholics.de/index.php/EN:Linux_Kernel_Optimization.  The srcds forums has a large user-group of dedicated server operators (and a few GSP’s) that will weigh in on problems.