Your server speed, cpu usage, memory usage, and overall Linux performance depends also on the amount of system resources your desktop environment is using. A lightweight desktop environment or a minimal desktop, will consume less resources and therefore make more resources available for serving purposes. Many system administrators argue that headless server is the best way to go, as a GUI desktop environment is only a convenience for newbies to make certain tasks easier. Nevertheless, I like to have an on-demand (therefore no little or know resource wastage when not needed) lightweight desktop environment on my server.
Recommended Guides for Ubuntu:
While I am not a big fan of what is happening in the Ubuntu world these days (Unity, termination of support for Kubuntu, etc.), I still run my Server on Ubuntu 10.04. Typically servers are headless but I like to have a minimalistic gui desktop installed in case of emergencies or for specific purposes.
There are several ways to install Ubuntu Desktop.
1. Full Ubuntu Desktop
sudo apt-get install ubuntu-desktop
This install the full-fledged Ubuntu Desktop with all its bells and whistles, which I normally would not use on a 24/7 server that is not used as a traditional desktop. It also uses more RAM and power, which is not energy efficient for a server.
2. Lightweight Ubuntu Desktop
sudo apt-get install --no-install-recommends ubuntu-desktop
This install a Ubuntu Desktop without all the bells and whistles of Ubuntu Desktop (i.e. without libreoffice, firefox, evolution, etc.). This is still an over kill for a server if you ask me. So I like to go the more minimalistic way listed below.
3. Lightweight Gnome Desktop
sudo apt-get install xorg gnome-core gnome-media gnome-system-monitor gnome-applets gnome-system-tools gnome-utils gnome-app-install sysv-rc-conf gdm
This will install a very minimal gnome just enough to boot the server into a graphical environment for emergency purposes. Then, run the following command (optional):
sudo sysv-rc-conf
You will be presented with a screen that shows services and what runlevels they are configured to start on. Find 'gdm' and uncheck the boxes for runlevels 2 and 3. If you want graphical login to be enabled then skip this step.
You will have to restart the server after the install. Now you can start the graphical Gnome desktop on a need basis by typing:
startx
After this, do not forget to update and upgrade:
sudo apt-get update sudo apt-get upgrade
This setup consumes minimal power and RAM and you still have a graphical environment to use when you need it.
Update: For a more recent guide (Ubuntu 12.04), please check this post.