What kind of things were you planning on doing with the computer? And did you plan on installing anything other than a server OS on the system?
I don't know how xeons work for gaming. In theory it'd be good because those are powerful but my understanding is that the architecture is more focused to data delivery and processing, and not the calculations needed by a gaming machine. I tried doing a search to prove my understanding one way or the other and the best I found was this: Intel® Xeon® Processors Overview
That said, it would be great for photo/music editing or as word editor.
Also, I tend to find server OS's to be limiting when trying to do every day tasks until you disable a lot of the security checks (assuming Windows). I would highly recommend using an operating system not meant for serving if you plan on using it as a standard workstation.