Thoughts... ie: What should I do?

So as I investigate things like Sun servers, OpenSolaris, ZFS/Raid-Z, iSCSI, Embedded NetBSD, etc etc... I start thinking that perhas a better approach would be to start off with a list of requirements... It is a little late to "start" there, but, what the hell...

* We do not want a repeat of the data wipeout we got a year ago... As such, if one domain gets hacked, it can not take down the rest of the domains.
- I had been planning on using Jail to alleviate this problem, but at this point I think our best bet is Xen
- While many OSs support Xen domU, many fewer support Xen dom0.

* Also, if a domain gets hacked and wiped out, we need a backup to restore from.
- ZFS Snapshots (maybe full weekly and incremental daily) seems like a good solution for this
- SVN might be a good solution to this
- It might be good if there was a way to automate this (ie: allow the root user to do it instead of emailing me)

* We want to get disk redundancy, while keeping cost and maintenance low.
- RAID-0 gives us the redundancy, but at a fairly high cost. The current 1TB gives us less than 500GB this way.
- RAID-5 could be acceptable, as long as we could fix the system if the root partition were the one to screw up
- ZFS and Raid-Z seem like excellent options to provide the redundancy and automatic corrections

* Each domain should be able to have it's own root user and OS.
- Jail could have provided for this, but seems very buggy ('man man' in a jail crashed the entire OS)
- Xen should allow for this fairly easy. As such, the base OS really doesn't do anything other than launch the domUs.

* If multiple OSs share a base OS, it would be nice (though not required) if we were able to not waste a lot of duplicated space
- Unionfs is ideal for this. Not sure how easy it is to use with Xen
- At some point, it has to be better to have a full copy (like if they upgraded every single app)

* It would be preferable if all OS's could run unmodified, thus allowing us greater flexibility and choices
- Intel VT or AMD-V technology. If we are planning on doing any RSA work on any domain, it can't be the Intel option. So probably an Opteron 2xxx (or 2).

* If we run low on drive space, it should be fairly simple to add more
- ZFS is supposed to allow for this with 'zfs add'
- We *could* do separate drives for each domain, but that would waste a lot of drive space
- iSCSI might be a better option than internal drives for this. Instead of running out of hard drive bays, we'd just have to worry about running out of ethernet ports -- which realistically, is much easier to come up with more of. ** SEE BELOW

** iSCSI Thoughts:
- IDEALLY, I would have a set of cheap 250GB or 500GB drives that host themselves as iSCSI Targets. Then, adding new drives is literally adding another drive to the network.
- We could, in theory, run Solaris or NetBSD to easily provide iSCSI Targets to the network - but then we have to wonder why we are creating a separate box to host Xen
- If we have multiple iSCSI Targets, which our Solaris(?) box were to RAID-Z together, we could also have extra iSCSI Targets that VMWare and/or other hardware could use directly (even Windows)
- If we have multiple iSCSI Targets, which our Solaris(?) box were to RAID-Z together, the Solaris box could provide the pool as a iSCSCI Target to other devices/OSes
- If we were to have Solaris running RAID-Z directly on the iSCSI Target (ie: providing ZFS pools as iSCSI Targets), then separate Xen domUs could point to these iSCSI Targets for their primary data storage and POTENTIALLY get the ZFS benefits regardless of their OS (snapshots, error-correction [at least from hardware errors], etc)
- In theory, if we could come up with mini embeddable systems that could boot from iSCSI Targets, then we could do away with Xen entirely and just have a little device on the network for each domain... this might be overkill
- Note: Can Solaris boot off iSCSI?

EMail conversation with Sun

I received this email....

Hi Malachi,

Thank you for visiting the web chat forum on the Sun website yesterday, when you expressed interest in the Try and Buy promotion being currently run by Sun.

You mentioned that you are deciding which machine you will trial on the promotion - excellent! You will see the full list of products on the promotion at www.sun.com/tryandbuy

You can email me with any questions that you may have, or you can visit our team at the web chat forum. Here's some additional information on the Try and Buy promotion, which you may find useful:

When you receive the Server on the Try and Buy promotion you will also receive a Welcome Pack.
Inside the Welcome Pack you will find-
1. A Quick Guide to Installation.
2. Access to Tuning and Optimization documentation.
3. Access to an online portal and forum for system and application tuning hints and tips.
4. Access to Sun's performance engineers.
5. All of the latest patches and configuration files for the system on the portal.
Additional options for purchase are available for system ready and business ready services.
The Solaris 10 Operating System and other software provided with the system is covered under the Warranty support during the Try and Buy period.


Best Regards

Mark Cradock
Sun Microsystems
Tel: + 353 (0)599136768
mark.cradock@your-sun.com
www.sun.com

My response:

Hi Mark,

I am debating the Sun Fire X4200 or the Sun Ultra 40 Workstation. Perhaps it would be better for me to explain the intended usage and get your feedback.

First, a little background. We had a FreeBSD box running for about 4 years without reboot. As is probably obvious from that statement, we were behind on the security updates. While we were hosting multiple domains, they were not in Xen or Jail or anything. On Christmas eve a year ago, a hacker from a Polish cable isp hacked in through one of the user accounts and wiped out the entire system. Because of this, I decided this time that I wanted to ensure that the various domains stay secure even if one is compromised.

We bought the latest top-of-the-line Asus, AMD, etc. Unfortunately, AMD lied about the capabilities of the board. They said it was capable of RAID-5 on SATAII, but in actuality it is capable of RAID-5 OR SATAII. Then we installed FreeBSD-CURRENT, and found out it doesn't support RAID-5 yet. Needless to say, it has been a hassle.

So we are debating buying a Sun server and running Solaris on it. My expectation is that there should not be any driver compatibility problems at that point.

So long story not so short -- we are looking at trying to run a bootable RAID-Z (yes, I know that requires some tweaking, but I want to ensure that the boot is also recoverable) with a Xen Dom0. On top of that, we want to allow each domain to run its own OS (whichever they choose, thus prefer AMD-V) as a DomU. Based on this, I think that each DomU would get the advantage of Raid-Z on the underlying filesystem, even if they didn't know about it directly, and regardless of which OS they are running.


Do you have any thoughts, concerns, or questions?

Malachi

A followup:

Hi Malachi,

First of all, sorry to hear about the experiences you've had with the FreeBSD and Asus! Not good.

Secondly, one of the reasons for the Try and Buy promotion is that you can test the capabilities of each machine prior to actually buying it. I'm sure that you will have a positive experience and benefit from this.

I would recommend that we assign a server specialist to you, that can be available prior to your choice of machine, and also be available for you right through the trial process to assist you in your testing, additional components, adjustments in configurations, etc. Would you have a
contact number that we can call you on?

Best Regards
Mark Cradock
Sun Microsystems
Tel: +
353
(0)599136768
mark.cradock@your-sun.com
www.sun.com

So the rest will most likely be via phone.

.tcshrc prompt

I have found that it is very helpful to know which domain and user you are logged in as... this is especially important with things like panic: userret which seems to be happening when using the man pages from within a jail...

complete sysctl 'n/*/`sysctl -Na`/'

# Miscellaneous Info
set HOST = `uname -n`
set HTYPE = "HOST"
#set user = `whoami`

set color_dark="00"
set color_light="01"
set color_black="30m"
set color_red="31m"
set color_green="32m"
set color_yellow="33m"
set color_blue="34m"
set color_purple="35m"
set color_cyan="36m"
set color_white="37m"
# Color Format: %{^[[DARKORLIGHT;COLOR%}
set userColor = "%{^[[$color_dark;$color_green%}"
set hostColor = "%{^[[$color_dark;$color_blue%}"
set pathColor = "%{^[[$color_dark;$color_cyan%}"
set resetColor = "%{^[[00m%}"

#set prompt="%{^[[01;31m%} `whoami`%{^[[01;37m%} %c %{^[[01;32m}%#%{^[[00m%} "

# Macros
alias setprompt 'set prompt="\n[$pathColor$HTYPE$resetColor]$hostColor$HOST$resetColor\:$userColor`whoami`$resetColor($pathColor$cwd$resetColor)\n>"'
alias cd 'cd \!* && setprompt'
alias pushd 'pushd \!* && setprompt'
alias popd 'popd \!* && setprompt'
alias ls 'ls -G'
setprompt

You should customize the non-highlighted part first, copy/paste this file and change the highlighted part per user and per domain.... for example:

User on Host (Above Example)
TYPE	HOST
userColor	$color_dark;$color_green
hostColor	$color_dark;$color_blue
[HOST]serveris.eoti.org:malachi(/home/malachi) >
Root on Host
TYPE	HOST
userColor	$color_dark;$color_red
hostColor	$color_dark;$color_blue
[HOST]serveris.eoti.org:root(/home/malachi) >
User in Jail
TYPE	JAIL
userColor	$color_dark;$color_green
hostColor	$color_dark;$color_purple
[JAIL]serveris.eoti.org:malachi(/home/malachi) >
Root in Jail
TYPE	JAIL
userColor	$color_dark;$color_red
hostColor	$color_dark;$color_purple
[JAIL]serveris.eoti.org:root(/home/malachi) >

nve0: device timeout (1)

Besides just seeing the message, the jail lost ping, and died during ssh...

nve0: device timeout (1)
nve0: link state changed to DOWN
nve0: link state changed to UP

Problem reported and patch available here, but instead...

cd /usr/src/sys/dev/nve
ftp sources.freebsd.org (anonymous,email,etc)
cd /pub/FreeBSD/sources/RELENG_6/src/sys/dev/nve/
mget *

cd /usr/src
make -DNOCLEAN buildkernel KERNCONF=CUSTOM
make installkernel KERNCONF=CUSTOM

panic: userret: Returning with 1 locks held.

Crash when trying to run 'man' from inside jail

Formatting page, please wait...
  

panic: userret: Returning with 1 locks held.
cpuid=0
KDB: enter: panic
[thread pid 744 tid 100073]
Stopped at kdb_enter+0x2f: nop
db>

Seems to be another lockmgr/unionfs problem.
Solution unknown.

panic: mutex Giant not owned at /usr/src/sys/kern/vfs_subr.c: 2031

panic: mutex Giant not owned at /usr/src/sys/kern/vfs_subr.c: 2031
cpuid=0
KDB: enter: panic
[thread pid 3066 tid 100111]
Stopped at kdb_enter+0x2f: nop
db>

Reportedly starting around 2/1/06.
Solution unknown

Jail SSH fails

This message can be seen when running /etc/rc in the shell (or looking at /var/log/messages)

May 12 20:26:53 serveris sshd[753]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:26:53 serveris sshd[753]: error: @         WARNING: UNPROTECTED PRIVATE KEY FILE!    @
May 12 20:26:53 serveris sshd[753]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:26:53 serveris sshd[753]: error: Permissions 0755 for '/etc/ssh/ssh_host_dsa_key' are too open.
May 12 20:26:53 serveris sshd[753]: error: It is recommended that your private key files are NOTaccessible by others.
May 12 20:26:53 serveris sshd[753]: error: This private key will be ignored.
May 12 20:26:53 serveris sshd[753]: error: bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key
May 12 20:26:53 serveris sshd[753]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key
May 12 20:27:03 serveris sshd[755]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:27:03 serveris sshd[755]: error: @         WARNING: UNPROTECTED PRIVATE KEY FILE!    @
May 12 20:27:03 serveris sshd[755]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:27:03 serveris sshd[755]: error: Permissions 0755 for '/etc/ssh/ssh_host_dsa_key' are too open.
May 12 20:27:03 serveris sshd[755]: error: It is recommended that your private key files are NOT accessible by others.
May 12 20:27:03 serveris sshd[755]: error: This private key will be ignored.
May 12 20:27:03 serveris sshd[755]: error: bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key
May 12 20:27:03 serveris sshd[755]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key

This is caused by the jail trying to use the hosts' SSH keys that were unionfs'd in.

jail> rm /etc/ssh/ssh_host*
jail> rm /etc/ssh/moduli
jail> sh /etc/rc

This will create new ssh keys. We run /etc/rc explicitely because entropy will ask that you type for 30 seconds.

Jail ifconfig shows no IP address

One problem I encountered was that the jail didn't have any ip's listed under ifconfig. The solution for that was to make sure to do the ifconfig_nve0_alias0 in /etc/rc.conf instead of /jail/rc.conf or /jail/eoti.org/rc.conf. I believe it was caused by the ip's being assigned before the /jail partition was mounted.