Bootable OS

So one of the issues is that (at least for Solaris atm) you can't really boot off of RAID without a lot of effort... So, what if we use something like this to boot off of?

Learning Solaris: ZFS

ZFS Best Practices Guide

ZFS, Raid-Z, etc

Why is Raid-Z better than Raid5?
"10 Reasons" that ZFS is worth it.

And from the FAQ:

Q: Can I boot my system by using ZFS?
A: The initial release of ZFS does not support a bootable root file system. This functionality is currently under development and will be available in a future release.

Q: Can I use a single disk with ZFS?
A: Yes. With a single disk, you can do one of the following:
* Use your disk as a single device, in which case you cannot benefit from the recovery capabilities provided by a ZFS mirrored or RAID-Z configuration, but will get the greatest capacity out of your device.
* Split your disk into multiple partitions and use them to build a ZFS mirrored or RAID-Z based pool. This options allows you to benefit from all of the ZFS recovery capabilities (unless your disk suffers a total failure), but you will have a smaller capacity in your storage pool.

Q: Can I use ZFS on USB storage devices?
A:Yes, provided that your USB device looks like a block storage device and can support an EFI label.

....Overall, ZFS functions as designed with SAN devices, but if you expose more simple devices to ZFS, you can better leverage all available features....

Interesting analysis of building your own Raid-Z fileserver.

The OpenSolaris forum also has a couple threads on building a NAS box here and here. But then there is also this OpenSolaris NAS Appliance page (and appliances in general).

OpenSolaris

What is the difference between the OpenSolaris project and the Solaris Operating System?

The OpenSolaris Project consists of three key elements: (1) the OpenSolaris source code, (2) the OpenSolaris developer community, and (3) a website for collaboration: opensolaris.org. The Solaris OS, however, is Sun's operating system product, and future versions of Solaris will be based on technology from the OpenSolaris project. Solaris is available as a free binary download, and Sun offers service packages and regular updates. So, Solaris is a product supported by a company; OpenSolaris is a development project run by an community of developers.

There are several related items that need to be understood:

Solaris Express: This is a binary release for customers. It's Sun's official release of the OpenSolaris bits as well as additional technology that has not been released into the OpenSolaris source base. Sun offers limited support for this release. It's primarily intended for Solaris customers to try out the very latest technology that will eventually be productized by Sun. Solaris Express is updated monthly and is available as a free binary download.

Solaris Express Community Release: This is a binary release for developers. It's Sun's latest, unsupported release of the OpenSolaris bits as well as additional technology that has not been released into the OpenSolaris source base. Developers can build the OpenSolaris source by using this release as the base system. The release also has a code name -- Nevada -- and it's updated every two weeks.

OpenSolaris: This is the source base for Solaris development. Currently, it consists of several technology components called consolidations. See the Downloads Page for details on the technologies that have been released, and the OpenSolaris Roadmap for the schedule of upcoming technology releases. At present, the OpenSolaris source base is not enough to bootstrap an entire system, so developers start by downloading an OpenSolaris distribution and installing the OpenSolaris bits on top.

In Summary: Customers concerned about stability should use Solaris 10 since that's the company's officially-supported enterprise product. Customers interested in trying out the latest Solaris technologies within an official binary distribution are free to use Solaris Express. Developers working in the OpenSolaris community should use Solaris Express Community Release as the base on which to build the OpenSolaris source.

Zfs Demos

There are some excellent Zfs Demos here:
http://www.opensolaris.org/os/community/zfs/demos/basics/
http://www.opensolaris.org/os/community/zfs/demos/selfheal/

And some PDF Slides

Top of the line AMD Processors

I have not yet figured out what the difference between these are... except $2k.

Processor Details
CPU ID	Next-Generation AMD Opteron™ Processor Model 8220	Next-Generation AMD Opteron™ Processor Model 2220
Model	8220 SE	2220 SE
Ordering Parts Number (OPN)	OSY8220GAA6CR	OSY2220GAA6CQ
Stepping	F2
Frequency	2.8GHz
HT Speed	1000MHz
Integrated Memory Controller	2.8GHz
Core Voltage	1.325 V/1.375 V
Case Temperature	55°C to 69° C
Wattage	119.2W
L2 Cache Size	2 MB
L2 Cache Speed	2.8GHz
Manf. Technology	.09 micron SOI
Socket	Socket F (1207)
Amperage	86.9 A

Or we can go with the 2.4GHz and get the low-power version...

Processor Details
CPU ID	Next-Generation AMD Opteron™ Processor Model 8216	Next-Generation AMD Opteron™ Processor Model 2216
Model	8216 HE	2216 HE
Ordering Parts Number (OPN)	OSP8216GAA6CR	OSP2216CQWOF	OSP2216GAA6CQ
Stepping	F2
Frequency	2.4GHz
HT Speed	1000MHz
Integrated Memory Controller	2.4GHz
Core Voltage	1.20 V/1.25 V
Case Temperature	55°C to 72° C
Wattage	68.0W
L2 Cache Size	2 MB
L2 Cache Speed	2.4GHz
Manf. Technology	.09 micron SOI
Socket	Socket F (1207)
Amperage	53.3 A
	Remove Item	Remove Item	Remove Item

Server Ideas

Well, I realize that this plan might not perform the best... and might cost a bit... but....

I was thinking... what if we built a minimal Domain0 server... and then used Network Attached Storage (doing RAID5 or somesuch) for the DomainUs...

See, what I am thinking... A) Easier maintenance... B) Easier expandability (out of room, add another to the network). C) Less wasted hard drive space (if the NAS unit handles the RAID for me, since FreeBSD doesn't do RAID5 yet).

What about the old server machine you ask? Well, probably convert that to be the CEO's new machine, since she doesn't have one yet.

Xen Info

Vnets - Domain Virtual Networking

Unmodified Guest Domains

FAQ

Wiki

UnionFS on Xen

Consoles:
Econolism
XenMan

Xen Images:
Jailtime.org
Asterisk@HOME 2.8

Alternative Storage Mechanism?

One other idea we COULD do is have a minimal Xen domain0 server, then run all the domainU off of something like this. While slower, that would give us 1TB of RAID5... but we could easily use more than one ;) Thus easier to expand... not sure...

Xen: Current OS Compatibility

Xen 3.0

Operating System	Runs as Dom0 (host os)	Runs as DomU(guest os)
Linux 2.6	Yes	Yes
NetBSD 3.0	No	currently broken? Actively being worked on
FreeBSD 5.3	No	currently broken? Actively being worked on
FreeBSD 7-CURRENT	no	can be patched; works. see http://www.fsmware.com/xenofreebsd/7.0/
Plan 9	No	currently broken?
ReactOS	No	planned, development stalled
Solaris 10	Unknown	Yes
Un-Modified OS	No	Initial support for unmodified guests when using Intel VTX hardware, e.g. Windows

Virtualization Support in Athlon 64

Athlon 64 FX models

Windsor (90 nm SOI)

Dual-core CPU

• CPU-Stepping: F2
• L1-Cache: 64 + 64 KiB (Data + Instructions), per core
• L2-Cache: 1024 KiB fullspeed, per core
• MMX, Extended 3DNow!, SSE, SSE2, SSE3, AMD64, Cool'n'Quiet, NX Bit, AMD Virtualization
• Socket AM2, 1000 MHz HyperTransport (HT1000)
• VCore: 1.30 V - 1.35 V
• Power Consumption (TDP): 125 Watt max
• First Release: May 23, 2006
• Clockrate: 2800MHz (FX-62)

Windsor (90 nm SOI) - Quad FX platform

Dual-core, dual CPUs (four cores total)

• CPU-Stepping: F3
• L1-Cache: 64 + 64 KiB (Data + Instructions), per core
• L2-Cache: 1024 KiB fullspeed, per core
• MMX, Extended 3DNow!, SSE, SSE2, SSE3, AMD64, Cool'n'Quiet, NX Bit, AMD Virtualization
• Socket F (1207 FX), 1000 MHz HyperTransport (HT1000)
• VCore: 1.35 V - 1.40 V
• Power Consumption (TDP): 125 Watt max per CPU
• First Release: November 30, 2006
• Clockrate: 2600MHz (FX-70), 2800MHz (FX-72), 3000MHz (FX-74)

Athlon 64 models

Orleans (90 nm SOI)

• CPU-Stepping: F2
• L1-Cache: 64 + 64 KiB (Data + Instructions)
• L2-Cache: 512 KiB, fullspeed
• MMX, Extended 3DNow!, SSE, SSE2, SSE3, AMD64, Cool'n'Quiet, NX Bit, AMD Virtualization
• Socket AM2, 1000 MHz HyperTransport (HT1000)
• VCore: 1.35 V or 1.40 V
• Power Consumption (TDP): 62 Watt max
• First Release: May 23, 2006
• Clockrate: 2000 - 2400 MHz

AMD virtualization (AMD-V)

AMD's virtualization extension to the 64-bit x86 architecture is named AMD Virtualization (also known by the abbreviation AMD-V), and is sometimes referred to by the code name "Pacifica".

AMD processors using Socket AM2, Socket S1, and Socket F include AMD Virtualization support. In May 2006, AMD introduced such versions of the Athlon 64 and Turion 64 processors. AMD Virtualization is also supported by release two (x2xx series) of the Opteron processors.

Xen 3.0 Supports AMD-V/Pacifica

With hardware CPU virtualization as provided by Intel VT and AMD Pacifica technology, the ability to run an unmodified guest OS kernel is available. No porting of the OS is required, although some additional driver support is necessary within Xen itself. Unlike traditional full virtualization hypervisors, which suffer a tremendous performance overhead, the combination of Xen and VT or Xen and Pacifica technology complement one another to offer superb performance for para-virtualized guest operating systems and full support for unmodified guests running natively on the processor. Full support for VT and Pacifica chipsets will appear in early 2006.

.tcshrc prompt

I have found that it is very helpful to know which domain and user you are logged in as... this is especially important with things like panic: userret which seems to be happening when using the man pages from within a jail...

complete sysctl 'n/*/`sysctl -Na`/'

# Miscellaneous Info
set HOST = `uname -n`
set HTYPE = "HOST"
#set user = `whoami`

set color_dark="00"
set color_light="01"
set color_black="30m"
set color_red="31m"
set color_green="32m"
set color_yellow="33m"
set color_blue="34m"
set color_purple="35m"
set color_cyan="36m"
set color_white="37m"
# Color Format: %{^[[DARKORLIGHT;COLOR%}
set userColor = "%{^[[$color_dark;$color_green%}"
set hostColor = "%{^[[$color_dark;$color_blue%}"
set pathColor = "%{^[[$color_dark;$color_cyan%}"
set resetColor = "%{^[[00m%}"

#set prompt="%{^[[01;31m%} `whoami`%{^[[01;37m%} %c %{^[[01;32m}%#%{^[[00m%} "

# Macros
alias setprompt 'set prompt="\n[$pathColor$HTYPE$resetColor]$hostColor$HOST$resetColor\:$userColor`whoami`$resetColor($pathColor$cwd$resetColor)\n>"'
alias cd 'cd \!* && setprompt'
alias pushd 'pushd \!* && setprompt'
alias popd 'popd \!* && setprompt'
alias ls 'ls -G'
setprompt

You should customize the non-highlighted part first, copy/paste this file and change the highlighted part per user and per domain.... for example:

User on Host (Above Example)
TYPE	HOST
userColor	$color_dark;$color_green
hostColor	$color_dark;$color_blue
[HOST]serveris.eoti.org:malachi(/home/malachi) >
Root on Host
TYPE	HOST
userColor	$color_dark;$color_red
hostColor	$color_dark;$color_blue
[HOST]serveris.eoti.org:root(/home/malachi) >
User in Jail
TYPE	JAIL
userColor	$color_dark;$color_green
hostColor	$color_dark;$color_purple
[JAIL]serveris.eoti.org:malachi(/home/malachi) >
Root in Jail
TYPE	JAIL
userColor	$color_dark;$color_red
hostColor	$color_dark;$color_purple
[JAIL]serveris.eoti.org:root(/home/malachi) >

Virtual Machines

Though VMWare is free (and has a nice interface), I lean towards Open Source rather than just free. Currently the plan is Xen on FreeBSD... but Slashdot linked to the new KVM if we decide to go with Linux instead.

nve0: device timeout (1)

Besides just seeing the message, the jail lost ping, and died during ssh...

nve0: device timeout (1)
nve0: link state changed to DOWN
nve0: link state changed to UP

Problem reported and patch available here, but instead...

cd /usr/src/sys/dev/nve
ftp sources.freebsd.org (anonymous,email,etc)
cd /pub/FreeBSD/sources/RELENG_6/src/sys/dev/nve/
mget *

cd /usr/src
make -DNOCLEAN buildkernel KERNCONF=CUSTOM
make installkernel KERNCONF=CUSTOM

panic: userret: Returning with 1 locks held.

Crash when trying to run 'man' from inside jail

Formatting page, please wait...
  

panic: userret: Returning with 1 locks held.
cpuid=0
KDB: enter: panic
[thread pid 744 tid 100073]
Stopped at kdb_enter+0x2f: nop
db>

Seems to be another lockmgr/unionfs problem.
Solution unknown.

panic: mutex Giant not owned at /usr/src/sys/kern/vfs_subr.c: 2031

panic: mutex Giant not owned at /usr/src/sys/kern/vfs_subr.c: 2031
cpuid=0
KDB: enter: panic
[thread pid 3066 tid 100111]
Stopped at kdb_enter+0x2f: nop
db>

Reportedly starting around 2/1/06.
Solution unknown

Jail SSH fails

This message can be seen when running /etc/rc in the shell (or looking at /var/log/messages)

May 12 20:26:53 serveris sshd[753]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:26:53 serveris sshd[753]: error: @         WARNING: UNPROTECTED PRIVATE KEY FILE!    @
May 12 20:26:53 serveris sshd[753]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:26:53 serveris sshd[753]: error: Permissions 0755 for '/etc/ssh/ssh_host_dsa_key' are too open.
May 12 20:26:53 serveris sshd[753]: error: It is recommended that your private key files are NOTaccessible by others.
May 12 20:26:53 serveris sshd[753]: error: This private key will be ignored.
May 12 20:26:53 serveris sshd[753]: error: bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key
May 12 20:26:53 serveris sshd[753]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key
May 12 20:27:03 serveris sshd[755]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:27:03 serveris sshd[755]: error: @         WARNING: UNPROTECTED PRIVATE KEY FILE!    @
May 12 20:27:03 serveris sshd[755]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:27:03 serveris sshd[755]: error: Permissions 0755 for '/etc/ssh/ssh_host_dsa_key' are too open.
May 12 20:27:03 serveris sshd[755]: error: It is recommended that your private key files are NOT accessible by others.
May 12 20:27:03 serveris sshd[755]: error: This private key will be ignored.
May 12 20:27:03 serveris sshd[755]: error: bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key
May 12 20:27:03 serveris sshd[755]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key

This is caused by the jail trying to use the hosts' SSH keys that were unionfs'd in.

jail> rm /etc/ssh/ssh_host*
jail> rm /etc/ssh/moduli
jail> sh /etc/rc

This will create new ssh keys. We run /etc/rc explicitely because entropy will ask that you type for 30 seconds.

Jail ifconfig shows no IP address

One problem I encountered was that the jail didn't have any ip's listed under ifconfig. The solution for that was to make sure to do the ifconfig_nve0_alias0 in /etc/rc.conf instead of /jail/rc.conf or /jail/eoti.org/rc.conf. I believe it was caused by the ip's being assigned before the /jail partition was mounted.

unionfs-- panic: lockmgr: locking against myself

Error looks like this:

panic: lockmgr: locking against myself
cpuid = 0
KBD: enter: panic
[thread pid 74167 tid 100077]
Stopped at kdb_enter+02f: nop
db>

I found this problem related to kern/84107: unionfs related panic.
Solution:

# patch < /anywhere/unionfs-p8.diff
# cp /usr/src/sys/fs/unionfs/union.h /usr/include/fs/unionfs
# cd /usr/src/sbin/mount_unionfs/
# make obj
# make depend
# make all
# make install
# make clean
# cd /usr/src
# make buildkernel KERNCONF=CUSTOM
# make installkernel KERNCONF=CUSTOM
# shutdown -r now
* Note: SERVERIS: /root/patches/unionfs/unionfs-p8.diff

  

* Note: SERVERIS: /root/patches/unionfs/unionfs-p8.diff

Welcome

This blog is to archive notes, steps taken, kernel fixes, etc... Currently, it is focused on FreeBSD since that is what we are trying to use - but that may change based on functionality...