18 December 2006
Bootable OS
So one of the issues is that (at least for Solaris atm) you can't really boot off of RAID without a lot of effort... So, what if we use something like this to boot off of?
ZFS, Raid-Z, etc
Why is Raid-Z better than Raid5?
"10 Reasons" that ZFS is worth it.
And from the FAQ:
Q: Can I boot my system by using ZFS?
A: The initial release of ZFS does not support a bootable root file system. This functionality is currently under development and will be available in a future release.
Q: Can I use a single disk with ZFS?
A: Yes. With a single disk, you can do one of the following:
* Use your disk as a single device, in which case you cannot benefit from the recovery capabilities provided by a ZFS mirrored or RAID-Z configuration, but will get the greatest capacity out of your device.
* Split your disk into multiple partitions and use them to build a ZFS mirrored or RAID-Z based pool. This options allows you to benefit from all of the ZFS recovery capabilities (unless your disk suffers a total failure), but you will have a smaller capacity in your storage pool.
Q: Can I use ZFS on USB storage devices?
A:Yes, provided that your USB device looks like a block storage device and can support an EFI label.
....Overall, ZFS functions as designed with SAN devices, but if you expose more simple devices to ZFS, you can better leverage all available features....
Interesting analysis of building your own Raid-Z fileserver.
The OpenSolaris forum also has a couple threads on building a NAS box here and here. But then there is also this OpenSolaris NAS Appliance page (and appliances in general).
"10 Reasons" that ZFS is worth it.
And from the FAQ:
Q: Can I boot my system by using ZFS?
A: The initial release of ZFS does not support a bootable root file system. This functionality is currently under development and will be available in a future release.
Q: Can I use a single disk with ZFS?
A: Yes. With a single disk, you can do one of the following:
* Use your disk as a single device, in which case you cannot benefit from the recovery capabilities provided by a ZFS mirrored or RAID-Z configuration, but will get the greatest capacity out of your device.
* Split your disk into multiple partitions and use them to build a ZFS mirrored or RAID-Z based pool. This options allows you to benefit from all of the ZFS recovery capabilities (unless your disk suffers a total failure), but you will have a smaller capacity in your storage pool.
Q: Can I use ZFS on USB storage devices?
A:Yes, provided that your USB device looks like a block storage device and can support an EFI label.
....Overall, ZFS functions as designed with SAN devices, but if you expose more simple devices to ZFS, you can better leverage all available features....
Interesting analysis of building your own Raid-Z fileserver.
The OpenSolaris forum also has a couple threads on building a NAS box here and here. But then there is also this OpenSolaris NAS Appliance page (and appliances in general).
17 December 2006
OpenSolaris
What is the difference between the OpenSolaris project and the Solaris Operating System?
The OpenSolaris Project consists of three key elements: (1) the OpenSolaris source code, (2) the OpenSolaris developer community, and (3) a website for collaboration: opensolaris.org. The Solaris OS, however, is Sun's operating system product, and future versions of Solaris will be based on technology from the OpenSolaris project. Solaris is available as a free binary download, and Sun offers service packages and regular updates. So, Solaris is a product supported by a company; OpenSolaris is a development project run by an community of developers.
There are several related items that need to be understood:
Solaris Express: This is a binary release for customers. It's Sun's official release of the OpenSolaris bits as well as additional technology that has not been released into the OpenSolaris source base. Sun offers limited support for this release. It's primarily intended for Solaris customers to try out the very latest technology that will eventually be productized by Sun. Solaris Express is updated monthly and is available as a free binary download.
Solaris Express Community Release: This is a binary release for developers. It's Sun's latest, unsupported release of the OpenSolaris bits as well as additional technology that has not been released into the OpenSolaris source base. Developers can build the OpenSolaris source by using this release as the base system. The release also has a code name -- Nevada -- and it's updated every two weeks.
OpenSolaris: This is the source base for Solaris development. Currently, it consists of several technology components called consolidations. See the Downloads Page for details on the technologies that have been released, and the OpenSolaris Roadmap for the schedule of upcoming technology releases. At present, the OpenSolaris source base is not enough to bootstrap an entire system, so developers start by downloading an OpenSolaris distribution and installing the OpenSolaris bits on top.
In Summary: Customers concerned about stability should use Solaris 10 since that's the company's officially-supported enterprise product. Customers interested in trying out the latest Solaris technologies within an official binary distribution are free to use Solaris Express. Developers working in the OpenSolaris community should use Solaris Express Community Release as the base on which to build the OpenSolaris source.
16 December 2006
Zfs Demos
There are some excellent Zfs Demos here:
http://www.opensolaris.org/os/community/zfs/demos/basics/
http://www.opensolaris.org/os/community/zfs/demos/selfheal/
And some PDF Slides
http://www.opensolaris.org/os/community/zfs/demos/basics/
http://www.opensolaris.org/os/community/zfs/demos/selfheal/
And some PDF Slides
Top of the line AMD Processors
I have not yet figured out what the difference between these are... except $2k.
Or we can go with the 2.4GHz and get the low-power version...
| Processor Details | ||
| CPU ID | Next-Generation AMD Opteron™ Processor Model 8220 | Next-Generation AMD Opteron™ Processor Model 2220 |
| Model | 8220 SE | 2220 SE |
| Ordering Parts Number (OPN) | OSY8220GAA6CR | OSY2220GAA6CQ |
| Stepping | F2 | |
| Frequency | 2.8GHz | |
| HT Speed | 1000MHz | |
| Integrated Memory Controller | 2.8GHz | |
| Core Voltage | 1.325 V/1.375 V | |
| Case Temperature | 55°C to 69° C | |
| Wattage | 119.2W | |
| L2 Cache Size | 2 MB | |
| L2 Cache Speed | 2.8GHz | |
| Manf. Technology | .09 micron SOI | |
| Socket | Socket F (1207) | |
| Amperage | 86.9 A | |
Or we can go with the 2.4GHz and get the low-power version...
| Processor Details | |||
| CPU ID | Next-Generation AMD Opteron™ Processor Model 8216 | Next-Generation AMD Opteron™ Processor Model 2216 | |
| Model | 8216 HE | 2216 HE | |
| Ordering Parts Number (OPN) | OSP8216GAA6CR | OSP2216CQWOF | OSP2216GAA6CQ |
| Stepping | F2 | ||
| Frequency | 2.4GHz | ||
| HT Speed | 1000MHz | ||
| Integrated Memory Controller | 2.4GHz | ||
| Core Voltage | 1.20 V/1.25 V | ||
| Case Temperature | 55°C to 72° C | ||
| Wattage | 68.0W | ||
| L2 Cache Size | 2 MB | ||
| L2 Cache Speed | 2.4GHz | ||
| Manf. Technology | .09 micron SOI | ||
| Socket | Socket F (1207) | ||
| Amperage | 53.3 A | ||
| Remove Item | Remove Item | Remove Item | |
Server Ideas
Well, I realize that this plan might not perform the best... and might cost a bit... but....
I was thinking... what if we built a minimal Domain0 server... and then used Network Attached Storage (doing RAID5 or somesuch) for the DomainUs...
See, what I am thinking... A) Easier maintenance... B) Easier expandability (out of room, add another to the network). C) Less wasted hard drive space (if the NAS unit handles the RAID for me, since FreeBSD doesn't do RAID5 yet).
What about the old server machine you ask? Well, probably convert that to be the CEO's new machine, since she doesn't have one yet.
I was thinking... what if we built a minimal Domain0 server... and then used Network Attached Storage (doing RAID5 or somesuch) for the DomainUs...
See, what I am thinking... A) Easier maintenance... B) Easier expandability (out of room, add another to the network). C) Less wasted hard drive space (if the NAS unit handles the RAID for me, since FreeBSD doesn't do RAID5 yet).
What about the old server machine you ask? Well, probably convert that to be the CEO's new machine, since she doesn't have one yet.
15 December 2006
Alternative Storage Mechanism?
One other idea we COULD do is have a minimal Xen domain0 server, then run all the domainU off of something like this. While slower, that would give us 1TB of RAID5... but we could easily use more than one ;) Thus easier to expand... not sure...
14 December 2006
Xen: Current OS Compatibility
Xen 3.0
Operating System | Runs as Dom0 (host os) | Runs as DomU(guest os) |
Linux 2.6 | Yes | Yes |
NetBSD 3.0 | No | currently broken? Actively being worked on |
FreeBSD 5.3 | No | currently broken? Actively being worked on |
FreeBSD 7-CURRENT | no | can be patched; works. see http://www.fsmware.com/xenofreebsd/7.0/ |
Plan 9 | No | currently broken? |
ReactOS | No | planned, development stalled |
Solaris 10 | Unknown | Yes |
Un-Modified OS | No | Initial support for unmodified guests when using Intel VTX hardware, e.g. Windows |
Virtualization Support in Athlon 64
Athlon 64 FX models
Windsor (90 nm SOI)
Dual-core CPU
- CPU-Stepping: F2
- L1-Cache: 64 + 64 KiB (Data + Instructions), per core
- L2-Cache: 1024 KiB fullspeed, per core
- MMX, Extended 3DNow!, SSE, SSE2, SSE3, AMD64, Cool'n'Quiet, NX Bit, AMD Virtualization
- Socket AM2, 1000 MHz HyperTransport (HT1000)
- VCore: 1.30 V - 1.35 V
- Power Consumption (TDP): 125 Watt max
- First Release: May 23, 2006
- Clockrate: 2800MHz (FX-62)
Windsor (90 nm SOI) - Quad FX platform
Dual-core, dual CPUs (four cores total)
- CPU-Stepping: F3
- L1-Cache: 64 + 64 KiB (Data + Instructions), per core
- L2-Cache: 1024 KiB fullspeed, per core
- MMX, Extended 3DNow!, SSE, SSE2, SSE3, AMD64, Cool'n'Quiet, NX Bit, AMD Virtualization
- Socket F (1207 FX), 1000 MHz HyperTransport (HT1000)
- VCore: 1.35 V - 1.40 V
- Power Consumption (TDP): 125 Watt max per CPU
- First Release: November 30, 2006
- Clockrate: 2600MHz (FX-70), 2800MHz (FX-72), 3000MHz (FX-74)
Athlon 64 models
Orleans (90 nm SOI)
- CPU-Stepping: F2
- L1-Cache: 64 + 64 KiB (Data + Instructions)
- L2-Cache: 512 KiB, fullspeed
- MMX, Extended 3DNow!, SSE, SSE2, SSE3, AMD64, Cool'n'Quiet, NX Bit, AMD Virtualization
- Socket AM2, 1000 MHz HyperTransport (HT1000)
- VCore: 1.35 V or 1.40 V
- Power Consumption (TDP): 62 Watt max
- First Release: May 23, 2006
- Clockrate: 2000 - 2400 MHz
AMD virtualization (AMD-V)
AMD's virtualization extension to the 64-bit x86 architecture is named AMD Virtualization (also known by the abbreviation AMD-V), and is sometimes referred to by the code name "Pacifica".
AMD processors using Socket AM2, Socket S1, and Socket F include AMD Virtualization support. In May 2006, AMD introduced such versions of the Athlon 64 and Turion 64 processors. AMD Virtualization is also supported by release two (x2xx series) of the Opteron processors.
Xen 3.0 Supports AMD-V/Pacifica
With hardware CPU virtualization as provided by Intel VT and AMD Pacifica technology, the ability to run an unmodified guest OS kernel is available. No porting of the OS is required, although some additional driver support is necessary within Xen itself. Unlike traditional full virtualization hypervisors, which suffer a tremendous performance overhead, the combination of Xen and VT or Xen and Pacifica technology complement one another to offer superb performance for para-virtualized guest operating systems and full support for unmodified guests running natively on the processor. Full support for VT and Pacifica chipsets will appear in early 2006.
12 December 2006
.tcshrc prompt
I have found that it is very helpful to know which domain and user you are logged in as... this is especially important with things like panic: userret which seems to be happening when using the man pages from within a jail...
complete sysctl 'n/*/`sysctl -Na`/'You should customize the non-highlighted part first, copy/paste this file and change the highlighted part per user and per domain.... for example:
# Miscellaneous Info
set HOST = `uname -n`
set HTYPE = "HOST"
#set user = `whoami`
set color_dark="00"
set color_light="01"
set color_black="30m"
set color_red="31m"
set color_green="32m"
set color_yellow="33m"
set color_blue="34m"
set color_purple="35m"
set color_cyan="36m"
set color_white="37m"
# Color Format: %{^[[DARKORLIGHT;COLOR%}
set userColor = "%{^[[$color_dark;$color_green%}"
set hostColor = "%{^[[$color_dark;$color_blue%}"
set pathColor = "%{^[[$color_dark;$color_cyan%}"
set resetColor = "%{^[[00m%}"
#set prompt="%{^[[01;31m%} `whoami`%{^[[01;37m%} %c %{^[[01;32m}%#%{^[[00m%} "
# Macros
alias setprompt 'set prompt="\n[$pathColor$HTYPE$resetColor]$hostColor$HOST$resetColor\:$userColor`whoami`$resetColor($pathColor$cwd$resetColor)\n>"'
alias cd 'cd \!* && setprompt'
alias pushd 'pushd \!* && setprompt'
alias popd 'popd \!* && setprompt'
alias ls 'ls -G'
setprompt
| User on Host (Above Example) | |
|---|---|
| TYPE | HOST |
| userColor | $color_dark;$color_green |
| hostColor | $color_dark;$color_blue |
[HOST]serveris.eoti.org:malachi(/home/malachi) | |
| Root on Host | |
| TYPE | HOST |
| userColor | $color_dark;$color_red |
| hostColor | $color_dark;$color_blue |
[HOST]serveris.eoti.org:root(/home/malachi) | |
| User in Jail | |
| TYPE | JAIL |
| userColor | $color_dark;$color_green |
| hostColor | $color_dark;$color_purple |
[JAIL]serveris.eoti.org:malachi(/home/malachi) | |
| Root in Jail | |
| TYPE | JAIL |
| userColor | $color_dark;$color_red |
| hostColor | $color_dark;$color_purple |
[JAIL]serveris.eoti.org:root(/home/malachi) | |
Virtual Machines
nve0: device timeout (1)
Besides just seeing the message, the jail lost ping, and died during ssh...
Problem reported and patch available here, but instead...
nve0: device timeout (1)
nve0: link state changed to DOWN
nve0: link state changed to UP
Problem reported and patch available here, but instead...
cd /usr/src/sys/dev/nve
ftp sources.freebsd.org (anonymous,email,etc)
cd /pub/FreeBSD/sources/RELENG_6/src/sys/dev/nve/
mget *
cd /usr/src
make -DNOCLEAN buildkernel KERNCONF=CUSTOM
make installkernel KERNCONF=CUSTOM
panic: userret: Returning with 1 locks held.
Crash when trying to run 'man' from inside jail
Formatting page, please wait...
panic: userret: Returning with 1 locks held.
cpuid=0
KDB: enter: panic
[thread pid 744 tid 100073]
Stopped at kdb_enter+0x2f: nop
db>
Seems to be another lockmgr/unionfs problem.
Solution unknown.
panic: mutex Giant not owned at /usr/src/sys/kern/vfs_subr.c: 2031
panic: mutex Giant not owned at /usr/src/sys/kern/vfs_subr.c: 2031
cpuid=0
KDB: enter: panic
[thread pid 3066 tid 100111]
Stopped at kdb_enter+0x2f: nop
db>
Reportedly starting around 2/1/06.
Solution unknown
Jail SSH fails
This message can be seen when running /etc/rc in the shell (or looking at /var/log/messages)
This is caused by the jail trying to use the hosts' SSH keys that were unionfs'd in.
This will create new ssh keys. We run /etc/rc explicitely because entropy will ask that you type for 30 seconds.
May 12 20:26:53 serveris sshd[753]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:26:53 serveris sshd[753]: error: @ WARNING: UNPROTECTED PRIVATE KEY FILE! @
May 12 20:26:53 serveris sshd[753]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:26:53 serveris sshd[753]: error: Permissions 0755 for '/etc/ssh/ssh_host_dsa_key' are too open.
May 12 20:26:53 serveris sshd[753]: error: It is recommended that your private key files are NOTaccessible by others.
May 12 20:26:53 serveris sshd[753]: error: This private key will be ignored.
May 12 20:26:53 serveris sshd[753]: error: bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key
May 12 20:26:53 serveris sshd[753]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key
May 12 20:27:03 serveris sshd[755]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:27:03 serveris sshd[755]: error: @ WARNING: UNPROTECTED PRIVATE KEY FILE! @
May 12 20:27:03 serveris sshd[755]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
May 12 20:27:03 serveris sshd[755]: error: Permissions 0755 for '/etc/ssh/ssh_host_dsa_key' are too open.
May 12 20:27:03 serveris sshd[755]: error: It is recommended that your private key files are NOT accessible by others.
May 12 20:27:03 serveris sshd[755]: error: This private key will be ignored.
May 12 20:27:03 serveris sshd[755]: error: bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key
May 12 20:27:03 serveris sshd[755]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key
This is caused by the jail trying to use the hosts' SSH keys that were unionfs'd in.
jail> rm /etc/ssh/ssh_host*jail> rm /etc/ssh/moduli
jail> sh /etc/rc
This will create new ssh keys. We run /etc/rc explicitely because entropy will ask that you type for 30 seconds.
Jail ifconfig shows no IP address
One problem I encountered was that the jail didn't have any ip's listed under ifconfig. The solution for that was to make sure to do the ifconfig_nve0_alias0 in /etc/rc.conf instead of /jail/rc.conf or /jail/eoti.org/rc.conf. I believe it was caused by the ip's being assigned before the /jail partition was mounted.
unionfs-- panic: lockmgr: locking against myself
Error looks like this:
I found this problem related to kern/84107: unionfs related panic.
Solution:
panic: lockmgr: locking against myself
cpuid = 0
KBD: enter: panic
[thread pid 74167 tid 100077]
Stopped at kdb_enter+02f: nop
db>
I found this problem related to kern/84107: unionfs related panic.
Solution:
# patch < /anywhere/unionfs-p8.diff* Note: SERVERIS: /root/patches/unionfs/unionfs-p8.diff
# cp /usr/src/sys/fs/unionfs/union.h /usr/include/fs/unionfs
# cd /usr/src/sbin/mount_unionfs/
# make obj
# make depend
# make all
# make install
# make clean
# cd /usr/src
# make buildkernel KERNCONF=CUSTOM
# make installkernel KERNCONF=CUSTOM
# shutdown -r now
* Note: SERVERIS: /root/patches/unionfs/unionfs-p8.diff
Welcome
This blog is to archive notes, steps taken, kernel fixes, etc... Currently, it is focused on FreeBSD since that is what we are trying to use - but that may change based on functionality...
Subscribe to:
Posts (Atom)