Memory Upgrade

I upgraded the memory on the server from 4G to 8G today. Next step is to see if I can increase the swap space to compensate. Then start creating some domUs ;)

Installing xVM

I used this as a starting point to get Xen running.

1. Install the base xVM

pkg install SUNWxvmhvm
pkg install SUNWvirtinst
pkg install SUNWlibvirt
pkg install SUNWurlgrabber

2. Configure GRUB (I did this step completely different)

pfexec gedit /rpool/boot/grub/menu.lst

add:

title   ... xVM
 findroot (pool_rpool,0,a)
 kernel$ /boot/$ISADIR/xen.gz
 module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
 module$ /platform/i86pc/$ISADIR/boot_archive
 bootfs rpool/ROOT/opensolaris-1

3. Reboot into new configuration

4. 'uname -a' should contain 'i86xpv'

5. Startup the services

svcadm enable store
svcadm enable xend
svcadm enable console
svcadm enable domains
svcadm enable virtd 

6. Make sure they are running

malachi@serveris:~# svcs | grep xvm
online         19:34:15 svc:/system/xvm/store:default
online         19:34:22 svc:/system/xvm/xend:default
online         19:34:24 svc:/system/xvm/console:default
online         19:34:29 svc:/system/xvm/domains:default
online         19:34:33 svc:/system/xvm/virtd:default
 

Now to figure out how to do the domU :)

Default Router being dropped

When setting up the static routes, we did this:

svcadm disable svc:/network/physical:default
svcadm enable svc:/network/physical:nwam

At first that worked. Once I upgraded to the snv_94 it seemed to start disabling the default route (like here).  I fixed this by just re-enabling the physical:default service, which then reads the /etc/defaultrouter

svcadm enable svc:/network/physical:default

On boot, it started working again...

Useful Commands

* Note: It is intended that this list be regularly updated and just provide a common place to find these...

• pfexec reboot
• pfexec gedit FILENAME

Upgrading to 2008.11 snv_94 X86

I followed those instructions to do the update. The only exception to the instructions was that 2008.05 already had SUNWipkg@0.5.11,5.11-0.86 but I went ahead and updated it to -0.94 via the gnome System|Administration|Package Manager before doing the steps below that.

One weird thing on boot though is that I now have:
OpenSolaris 2008.05 snv_86_rc3 X86
opensolaris-1
Solaris 2008.11 snv_94 X86

I booted into the last one... even though the 2nd one that shows as active with beadm list. Seems to be working OK, but I should figure out what exactly is supposed to be there.

Roles can only be assumed by authorized users

The other day I went to reboot the system and found out I no longer had root access. I was very confused since I could not even su.  It kept telling me 'Roles can only be assumed by authorized users'.  I eventually figured out where I thought the problem was, but unfortunately didn't have permissions to fix it. In fact, no users on the system were authorized to fix it.  I ended up posting a question on the OpenSolaris Forums asking for help.  When I was at work today, I printed out a response so that I could try it when I got home. Unfortunately, the reply is no longer there for some reason, I can't give credit to whomever it was. Roumen provided this link to fix the problem. For those of you having this problem, here's what I did to cause it and to resolve it.

The Cause
When I was playing around with the Users and Groups, I went ahead and checked every single box. I mean, why not, I want full access, right? Nevermind that looking back I realize that some specifically say not to add them to users.

The Symptoms
I had basic user rights, but no root rights. I could not login as root, su to root, pfexec as root, save files owned by root, reboot the system, anything.

The Clue
Looking at /etc/user_attr, I noticed that while I did have the role=root attached to my name, there was a lot of garbage (read: all those checkmarks listed as profiles) before it.  Maybe 256 characters, maybe 512, I didn't honestly count.

I logged into the 2008.05 livecd and saw that jack only had Primary Administrator. Ok, I'll duplicate that.  Unfortunately, I couldn't find the real /etc directory in order to just rewrite it as jack.

Since I didn't have root access, I couldn't change the file.

The Solution

1. Reboot
2. At the GRUB prompt, hit e for edit
3. Arrow down to the kernel$ line
4. Hit e to edit
5. Append -s to the end of the line and hit enter
6. Hit b to start booting
7. When prompted, type in the root password
8. vi /etc/user_attr and remove all profiles except Primary Administrator from my account
9. save the file, exit to the prompt, hit ctrl-D to exit single-user mode

You now have your root access back.

Mirroring resolved the correct way!

Thanks to Mike Ellis for helping get this resolved correctly...

malachi@serveris:~# zpool detach rpool c5t1d0s2
malachi@serveris:~# zpool status -v
  pool: rpool
 state: ONLINE
 scrub: resilver completed after 0h1m with 0 errors on Tue Aug  5 21:16:53 2008
config:

    NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      c5t0d0s0  ONLINE       0     0     0

errors: No known data errors

malachi@serveris:~# prtvtoc /dev/rdsk/c5t0d0s2 | fmthard -s - /dev/rdsk/c5t1d0s2
fmthard:  New volume table of contents now in place.

malachi@serveris:~# prtvtoc /dev/rdsk/c5t0d0s2
* /dev/rdsk/c5t0d0s2 partition map
*
* Dimensions:
*     512 bytes/sector
*      63 sectors/track
*     255 tracks/cylinder
*   16065 sectors/cylinder
*   30400 cylinders
*   30398 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00    4209030 484134840 488343869
       1      3    01      16065   4192965   4209029
       2      5    01          0 488343870 488343869
       8      1    01          0     16065     16064
malachi@serveris:~# prtvtoc /dev/rdsk/c5t1d0s2
* /dev/rdsk/c5t1d0s2 partition map
*
* Dimensions:
*     512 bytes/sector
*      63 sectors/track
*     255 tracks/cylinder
*   16065 sectors/cylinder
*   30400 cylinders
*   30398 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00    4209030 484134840 488343869
       1      3    01      16065   4192965   4209029
       2      5    01          0 488343870 488343869
       8      1    01          0     16065     16064
malachi@serveris:~#

malachi@serveris:~# zpool status
  pool: rpool
 state: ONLINE
 scrub: resilver completed after 0h1m with 0 errors on Tue Aug  5 21:16:53 2008
config:

    NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      c5t0d0s0  ONLINE       0     0     0

errors: No known data errors
malachi@serveris:~# zpool attach rpool c5t0d0s0 c5t1d0s0
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c5t1d0s0 overlaps with /dev/dsk/c5t1d0s2
malachi@serveris:~# zpool attach -f rpool c5t0d0s0 c5t1d0s0

malachi@serveris:~# zpool status
  pool: rpool
 state: ONLINE
 scrub: resilver completed after 0h1m with 0 errors on Tue Aug  5 21:40:29 2008
config:

    NAME          STATE     READ WRITE CKSUM
    rpool         ONLINE       0     0     0
      mirror      ONLINE       0     0     0
        c5t0d0s0  ONLINE       0     0     0
        c5t1d0s0  ONLINE       0     0     0

errors: No known data errors

malachi@serveris:~# prtvtoc /dev/rdsk/c5t0d0s0
* /dev/rdsk/c5t0d0s0 partition map
*
* Dimensions:
*     512 bytes/sector
*      63 sectors/track
*     255 tracks/cylinder
*   16065 sectors/cylinder
*   30400 cylinders
*   30398 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00    4209030 484134840 488343869
       1      3    01      16065   4192965   4209029
       2      5    01          0 488343870 488343869
       8      1    01          0     16065     16064
malachi@serveris:~# prtvtoc /dev/rdsk/c5t1d0s0
* /dev/rdsk/c5t1d0s0 partition map
*
* Dimensions:
*     512 bytes/sector
*      63 sectors/track
*     255 tracks/cylinder
*   16065 sectors/cylinder
*   30400 cylinders
*   30398 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00    4209030 484134840 488343869
       1      3    01      16065   4192965   4209029
       2      5    01          0 488343870 488343869
       8      1    01          0     16065     16064
malachi@serveris:~#

malachi@serveris:~# installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5t1d0s0
stage1 written to partition 0 sector 0 (abs 16065)
stage2 written to partition 0, 260 sectors starting at 50 (abs 16115)

Mirroring resolved

This post is no longer valid. Please check out this one instead.

So it was a bit tricky, but I think I got it to work...

1. Save the source disk partition table:

• format c5t0d0
• save (to format.dat)
• quit

2. Repartition the destination disk:

• format c5t1d0
• fdisk (delete partition 1, exit)
• fdisk (use the default, exit)
• label ( 0 - SMI, use format.dat)
• quit

3. Mirror to the 2nd slice

• zpool attach -f rpool c5t0d0s0 c5t1d0s2

4. Install grub

• installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5t1d0s0

5. Verify partition table is still correct...

• format c5t0d0 and format c5t1d0
• partition
• print
• (they should match)

Importing the old data pool

malachi@serveris:~# zpool import
  pool: data
    id: 16202109509961920001
 state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric identifier, though
    some features will not be available without an explicit 'zpool upgrade'.
config:

    data        ONLINE
      raidz2    ONLINE
        c6t0d0  ONLINE
        c6t1d0  ONLINE
        c7t0d0  ONLINE
        c7t1d0  ONLINE
        c8t0d0  ONLINE

malachi@serveris:~# zpool import data
cannot import 'data': pool may be in use from other system
use '-f' to import anyway
malachi@serveris:~# zpool import -f data
malachi@serveris:~# zpool status
  pool: data
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
    still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
    pool will no longer be accessible on older software versions.
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    data        ONLINE       0     0     0
      raidz2    ONLINE       0     0     0
        c6t0d0  ONLINE       0     0     0
        c6t1d0  ONLINE       0     0     0
        c7t0d0  ONLINE       0     0     0
        c7t1d0  ONLINE       0     0     0
        c8t0d0  ONLINE       0     0     0

errors: No known data errors

malachi@serveris:~# zpool upgrade data
This system is currently running ZFS pool version 10.

Successfully upgraded 'data' from version 6 to version 10

malachi@serveris:~# zpool status
  pool: data
 state: ONLINE
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    data        ONLINE       0     0     0
      raidz2    ONLINE       0     0     0
        c6t0d0  ONLINE       0     0     0
        c6t1d0  ONLINE       0     0     0
        c7t0d0  ONLINE       0     0     0
        c7t1d0  ONLINE       0     0     0
        c8t0d0  ONLINE       0     0     0

errors: No known data errors
malachi@serveris:~# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
data                 713260393        55 713260339   1% /data

[trimmed...] 

malachi@serveris:~# cd /data

Mirrored ZFS bootable root

This post is no longer valid. Please check out this one instead.
 
So the instructions say that you are supposed to select two disks during installation to have it auto-mirror them. Unfortunately, it didn't allow me to select more than one disk (tried CTRL, Shift, mouse, keyboard, etc etc)... So, we are going to try to do it manually...

First, let's check the initial setup...
malachi@serveris:~$ zpool status
  pool: rpool
 state: ONLINE
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      c5t0d0s0  ONLINE       0     0     0

errors: No known data errors
malachi@serveris:~$ zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
rpool   230G  2.45G   228G     1%  ONLINE  -
malachi@serveris:~$ zfs list
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
rpool                               2.44G   224G  55.5K  /rpool
rpool@install                         16K      -  55.5K  -
rpool/ROOT                          2.29G   224G    18K  /rpool/ROOT
rpool/ROOT@install                      0      -    18K  -
rpool/ROOT/opensolaris              2.29G   224G  2.23G  legacy
rpool/ROOT/opensolaris@install      58.4M      -  2.22G  -
rpool/ROOT/opensolaris/opt          3.60M   224G  3.60M  /opt
rpool/ROOT/opensolaris/opt@install      0      -  3.60M  -
rpool/export                         157M   224G    19K  /export
rpool/export@install                  15K      -    19K  -
rpool/export/home                    157M   224G   157M  /export/home
rpool/export/home@install             18K      -    21K  -

So how do we know which drives are available for mirroring? We know that c5t0d0s0 is the one we want mirrored, but....

su then format
malachi@serveris:~# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c5t0d0
          /pci@0,0/pci1043,81fb@5/disk@0,0
       1. c5t1d0
          /pci@0,0/pci1043,81fb@5/disk@1,0
       2. c6t0d0
          /pci@0,0/pci1043,81fb@5,1/disk@0,0
       3. c6t1d0
          /pci@0,0/pci1043,81fb@5,1/disk@1,0
       4. c7t0d0
          /pci@0,0/pci1043,81fb@5,2/disk@0,0
       5. c7t1d0
          /pci@0,0/pci1043,81fb@5,2/disk@1,0
       6. c8t0d0
          /pci@7b,0/pci1043,81fb@5/disk@0,0
Specify disk (enter its number):
Specify disk (enter its number): ^C

It looks like c5t1d0 is the best choice...

malachi@serveris:~# zpool create rpool mirror c5t0d0s0 c5t1d0s0
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c5t0d0s0 is part of active ZFS pool rpool. Please see zpool(1M).
malachi@serveris:~# 

hmmm.. yes, we knew it was... we wanted to make a copy... hmmm
malachi@serveris:~# zpool add rpool c5t1d0s0
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c5t1d0s0 is part of exported or potentially active ZFS pool rootpool. Please see zpool(1M).


hmmm... is that from the current install or the previous install?
malachi@serveris:~# zpool attach rpool c5t0d0s0 c5t1d0s0
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c5t1d0s0 is part of exported or potentially active ZFS pool rootpool. Please see zpool(1M).


hmmm....
malachi@serveris:~# zpool attach -f rpool c5t0d0s0 c5t1d0s0
cannot attach c5t1d0s0 to c5t0d0s0: device is too small


hmmm, I tried formatting c5t1d0s0, but it still won't attach
zpool import shows that the other drives are still set as a raidz, so this is definitely the right drive... how to get it to import?

I think I may have to start over and use the manual installer rather than the GUI installer :(

Setting up a static network configuration with NWAM

Step 1: Figure out which network interface to use...

malachi@serveris:/etc/nwam# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
    inet 127.0.0.1 netmask ff000000
nge0: flags=201004843 mtu 1500 index 12
    inet 10.10.1.155 netmask ffffff00 broadcast 10.10.1.255
    ether 0:18:f3:d5:cd:9b
nge1: flags=201000802 mtu 1500 index 13
    inet 0.0.0.0 netmask 0
    ether 0:18:f3:d5:cd:9c
lo0: flags=2002000849 mtu 8252 index 1
    inet6 ::1/128

Step 2: Setup the nameservers....
gedit /etc/resolv.conf
add in the DNS servers...
EX:
nameserver 10.10.1.1
nameserver 66.93.87.2
nameserver 216.231.41.2

Step 3: Switch to using the DNS servers...

malachi@serveris:/etc# cp nsswitch.conf nsswitch.conf.original
malachi@serveris:/etc# cp nsswitch.dns nsswitch.conf
malachi@serveris:/etc# svcadm restart svc:/network/dns/client:default

Step 4: Switch to a static IP
gedit /etc/nwam/llp
change the 'nge0 dhcp' (or whatever device from Step 1) to 'nge0 static 10.10.1.10/24' (or whatever ip)

svcadm restart svc:/network/physical:nwam

Only video mode: 640x480

After a fresh install, the only video mode I had available was 640x480 @ 50 Hz. Well, that's definitely not right. It could be because of the KVM, but I noticed a lot of people using nVidia had the same problem. I used the notes I found here to fix it...

I changed quite a few things, saved, exited, came back in, ran the utility again and saved the config once I had it where I wanted it... That of course overwrote most of the changes (like the monitor name), so... here's the tidbit end result that matters...

Section "Monitor"
    # HorizSync source: xconfig, VertRefresh source: xconfig
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "CRT-0"
    HorizSync       30.0 - 81.0
    VertRefresh     56.0 - 75.0
    Option         "DPMS"
EndSection

Fresh Install!

So I decided to start over... I just installed OpenSolaris 2008.05

Will be posting configuration notes as I go so that it is easy to find them again later =)