Zones getting out of sync

The other day, I started an update and walked away. I noticed today it was done, so I zfs snapshotted the zones and xvm sessions then rebooted.

Upon bootup, everything seemed OK until I tried to access the web pages on one of the zones. Checking svcs -xv, I saw that the local filesystem was down! That's not good!

While troubleshooting, it occurred to me that the global is reporting snv_130 and zlogin to a zone is reporting snv_101a. Oops. Just a little outdated.

I looked around for how to get them into sync, and ran across this article.

The solution is actually quite simple.  It takes *forever* so I really wish I would have checked the time for you before I started.  The basic jist of it is:

# pfexec zoneadm -z MYZONE halt

# pfexec zoneadm -z MYZONE detach

# pfexec zoneadm -z MYZONE attach -u

At first, I forgot the '-u' and it came back and told me it was out of sync and reminded me to try again with -u.

root@serveris:~# zoneadm -z eoti.org attach -u

Log File: /var/tmp/eoti.org.attach_log.voaWef
Attaching...

       Global zone version: entire@0.5.11,5.11-0.130:20091219T044839Z
   Non-Global zone version: entire@0.5.11,5.11-0.101:20081204T010954Z
           Publisher Check: Zone preferred publisher does not contain
                            entire@0.5.11,5.11-0.130:20091219T044839Z.
           Publisher Reset: Copying preferred publisher from global zone.
  Updating non-global zone: (Stage 1).  Output follows
Creating Plan                
ERROR: Could not update attaching zone

Oops, now what?  At first I started looking online at things like Bug 13190... Then I realized I was missing the obvious...
Log File: /var/tmp/eoti.org.attach_log.voaWef
hmm, ok, so what does it say?

root@serveris:~# cat /var/tmp/eoti.org.attach_log.voaWef

[Monday, January  4, 2010  8:08:03 PM PST] Log File: /var/tmp/eoti.org.attach_log.voaWef
[Monday, January  4, 2010  8:08:07 PM PST] Attaching...
[Monday, January  4, 2010  8:08:07 PM PST] existing
[Monday, January  4, 2010  8:08:07 PM PST]
[Monday, January  4, 2010  8:08:07 PM PST]   Sanity Check: Passed.  Looks like an OpenSolaris system.

Unable to retrieve package data for publisher 'os-dev' from one
of the following origin(s):

http://pkg.opensolaris.org/dev/

The catalog retrieved from one of the origin(s) listed above only
contains package data for: opensolaris.org.

To resolve this issue, correct the origin information provided for
publisher 'os-dev' using the pkg set-publisher subcommand, or re-add
the publisher using the correct name and remove the 'os-dev'
publisher.

To re-add this publisher with the correct name, execute the following
commands as a privileged user:

pkg set-publisher -P -g http://pkg.opensolaris.org/dev/ opensolaris.org
pkg unset-publisher os-dev


Unable to retrieve package data for publisher 'os-dev' from one
of the following origin(s):

http://pkg.opensolaris.org/dev/

The catalog retrieved from one of the origin(s) listed above only
contains package data for: opensolaris.org.

To resolve this issue, correct the origin information provided for
publisher 'os-dev' using the pkg set-publisher subcommand, or re-add
the publisher using the correct name and remove the 'os-dev'
publisher.

To re-add this publisher with the correct name, execute the following
commands as a privileged user:

pkg set-publisher -P -g http://pkg.opensolaris.org/dev/ opensolaris.org
pkg unset-publisher os-dev


pkg: The following pattern(s) did not match any packages in the current catalog.
Try relaxing the pattern, refreshing and/or examining the catalogs:
    entire@0.5.11,5.11-0.130:20091219T044839Z

This may seem odd to most of you... but I had the default opensolaris package repository AND the dev repository setup at one point; so it was getting confused.  I followed the instructions (the pkg set-publisher and pkg unset-publisher) then retried the attach...

root@serveris:~# zoneadm -z eoti.org attach -u

Log File: /var/tmp/eoti.org.attach_log.4kaGTg
Attaching...

       Global zone version: entire@0.5.11,5.11-0.130:20091219T044839Z
   Non-Global zone version: entire@0.5.11,5.11-0.101:20081204T010954Z
           Publisher Check: Zone preferred publisher does not contain
                            entire@0.5.11,5.11-0.130:20091219T044839Z.
           Publisher Reset: Copying preferred publisher from global zone.
  Updating non-global zone: (Stage 1).  Output follows
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                              121/121 19959/19959  125.0/125.0

PHASE                                        ACTIONS
Removal Phase                              5811/5811
Install Phase                            15876/15876
Update Phase                             12076/12076
PHASE                                          ITEMS
Indexing Packages                            122/122
  Updating non-global zone: (Stage 2).  Output follows
No updates necessary for this image.  
  Updating non-global zone: Zone updated to entire@0.5.11,5.11-0.130:20091219T044839Z
Attach complete.

Yay!

JForum on OpenSolaris

I decided to go ahead and setup a forum for one of the projects I am working on.  I already had a zone setup that was hosting a "coming soon" page.

Looking around, I spent quite awhile trying to decide which forum software I was going to use.  As a Java developer, I decided to go with a Java-based one so that it is easier for me to maintain.  I ended up choosing JForum.

Installation was actually pretty easy.  Getting the forums configured not so much (by default new forums were not visible until I went and reconfigured every user group -- each time I added a forum -- very annoying).

I had zlogin'd to the domain and:

# pkg install SUNWtcat
# pkg install SUNWj6dvx
# svcadm enable tomcat6

Checking port 8080 on the domain I saw that tomcat was indeed running.

I downloaded jforum-2.1.8.war and placed it as: /var/tomcat6/webapps/jforum.war

Going to http://THEMACHINE:8080/jforum/install.jsp I specified to use HSQLDB, UTF-8, my URLs and a password.

Still a few things I'd like to fix, but as of now the forums are working.

FTPConnectionClosedException: Connection closed without indication

After installing the ftp server yesterday, I was trying to connect to it with the commons-net library.  Unfortunately it kept failing:

org.apache.commons.net.ftp.FTPConnectionClosedException: Connection closed without indication.
    at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:297)
    at org.apache.commons.net.ftp.FTP.getReply(FTP.java:619)
    at org.apache.commons.net.ftp.FTPClient.completePendingCommand(FTPClient.java:1244)
    at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1301)

As download worked from the browser, I thought this kinda odd.  Looking a little closer, I decided to manually try the steps from the command line and see if I could figure out what was going on.

I ftp'd in and did the same commands I was doing from Java (passive, binary, etc).

When I tried to 'get ' I saw:

150 Opening BINARY mode data connection for aopalliance-1.0.jar (6371 bytes).
421 Service not available, remote server has closed connection
ftp> 

Since I had found through my debugging that after the exception, the 150 FILE_STATUS_OK was the last reply I got; I looked up 421.  It appears that there is a current bug on OpenSolaris that causes this behavior.

So here's the workaround (for the setup I did yesterday):

cp /lib/libsendfile.so.1 /home/ftp/lib
chown root:bin /home/ftp/lib/libsendfile.so.1

svcadm restart ftp

Everything works beautifully now :)

Installing ftpd into a zone

from zone:

• pkg install SUNWftp
  
• svcadm enable ftp
• ftpconfig /home/ftp ((it will fail at the end))

from global:

• ftpconfig -d ((zonepath))/root/home/ftp

should work now

Server Died

So I got home last night to find the server had power but was not responding (mouse, keyboard, ssh, anything). I tried rebooting, but it kept handing at "Hostname: serveris" and wouldn't go any further (even in single-user mode). I saw some chatlogs online that suggested adding '-k -a -d verbose' and using '/dev/null' to the answer of any questions (like /etc/system replacement)...

I tried that and got this far (see left)...

Looking around some more, I saw that if I changed the '-k' to '-kd' it would drop it into debug mode. At that point, I did the following:



[0] moddebug/W 80000000
[0] :c

This allowed me to see a few more details.... (sorry for the blurriness of the pic - it was about 3am)

After trying to find anything online that would help (and the IRC channel) I finally said screw it and decided I would reinstall opensolaris on the root mirror.

I downloaded the USB version of OpenSolaris 0906 111a, but evidentially my quad core machine does not have the option of booting from USB (WTF?). I reburned the CD version and installed it. One thing that confused me is that although my old system was 10/08 upgraded to 111a and the new version was supposed to be 0906 111a, it now says 101b.

Trying to boot the new one, it again hung. At a different position, but... I was starting to think it was a hardware problem. I let it try to boot overnight and the next morning it was finally at the login prompt... with the old install.

The logs showed that it had tried to load the Belkin UPS a few steps after where it locked up, so I unplugged the UPS. I went ahead and applied all updates and rebooted. It took about 5 hours for it to finally boot again (though it did). It still says I am using 101b and that there are no new updates.

The xVM instance is there and I was able to start it. The whole root zone however is gone. The ZFS partition is there, and empty. zoneadm doesn't show anything but global. So, I am going to try to recreate the global zone, but... I still don't know what happened. I am also concerned that it currently takes about 5 hours to boot.

Compiling in the zone; eg. where's the damned math.h?

So it appears that some of the basic include files were not included by default in the zone either. To get that working (and thus get random stuff compiling correctly)...

pfexec pkg install SUNWlibm

pkgadd, pkginfo, etc missing

Upon installation of the new zone, I was surprised that some of the basic pieces like pkgadd was missing. It appears (at least for zones) that IPS is definitely the default now.

Unfortunately, there are times when you still want those commands...  like when testing build processes written for Solaris 10.

Following the instructions in the previous blog, here's what you need to do to get it working:

If this is the first time...
[kallisti:malachi(127)] ~/work% pfexec pkg set-authority -O http://pkg.opensolaris.org/dev/ os-dev


Then do
[kallisti:malachi(127)] ~/work% pfexec pkg install SUNWpkgcmds

Installing a named IPS package in a zone

Let's say you have a package you'd like to install, but don't have a .pkg for it.... For example, gtar.

Go to http://pkg.opensolaris.org/dev/ and locate the package (search box)
For example, SUNWgtar@1.17,5.11-0.111:20090331T082820Z
Note the name before the @
IE, SUNWgtar

If this is the first time...
root@kallisti:/opt# pkg set-authority -O http://pkg.opensolaris.org/dev/ os-dev

Then to do the install...
root@kallisti:/opt# pkg refresh
root@kallisti:/opt# pkg install SUNWgtar

Hudson

I've been thinking about setting up a continuous integration server for awhile. After doing some configuration of Cruise Control at work, I decided to try Hudson at home.

First things first... Virtualization? I decided I would rather have it virtualized if for no other reason than that dotster already had dns setup for one of my currently unused ips and I didn't want to wait for the ip change to propagate.

I can't run VirtualBox because I am running xVM and VirtualBox refuses to start if you are in xVM mode. I could run it in xVM... but I really don't like dedicating a large chunk of my memory to it... so zone it is.

malachi@serveris[0]:/data % pfexec zonecfg -z eoti.org
eoti.org: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:eoti.org> create
zonecfg:eoti.org> set zonepath=/data/eoti.org
zonecfg:eoti.org> set autoboot=true
zonecfg:eoti.org> add net
zonecfg:eoti.org:net> set address=10.10.1.13/16
zonecfg:eoti.org:net> set physical=nge0
zonecfg:eoti.org:net> set defrouter=10.10.1.1
zonecfg:eoti.org:net> end
zonecfg:eoti.org> verify
zonecfg:eoti.org> commit
zonecfg:eoti.org> exit

malachi@serveris[1]:/data % pfexec zoneadm -z eoti.org install
/data/eoti.org must not be group readable.
/data/eoti.org must not be group writable.
/data/eoti.org must not be group executable.
/data/eoti.org must not be world readable.
/data/eoti.org must not be world executable.
could not verify zonepath /data/eoti.org because of the above errors.
zoneadm: zone eoti.org failed to verify

malachi@serveris[1]:/data % pfexec chmod 700 /data/eoti.org

malachi@serveris[1]:/data % pfexec zoneadm -z eoti.org install

malachi@serveris[1]:/data % pfexec zoneadm -z eoti.org boot

malachi@serveris[1]:/data % pfexec zlogin -C eoti.org
(chose option 6 - xterms)
Host name for nge0:1 kallisti
domain: eoti.org
Server's IP address: 10.10.1.1
Server's IP address: 66.93.87.2
Server's IP address: 216.231.41.2
pwd: ***********

malachi@serveris[1]:/data % pfexec zoneadm -z eoti.org reboot
malachi@serveris[1]:/data % zoneadm list -v
malachi@serveris[1]:/data % pfexec zlogin eoti.org

root@kallisti:~# svcadm disable autofs
root@kallisti:~# useradd -g staff -s /usr/bin/bash -c "Malachi de AElfweald" -m -P "Primary Administrator" malachi
root@kallisti:~# passwd malachi

malachi@serveris[1]:/data % pfexec vi /etc/hosts
10.10.1.13 kallisti.eoti.org kallisti

http://wiki.hudson-ci.org/display/HUDSON/OpenSolaris+IPS+package
but had to change this line to include java
<envvar name="'PATH'" value="'/opt/jdk1.7.0/bin:/usr/bin:/usr/sbin:/usr/ccs/bin:/usr/local/bin:/usr/local/sbin:/usr/sfw/bin' />

couldn't have it create the zfs filesystem within the zone... but it was already on a zfs filesystem anyway, so no worries