Hanging at Hostname: serveris

I had to reboot last night, so I made sure to keep track of the time.
10:58pm I had told Grub which instance to run and it was waiting at Hostname: serveris
1:30am I went to bed and it was still sitting there
7:30 this morning I check 'dmesg' and it shows nothing at all until 2:55am

So it took 4 hours to boot. That's pretty lame.

Project Darkstar

I recently had this conversation with Owen (rest of the conversation is here):

Owen F. Kellett to me

Wed, May 6, 2009 at 12:11 PM

Hi Malachi,

I haven't forgotten about you. Good news though. We have recently gotten approval to make the server api as well as the interfaces for writing custom services, authenticators, etc GPLv2 + classpath exception. We haven't updated the code yet, but this is coming very soon. I would anticipate the next official release should contain this new licensing scheme. In the meantime, keep an eye on our incremental releases: http://download.java.net/maven/2/com/projectdarkstar/server/sgs-server-dist/

Either the next one, or the one following will likely have this change.

thanks, Owen

Malachi de Ælfweald to Owen

Wed, May 6, 2009 at 3:39 PM

Hey Owen,

That is great news! So if I am understanding correctly, that will allow me to specify the sgs-server-api as a maven dependency without dropping the MIT licensing on my game?

Thanks, Malachi de Ælfweald http://www.google.com/profiles/malachid

Owen F. Kellett to me

Wed, May 6, 2009 at 5:09 PM

Yes that is correct. The classpath exception should free you from the requirement of using GPL for any linked code.

Based on that, I decided it was time to move forward with the darkstar-based projects...

Since I have a Hudson server, I decided to let it keep track of how to make it successfully build...

At first I was trying to build with JDK7, but it kept failing on the Javadoc module saying that the jars had invalid characters in it. That went away once I changed to JDK6. I also had to tell it to skip the tests, because one of them just hung forever...

The Hudson job for the server is here.
The Hudson job for the client is here.
The Hudson job for the maven plugin is here.

The distributables were delivered to Nexus.
sgs-server-dist-0.9.10-SNAPSHOT.zip
sgs-tutorial-server-dist-0.9.10-SNAPSHOT.zip
sgs-client-dist-0.9.10-SNAPSHOT.zip
sgs-tutorial-client-dist-0.9.10-SNAPSHOT.zip
sgs-maven-plugin-1.0-alpha-4-SNAPSHOT.jar

I created a new user on the server ('darkstar'/'darkstar') and unzipped the sgs-server-dist-0.9.10-SNAPSHOT.zip

I then created a symlink (/home/darkstar/current to /home/darkstar/sgs-server-dist-0.9.10-SNAPSHOT)

I don't know if it is correct, but I made this SMF manifest for it.

darkstar@kallisti [0] ~> svccfg import hudson.xml

darkstar@kallisti [0] ~> svcs -xv
svc:/application/darkstar:default (Project Darkstar Server)
State: maintenance since Thu May 14 23:37:36 2009
Reason: Restarting too quickly.
See: http://sun.com/msg/SMF-8000-L5
See: http://www.projectdarkstar.com/
See: /var/svc/log/application-darkstar:default.log
Impact: This service is not running.


darkstar@kallisti [0] ~> cat /var/svc/log/application-darkstar:default.log
WARNING: No application jar found with a META-INF/app.properties configuration file in the /home/darkstar/sgs-server-dist-0.9.10-SNAPSHOT/deploy directory
May 14, 2009 11:37:36 PM com.sun.sgs.impl.kernel.Kernel checkProperties
SEVERE: Missing required property com.sun.sgs.app.name
Exception in thread "main" java.lang.IllegalArgumentException: Missing required property com.sun.sgs.app.name
at com.sun.sgs.impl.kernel.Kernel.checkProperties(Kernel.java:855)
at com.sun.sgs.impl.kernel.Kernel.(Kernel.java:202)
at com.sun.sgs.impl.kernel.Kernel.main(Kernel.java:1123)
[ May 14 23:37:36 Stopping because all processes in service exited. ]
[ May 14 23:37:36 Restarting too quickly, changing state to maintenance. ]

Guess it's time to build an app...

Server Died

So I got home last night to find the server had power but was not responding (mouse, keyboard, ssh, anything). I tried rebooting, but it kept handing at "Hostname: serveris" and wouldn't go any further (even in single-user mode). I saw some chatlogs online that suggested adding '-k -a -d verbose' and using '/dev/null' to the answer of any questions (like /etc/system replacement)...

I tried that and got this far (see left)...

Looking around some more, I saw that if I changed the '-k' to '-kd' it would drop it into debug mode. At that point, I did the following:



[0] moddebug/W 80000000
[0] :c

This allowed me to see a few more details.... (sorry for the blurriness of the pic - it was about 3am)

After trying to find anything online that would help (and the IRC channel) I finally said screw it and decided I would reinstall opensolaris on the root mirror.

I downloaded the USB version of OpenSolaris 0906 111a, but evidentially my quad core machine does not have the option of booting from USB (WTF?). I reburned the CD version and installed it. One thing that confused me is that although my old system was 10/08 upgraded to 111a and the new version was supposed to be 0906 111a, it now says 101b.

Trying to boot the new one, it again hung. At a different position, but... I was starting to think it was a hardware problem. I let it try to boot overnight and the next morning it was finally at the login prompt... with the old install.

The logs showed that it had tried to load the Belkin UPS a few steps after where it locked up, so I unplugged the UPS. I went ahead and applied all updates and rebooted. It took about 5 hours for it to finally boot again (though it did). It still says I am using 101b and that there are no new updates.

The xVM instance is there and I was able to start it. The whole root zone however is gone. The ZFS partition is there, and empty. zoneadm doesn't show anything but global. So, I am going to try to recreate the global zone, but... I still don't know what happened. I am also concerned that it currently takes about 5 hours to boot.

Playing some music

So I wanted to listen to some music, but neither youtube or pandora were working (not sure why, they were yesterday)...

So I tried Songbird, which I use at work. It wouldn't work either:
Error: GStreamer error: Internal data flow error. Additional information: gstbasesrc.c(2234): gst_base_src_loop (): /GstPlayBin2:player/GstURIDecodeBin:uridecodebin1/GstFileSrc:source:
streaming task paused, reason not-linked (-1)

So I tried Rhythm, but it couldn't play anything either...

WTF? Right?
So I did this instead. Now I have music playing :)

Don't forget to escape the inner class

I think I have used Windows too long...

malachi@serveris[1]:~/work/ambrosia/syrinx % java -cp target/syrinx-2.3-SNAPSHOT.jar org.eoti.sound.BrainwaveOscillator$Test
Error: Main method not found in class org.eoti.sound.BrainwaveOscillator, please define the main method as:
public static void main(String[] args)
Exception in thread "main" java.lang.RuntimeException: Main method not found in org.eoti.sound.BrainwaveOscillator
at sun.launcher.LauncherHelper.signatureDiagnostic(LauncherHelper.java:209)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:197)

malachi@serveris[1]:~/work/ambrosia/syrinx % java -cp target/syrinx-2.3-SNAPSHOT.jar org.eoti.sound.BrainwaveOscillator\$Test

You have to escape the $ for the inner class :(

Hudson Maven Release Plug-in Plug-in + Hudson Google Code plugin

I was excited to be able to use the Hudson Google Code plugin for my project.



Unfortunately, when I tried to also use the Hudson Maven Release Plug-in Plug-in with it, it always failed to authenticate...


[INFO] Checking in modified POMs...
[INFO] Executing: /bin/sh -c cd /home/hudson/jobs/ambrosia-jdk6-standalone/workspace/. && svn --username hudson@eoti.org --password '*****' --non-interactive commit --file /var/tmp/maven-scm-2033681108.commit --targets /var/tmp/maven-scm-1162162468517704084-targets
[INFO] Working directory: /home/hudson/jobs/ambrosia-jdk6-standalone/workspace/.
[HUDSON] Archiving /home/hudson/jobs/ambrosia-jdk6-standalone/workspace/./pom.xml to /home/hudson/jobs/ambrosia-jdk6-standalone/modules/org.eoti$ambrosia/builds/2009-04-24_00-59-40/archive/org.eoti/ambrosia/2.2-SNAPSHOT/pom.xml
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Unable to commit files
Provider message:
The svn command failed.
Command output:
svn: Commit failed (details follow):
svn: MKACTIVITY of '/svn/!svn/act/0776c9b3-28c2-443e-a144-97770aaff2b2': authorization failed: Could not authenticate to server: rejected Basic challenge (http://ambrosia.googlecode.com)

[INFO] ------------------------------------------------------------------------
[INFO] Trace
org.apache.maven.BuildFailureException: Unable to commit files
Provider message:
The svn command failed.
Command output:
svn: Commit failed (details follow):
svn: MKACTIVITY of '/svn/!svn/act/0776c9b3-28c2-443e-a144-97770aaff2b2': authorization failed: Could not authenticate to server: rejected Basic challenge (http://ambrosia.googlecode.com)

at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:699)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:553)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:523)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:371)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:268)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:181)
at org.apache.maven.lifecycle.LifecycleExecutorInterceptor.execute(LifecycleExecutorInterceptor.java:65)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:356)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:137)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:356)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
at hudson.maven.agent.Main.launch(Main.java:158)
at hudson.maven.MavenBuilder.call(MavenBuilder.java:162)
at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:578)
at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:524)
at hudson.remoting.UserRequest.perform(UserRequest.java:97)
at hudson.remoting.UserRequest.perform(UserRequest.java:46)
at hudson.remoting.Request$2.run(Request.java:236)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.maven.plugin.MojoFailureException: Unable to commit files
Provider message:
The svn command failed.
Command output:
svn: Commit failed (details follow):
svn: MKACTIVITY of '/svn/!svn/act/0776c9b3-28c2-443e-a144-97770aaff2b2': authorization failed: Could not authenticate to server: rejected Basic challenge (http://ambrosia.googlecode.com)

at org.apache.maven.plugins.release.PrepareReleaseMojo.execute(PrepareReleaseMojo.java:190)
at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:483)
at hudson.maven.agent.PluginManagerInterceptor.executeMojo(PluginManagerInterceptor.java:182)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:678)
... 28 more
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 40 seconds
[INFO] Finished at: Fri Apr 24 01:00:22 PDT 2009
[INFO] Final Memory: 17M/218M
[INFO] ------------------------------------------------------------------------
channel stopped
Finished: FAILURE


This was because it was using http instead of https to do the checkout...
The solution was to only partially use the Google Code plugin...

Compiling in the zone; eg. where's the damned math.h?

So it appears that some of the basic include files were not included by default in the zone either. To get that working (and thus get random stuff compiling correctly)...

pfexec pkg install SUNWlibm

pkgadd, pkginfo, etc missing

Upon installation of the new zone, I was surprised that some of the basic pieces like pkgadd was missing. It appears (at least for zones) that IPS is definitely the default now.

Unfortunately, there are times when you still want those commands...  like when testing build processes written for Solaris 10.

Following the instructions in the previous blog, here's what you need to do to get it working:

If this is the first time...
[kallisti:malachi(127)] ~/work% pfexec pkg set-authority -O http://pkg.opensolaris.org/dev/ os-dev


Then do
[kallisti:malachi(127)] ~/work% pfexec pkg install SUNWpkgcmds

Installing a named IPS package in a zone

Let's say you have a package you'd like to install, but don't have a .pkg for it.... For example, gtar.

Go to http://pkg.opensolaris.org/dev/ and locate the package (search box)
For example, SUNWgtar@1.17,5.11-0.111:20090331T082820Z
Note the name before the @
IE, SUNWgtar

If this is the first time...
root@kallisti:/opt# pkg set-authority -O http://pkg.opensolaris.org/dev/ os-dev

Then to do the install...
root@kallisti:/opt# pkg refresh
root@kallisti:/opt# pkg install SUNWgtar

Hudson

I've been thinking about setting up a continuous integration server for awhile. After doing some configuration of Cruise Control at work, I decided to try Hudson at home.

First things first... Virtualization? I decided I would rather have it virtualized if for no other reason than that dotster already had dns setup for one of my currently unused ips and I didn't want to wait for the ip change to propagate.

I can't run VirtualBox because I am running xVM and VirtualBox refuses to start if you are in xVM mode. I could run it in xVM... but I really don't like dedicating a large chunk of my memory to it... so zone it is.

malachi@serveris[0]:/data % pfexec zonecfg -z eoti.org
eoti.org: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:eoti.org> create
zonecfg:eoti.org> set zonepath=/data/eoti.org
zonecfg:eoti.org> set autoboot=true
zonecfg:eoti.org> add net
zonecfg:eoti.org:net> set address=10.10.1.13/16
zonecfg:eoti.org:net> set physical=nge0
zonecfg:eoti.org:net> set defrouter=10.10.1.1
zonecfg:eoti.org:net> end
zonecfg:eoti.org> verify
zonecfg:eoti.org> commit
zonecfg:eoti.org> exit

malachi@serveris[1]:/data % pfexec zoneadm -z eoti.org install
/data/eoti.org must not be group readable.
/data/eoti.org must not be group writable.
/data/eoti.org must not be group executable.
/data/eoti.org must not be world readable.
/data/eoti.org must not be world executable.
could not verify zonepath /data/eoti.org because of the above errors.
zoneadm: zone eoti.org failed to verify

malachi@serveris[1]:/data % pfexec chmod 700 /data/eoti.org

malachi@serveris[1]:/data % pfexec zoneadm -z eoti.org install

malachi@serveris[1]:/data % pfexec zoneadm -z eoti.org boot

malachi@serveris[1]:/data % pfexec zlogin -C eoti.org
(chose option 6 - xterms)
Host name for nge0:1 kallisti
domain: eoti.org
Server's IP address: 10.10.1.1
Server's IP address: 66.93.87.2
Server's IP address: 216.231.41.2
pwd: ***********

malachi@serveris[1]:/data % pfexec zoneadm -z eoti.org reboot
malachi@serveris[1]:/data % zoneadm list -v
malachi@serveris[1]:/data % pfexec zlogin eoti.org

root@kallisti:~# svcadm disable autofs
root@kallisti:~# useradd -g staff -s /usr/bin/bash -c "Malachi de AElfweald" -m -P "Primary Administrator" malachi
root@kallisti:~# passwd malachi

malachi@serveris[1]:/data % pfexec vi /etc/hosts
10.10.1.13 kallisti.eoti.org kallisti

http://wiki.hudson-ci.org/display/HUDSON/OpenSolaris+IPS+package
but had to change this line to include java
<envvar name="'PATH'" value="'/opt/jdk1.7.0/bin:/usr/bin:/usr/sbin:/usr/ccs/bin:/usr/local/bin:/usr/local/sbin:/usr/sfw/bin' />

couldn't have it create the zfs filesystem within the zone... but it was already on a zfs filesystem anyway, so no worries

Every once in awhile, Firefox starts giving me this error when I choose to "Open" instead of "Save As...":

BLAH could not be saved, because you cannot change the contents of that folder. Change the folder properties and try again, or try saving in a different location.

I found this page that gives hints on how to fix it.

It seems just clearing my download history (Clear List) was enough to get it working again...

Failure to boot into graphical mode after upgrade

A coworker upgraded his OpenSolaris from snv_106 to snv_109. Upon rebooting, it failed to launch into X. For awhile, we assumed there was a problem in the grub menu.lst, but it seems the answer was a bit different.

When running OpenSolaris inside VirtualBox, you usually 'Install Guest Additions' so that you can run full-screen. Apparently, the upgrade from snv_106 to snv_109 broke the compatibility with installed guest additions.

The final solution was to uninstall and reinstall the guest additions.

From the menu (or CTRL-Home if you are already full screen) go to Device | Install Guest Additions. That will mount the virtual cdrom.
# cd /cdrom
# pfexec pkgrm SUNWvboxguest
# pfexec pkgadd -d SUNWvboxguest
# pfexec reboot

It should all work now :)

ZSH Tab Completion Fix for Subversion 1.5

So one of the annoying things I noticed was that tab autocompletion for zsh was broken if I was using it with svn; showing:
_arguments:comparguments:303: invalid argument: ARG

Based on notes here I fixed it by doing the following:

$ cd /usr/share/zsh/4.3.4/functions/Completion/Unix/_subversion
$ pfexec mv _subversion ~/original._subversion
$ pfexec wget http://gvn.googlecode.com/svn/trunk/contrib/zsh/_subversion
$ pfexec chmod 444 _subversion

It all works now :)