09 February 2007
05 February 2007
New Hardware Purchased
Well, temporarily giving up on the iSCSI drives being an economical option, I decided to go ahead and build a standard run-of-the-mill server... Here's the specs of what we ordered:
Asus L1N64-SLI WS
2x AMD FX-70
Thermaltake VA8004BWS
Thermaltake HardCano 13
2x Kingston KVR667D2E5K2/2G
Thermaltake 850Watt W0131
going to use an existing space nVidia 6600GT
2x Hitachi Deskstar T7K500 250GB Serial ATA II (mirrored boot)
5x Hitachi Deskstar T7K500 250GB Serial ATA II (raidz2 data)
2x Addonics AE5RCS35NSA
NEC Black Floppy (memtest86, hitachi drive tool, etc etc)
going to use an existing dual layer burner
Belkin 1500VA/1000 joules UPS
We're also upgrading the KVM:
Belkin F1DE208C
3x Belkin F1D9400-06 (for machines on the rack)
1x Belkin F1D9400-25 (for docking station on the desk)
Still need to grab a Cat6 cable from PCHCables when we get a chance to run out to Hillsboro.
Plan is to run OpenSolaris as Xen dom0, then host all Xen domU via locally shared ZFS NFS directory...
Asus L1N64-SLI WS
2x AMD FX-70
Thermaltake VA8004BWS
Thermaltake HardCano 13
2x Kingston KVR667D2E5K2/2G
Thermaltake 850Watt W0131
going to use an existing space nVidia 6600GT
2x Hitachi Deskstar T7K500 250GB Serial ATA II (mirrored boot)
5x Hitachi Deskstar T7K500 250GB Serial ATA II (raidz2 data)
2x Addonics AE5RCS35NSA
NEC Black Floppy (memtest86, hitachi drive tool, etc etc)
going to use an existing dual layer burner
Belkin 1500VA/1000 joules UPS
We're also upgrading the KVM:
Belkin F1DE208C
3x Belkin F1D9400-06 (for machines on the rack)
1x Belkin F1D9400-25 (for docking station on the desk)
Still need to grab a Cat6 cable from PCHCables when we get a chance to run out to Hillsboro.
Plan is to run OpenSolaris as Xen dom0, then host all Xen domU via locally shared ZFS NFS directory...
01 February 2007
31 January 2007
Testing OpenSolaris
So I decided to try OpenSolaris on our old server (old being a year or so). It won't be the long term solution, because I want AMD-V for Xen... but here's my observations so far.
- Installation was easy, but first time, it did not make the system bootable. I had to format all the drives to get it to quit presenting FreeBSD as a boot option as well.
- The first install, it tried to use c2d0, but the second install it used c4d0. Shouldn't this be consistently the same boot device?
- Even though the router has the NIC set to a specific IP, it defaulted to a different MAC address and thus a different IP. I can only assume that it is a virtual NIC.
- When I switch back and forth on the Belkin KVM, the mouse freaks out in Solaris -- even though it works in Linux and FreeBSD.
- When activating the remote desktop, via GNOME, it sets it to localhost and doesn't allow me to change it to use the server name or IP -- thus, can't avoid #4 that way. Might have to go in and edit the files by hand, but unlike Linux/FreeBSD, I am not too familiar with Solaris yet.
- When I went to do the Update via the GUI, it asked for my Sun account info (which I provided) then just said there was an error. Guess that won't fix #4 or #5.
- ZPOOL seemed to work fine -- but creating iSCSI, SMB and NFS targets (via CLI or GUI) only seems to work at the server end. No one else on the network can access it.
- Contrary to what most people have said, the GUI doesn't look so bad.
- I enabled Telnet in services, and that didn't seem to work either.
30 January 2007
Plan shot down - looking at new plan
Well, it appears that there are not cheap iSCSI drives (or gigabit drive enclosures) out there. Sure there are NPWR and Adaptec, but both of those solutions are $700/drive.
Which is wierd, since cases are usually around $20 and the drives (250GB Samsung) were only $52 and the OS (NetBSD or Linux) is free...
I was looking at possibly making my own via mini-itx or something. Found iSCSI-SCSI bridges (only $1200)...
maybe it is time to make my own...? well, I would, but I really don't have the time to experiment right now... so I guess I once again have to get a larger server box up and running; then take my time designing my iSCSI solution (rotozip must like the enclosures)... And THEN design my nice small cool (as in temp) server....
In the meantime, got to figure out what to do... build new AMD-V machine (which isn't on Solaris HCL) or convert the last machine I bought (that has memory controller bugs and Asus lies all over it)
Which is wierd, since cases are usually around $20 and the drives (250GB Samsung) were only $52 and the OS (NetBSD or Linux) is free...
I was looking at possibly making my own via mini-itx or something. Found iSCSI-SCSI bridges (only $1200)...
maybe it is time to make my own...? well, I would, but I really don't have the time to experiment right now... so I guess I once again have to get a larger server box up and running; then take my time designing my iSCSI solution (rotozip must like the enclosures)... And THEN design my nice small cool (as in temp) server....
In the meantime, got to figure out what to do... build new AMD-V machine (which isn't on Solaris HCL) or convert the last machine I bought (that has memory controller bugs and Asus lies all over it)
29 January 2007
25 January 2007
Possible Plan
Ok, what if we start with plugging raw drives into the network, fronting as iSCSI Targets (possibly using embedded netbsd)...
Then the Solaris dom0 sets these up as Raid-Z.
Solaris then exports a NFS root for each domain. In fact, we could also export CDROM images and such over NFS so that the various OSes could have access to them.
At this point we would either have a small grub floppy image that would do network boot; or we have Xen boot the domU directly over the network. In either case, the plan is the same, have the OS run 'diskless' over NFS (with rw permissions)... If they have access to change which kernel grub points to, probably better.... so what if we have a small local grub image that points to a remote NFS grub image that is local to that filesystem... and thus they could change which kernel image, but not where it is...
Anyways, then, if we want to increase storage capacity, we add a drive to the network (iSCSI) and tell ZFS to add it to the pool... voila, all the domains have more space.
And if we want to back up a domain, we can take a snapshot and clone, etc.... I see us taking a snapshot at say 12:01am on monday, then incremental snapshot every day; thus we can rollback up to a week... although, probably need to maintain 2 weeks if we want to restore 1 -- otherwise every monday when we kill the primary full snapshot, all the incrementals become useless...
And if there is a hardware failure, Raid-Z should autorecover and provide means for me to fix it.
And if we need a test domain, we should be able to snapshot/clone (although booting with the same network parameters could be a problem).
And we should be able to write a tool (web based) that they can run in their own domain to request that we revert back to a previous snapshot...
And if we need to, we can specify quotas or reservations on a per domain or even on a per directory...
And although NFS should cause some overhead, since it is on the same physical server, I doubt it will even talk to the network...
And if we ever need to offload one of the domU to a different server, it should be extremely simple since we'd only need to move the Xen configuration and the tiny grub floppy image.
Overall I am liking this idea. Not sure how well it will work, but I guess I should start looking at hardware for it.
BTW: The Solaris dom0 needs redundancy too. I figure that what we would do is have a small mirrored root to boot Solaris -- but the dom0 needs to be the one in charge of the Raid-Z, so...
Then the Solaris dom0 sets these up as Raid-Z.
Solaris then exports a NFS root for each domain. In fact, we could also export CDROM images and such over NFS so that the various OSes could have access to them.
At this point we would either have a small grub floppy image that would do network boot; or we have Xen boot the domU directly over the network. In either case, the plan is the same, have the OS run 'diskless' over NFS (with rw permissions)... If they have access to change which kernel grub points to, probably better.... so what if we have a small local grub image that points to a remote NFS grub image that is local to that filesystem... and thus they could change which kernel image, but not where it is...
Anyways, then, if we want to increase storage capacity, we add a drive to the network (iSCSI) and tell ZFS to add it to the pool... voila, all the domains have more space.
And if we want to back up a domain, we can take a snapshot and clone, etc.... I see us taking a snapshot at say 12:01am on monday, then incremental snapshot every day; thus we can rollback up to a week... although, probably need to maintain 2 weeks if we want to restore 1 -- otherwise every monday when we kill the primary full snapshot, all the incrementals become useless...
And if there is a hardware failure, Raid-Z should autorecover and provide means for me to fix it.
And if we need a test domain, we should be able to snapshot/clone (although booting with the same network parameters could be a problem).
And we should be able to write a tool (web based) that they can run in their own domain to request that we revert back to a previous snapshot...
And if we need to, we can specify quotas or reservations on a per domain or even on a per directory...
And although NFS should cause some overhead, since it is on the same physical server, I doubt it will even talk to the network...
And if we ever need to offload one of the domU to a different server, it should be extremely simple since we'd only need to move the Xen configuration and the tiny grub floppy image.
Overall I am liking this idea. Not sure how well it will work, but I guess I should start looking at hardware for it.
BTW: The Solaris dom0 needs redundancy too. I figure that what we would do is have a small mirrored root to boot Solaris -- but the dom0 needs to be the one in charge of the Raid-Z, so...
Remote booting FreeBSD
So one thing I have been debating is how to take full advantage of the ZFS capabilities if some other OS is just going to make a huge file on it... While we still get hardware redundancy, it is no longer trivial to add more disk space....
This link was showing how to remote boot FreeBSD from PXE/NFS... I *think* but am not sure that if we boot/run FreeBSD from NFS, it won't care if that filesystem gets larger... right?
This link was showing how to remote boot FreeBSD from PXE/NFS... I *think* but am not sure that if we boot/run FreeBSD from NFS, it won't care if that filesystem gets larger... right?
Subscribe to:
Posts (Atom)