24 January 2007
23 January 2007
Thoughts... ie: What should I do?
So as I investigate things like Sun servers, OpenSolaris, ZFS/Raid-Z, iSCSI, Embedded NetBSD, etc etc... I start thinking that perhas a better approach would be to start off with a list of requirements... It is a little late to "start" there, but, what the hell...
* We do not want a repeat of the data wipeout we got a year ago... As such, if one domain gets hacked, it can not take down the rest of the domains.
- I had been planning on using Jail to alleviate this problem, but at this point I think our best bet is Xen
- While many OSs support Xen domU, many fewer support Xen dom0.
* Also, if a domain gets hacked and wiped out, we need a backup to restore from.
- ZFS Snapshots (maybe full weekly and incremental daily) seems like a good solution for this
- SVN might be a good solution to this
- It might be good if there was a way to automate this (ie: allow the root user to do it instead of emailing me)
* We want to get disk redundancy, while keeping cost and maintenance low.
- RAID-0 gives us the redundancy, but at a fairly high cost. The current 1TB gives us less than 500GB this way.
- RAID-5 could be acceptable, as long as we could fix the system if the root partition were the one to screw up
- ZFS and Raid-Z seem like excellent options to provide the redundancy and automatic corrections
* Each domain should be able to have it's own root user and OS.
- Jail could have provided for this, but seems very buggy ('man man' in a jail crashed the entire OS)
- Xen should allow for this fairly easy. As such, the base OS really doesn't do anything other than launch the domUs.
* If multiple OSs share a base OS, it would be nice (though not required) if we were able to not waste a lot of duplicated space
- Unionfs is ideal for this. Not sure how easy it is to use with Xen
- At some point, it has to be better to have a full copy (like if they upgraded every single app)
* It would be preferable if all OS's could run unmodified, thus allowing us greater flexibility and choices
- Intel VT or AMD-V technology. If we are planning on doing any RSA work on any domain, it can't be the Intel option. So probably an Opteron 2xxx (or 2).
* If we run low on drive space, it should be fairly simple to add more
- ZFS is supposed to allow for this with 'zfs add'
- We *could* do separate drives for each domain, but that would waste a lot of drive space
- iSCSI might be a better option than internal drives for this. Instead of running out of hard drive bays, we'd just have to worry about running out of ethernet ports -- which realistically, is much easier to come up with more of. ** SEE BELOW
** iSCSI Thoughts:
- IDEALLY, I would have a set of cheap 250GB or 500GB drives that host themselves as iSCSI Targets. Then, adding new drives is literally adding another drive to the network.
- We could, in theory, run Solaris or NetBSD to easily provide iSCSI Targets to the network - but then we have to wonder why we are creating a separate box to host Xen
- If we have multiple iSCSI Targets, which our Solaris(?) box were to RAID-Z together, we could also have extra iSCSI Targets that VMWare and/or other hardware could use directly (even Windows)
- If we have multiple iSCSI Targets, which our Solaris(?) box were to RAID-Z together, the Solaris box could provide the pool as a iSCSCI Target to other devices/OSes
- If we were to have Solaris running RAID-Z directly on the iSCSI Target (ie: providing ZFS pools as iSCSI Targets), then separate Xen domUs could point to these iSCSI Targets for their primary data storage and POTENTIALLY get the ZFS benefits regardless of their OS (snapshots, error-correction [at least from hardware errors], etc)
- In theory, if we could come up with mini embeddable systems that could boot from iSCSI Targets, then we could do away with Xen entirely and just have a little device on the network for each domain... this might be overkill
- Note: Can Solaris boot off iSCSI?
* We do not want a repeat of the data wipeout we got a year ago... As such, if one domain gets hacked, it can not take down the rest of the domains.
- I had been planning on using Jail to alleviate this problem, but at this point I think our best bet is Xen
- While many OSs support Xen domU, many fewer support Xen dom0.
* Also, if a domain gets hacked and wiped out, we need a backup to restore from.
- ZFS Snapshots (maybe full weekly and incremental daily) seems like a good solution for this
- SVN might be a good solution to this
- It might be good if there was a way to automate this (ie: allow the root user to do it instead of emailing me)
* We want to get disk redundancy, while keeping cost and maintenance low.
- RAID-0 gives us the redundancy, but at a fairly high cost. The current 1TB gives us less than 500GB this way.
- RAID-5 could be acceptable, as long as we could fix the system if the root partition were the one to screw up
- ZFS and Raid-Z seem like excellent options to provide the redundancy and automatic corrections
* Each domain should be able to have it's own root user and OS.
- Jail could have provided for this, but seems very buggy ('man man' in a jail crashed the entire OS)
- Xen should allow for this fairly easy. As such, the base OS really doesn't do anything other than launch the domUs.
* If multiple OSs share a base OS, it would be nice (though not required) if we were able to not waste a lot of duplicated space
- Unionfs is ideal for this. Not sure how easy it is to use with Xen
- At some point, it has to be better to have a full copy (like if they upgraded every single app)
* It would be preferable if all OS's could run unmodified, thus allowing us greater flexibility and choices
- Intel VT or AMD-V technology. If we are planning on doing any RSA work on any domain, it can't be the Intel option. So probably an Opteron 2xxx (or 2).
* If we run low on drive space, it should be fairly simple to add more
- ZFS is supposed to allow for this with 'zfs add'
- We *could* do separate drives for each domain, but that would waste a lot of drive space
- iSCSI might be a better option than internal drives for this. Instead of running out of hard drive bays, we'd just have to worry about running out of ethernet ports -- which realistically, is much easier to come up with more of. ** SEE BELOW
** iSCSI Thoughts:
- IDEALLY, I would have a set of cheap 250GB or 500GB drives that host themselves as iSCSI Targets. Then, adding new drives is literally adding another drive to the network.
- We could, in theory, run Solaris or NetBSD to easily provide iSCSI Targets to the network - but then we have to wonder why we are creating a separate box to host Xen
- If we have multiple iSCSI Targets, which our Solaris(?) box were to RAID-Z together, we could also have extra iSCSI Targets that VMWare and/or other hardware could use directly (even Windows)
- If we have multiple iSCSI Targets, which our Solaris(?) box were to RAID-Z together, the Solaris box could provide the pool as a iSCSCI Target to other devices/OSes
- If we were to have Solaris running RAID-Z directly on the iSCSI Target (ie: providing ZFS pools as iSCSI Targets), then separate Xen domUs could point to these iSCSI Targets for their primary data storage and POTENTIALLY get the ZFS benefits regardless of their OS (snapshots, error-correction [at least from hardware errors], etc)
- In theory, if we could come up with mini embeddable systems that could boot from iSCSI Targets, then we could do away with Xen entirely and just have a little device on the network for each domain... this might be overkill
- Note: Can Solaris boot off iSCSI?
22 January 2007
Another thing to consider
Maybe get a 2 NIC server, and use one for a data-only network; with RAID-Z over the NAS
21 January 2007
20 January 2007
RaidZ Test
So I downloaded BeleniX and ran it under VMWare.
At first, I used 3 SCSI hard drives, but could not figure out for the life of me what the names were... Kept getting I/O errors when I ran tools to find out.
Redid it as 3 (cuz the CDRom took one of the slots) IDE drives (1GB each).
So far so good. It doesn't have Xen support yet, and I don't really feel like trying it right now since it is a LiveCD and all changes are lost when I reboot the vm -- maybe later.
At first, I used 3 SCSI hard drives, but could not figure out for the life of me what the names were... Kept getting I/O errors when I ran tools to find out.
Redid it as 3 (cuz the CDRom took one of the slots) IDE drives (1GB each).
# format
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@7,1/ide@0/cmdk@0,0
1. c0d1
/pci@0,0/pci-ide@7,1/ide@0/cmdk@1,0
0. c1d1
/pci@0,0/pci-ide@7,1/ide@1/cmdk@1,0
# zpool create mypool raidz c0d0 c0d1 c1d1
# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
mypool 2.95G 157K 2.95G 0% ONLINE -
So far so good. It doesn't have Xen support yet, and I don't really feel like trying it right now since it is a LiveCD and all changes are lost when I reboot the vm -- maybe later.
18 January 2007
More ZFS Links
Blog O' Matty >> Solaris ZFS
This blog has some various experiences and links of it's own. Actually the next two links here were obtained there.
ZFS Best Practices Guide
This page suggests that 3 disks use raidz and 5 disks use raidz2.
ZFS Management and Troubleshooting
Of importance at the top of this page, only Solaris 10 from 6/2006+ have ZFS. Also, they recommend 3-9 disks for Raid-Z pools. There is also info there on how to replace a drive. It says that Root pools (bootable) are not yet available as of 6/2006 -- but that once they are, they should be kept on a separate pool from the other data.
That last link also provided a link to:
Solaris ZFS Administration Guide
This blog has some various experiences and links of it's own. Actually the next two links here were obtained there.
ZFS Best Practices Guide
This page suggests that 3 disks use raidz and 5 disks use raidz2.
ZFS Management and Troubleshooting
Of importance at the top of this page, only Solaris 10 from 6/2006+ have ZFS. Also, they recommend 3-9 disks for Raid-Z pools. There is also info there on how to replace a drive. It says that Root pools (bootable) are not yet available as of 6/2006 -- but that once they are, they should be kept on a separate pool from the other data.
That last link also provided a link to:
Solaris ZFS Administration Guide
16 January 2007
EMail conversation with Sun
I received this email....
So the rest will most likely be via phone.
Hi Malachi,My response:
Thank you for visiting the web chat forum on the Sun website yesterday, when you expressed interest in the Try and Buy promotion being currently run by Sun.
You mentioned that you are deciding which machine you will trial on the promotion - excellent! You will see the full list of products on the promotion at www.sun.com/tryandbuy
You can email me with any questions that you may have, or you can visit our team at the web chat forum. Here's some additional information on the Try and Buy promotion, which you may find useful:
When you receive the Server on the Try and Buy promotion you will also receive a Welcome Pack.
Inside the Welcome Pack you will find-
1. A Quick Guide to Installation.
2. Access to Tuning and Optimization documentation.
3. Access to an online portal and forum for system and application tuning hints and tips.
4. Access to Sun's performance engineers.
5. All of the latest patches and configuration files for the system on the portal.
Additional options for purchase are available for system ready and business ready services.
The Solaris 10 Operating System and other software provided with the system is covered under the Warranty support during the Try and Buy period.
Best Regards
Mark Cradock
Sun Microsystems
Tel: + 353 (0)599136768
mark.cradock@your-sun.com
www.sun.com
Hi Mark,
I am debating the Sun Fire X4200 or the Sun Ultra 40 Workstation. Perhaps it would be better for me to explain the intended usage and get your feedback.
First, a little background. We had a FreeBSD box running for about 4 years without reboot. As is probably obvious from that statement, we were behind on the security updates. While we were hosting multiple domains, they were not in Xen or Jail or anything. On Christmas eve a year ago, a hacker from a Polish cable isp hacked in through one of the user accounts and wiped out the entire system. Because of this, I decided this time that I wanted to ensure that the various domains stay secure even if one is compromised.
We bought the latest top-of-the-line Asus, AMD, etc. Unfortunately, AMD lied about the capabilities of the board. They said it was capable of RAID-5 on SATAII, but in actuality it is capable of RAID-5 OR SATAII. Then we installed FreeBSD-CURRENT, and found out it doesn't support RAID-5 yet. Needless to say, it has been a hassle.
So we are debating buying a Sun server and running Solaris on it. My expectation is that there should not be any driver compatibility problems at that point.
So long story not so short -- we are looking at trying to run a bootable RAID-Z (yes, I know that requires some tweaking, but I want to ensure that the boot is also recoverable) with a Xen Dom0. On top of that, we want to allow each domain to run its own OS (whichever they choose, thus prefer AMD-V) as a DomU. Based on this, I think that each DomU would get the advantage of Raid-Z on the underlying filesystem, even if they didn't know about it directly, and regardless of which OS they are running.
Do you have any thoughts, concerns, or questions?
Malachi
A followup:
Hi Malachi,
First of all, sorry to hear about the experiences you've had with the FreeBSD and Asus! Not good.
Secondly, one of the reasons for the Try and Buy promotion is that you can test the capabilities of each machine prior to actually buying it. I'm sure that you will have a positive experience and benefit from this.
I would recommend that we assign a server specialist to you, that can be available prior to your choice of machine, and also be available for you right through the trial process to assist you in your testing, additional components, adjustments in configurations, etc. Would you have a
contact number that we can call you on?Best Regards
Mark Cradock
Sun Microsystems
Tel: +
353
(0)599136768
mark.cradock@your-sun.com
www.sun.com
So the rest will most likely be via phone.
15 January 2007
Chat with agent about Sun Try and Buy Products
Please wait while we find an agent to assist you.
You have been connected to Brenda Byrne.
Brenda Byrne: Welcome to Sun Microsystems. How may I help you?
Malachi de AElfweald: Try 2.
Brenda Byrne: Sorry about you getting pushed off a moment ago
Malachi de AElfweald: Hi Brenda. I have a question about Try and Buy products.
Brenda Byrne: Sure - go ahead!
Malachi de AElfweald: I am looking at replacing my server with one to run Solaris (or OpenSolaris), Raid-Z and Xen
Brenda Byrne: ok
Malachi de AElfweald: I assume that the Try and Buy products would all be 100% compatible with Solaris and OpenSolaris, correct?
Brenda Byrne: yes, of course
Malachi de AElfweald: How does the Try and Buy program work?
Brenda Byrne: You can simply receive one of the listed Sun machines on trial, no questions asked, for 60 days, with shipping costs provided by Sun, and at the end of the 60 days, or prior to then, make a choice to keep the machine....or buy it!
Brenda Byrne: Would you be trialing a machine on behalf of a company, or in a personal capacity?
Malachi de AElfweald: Company
Malachi de AElfweald: And do they come with the OS preloaded, or do I install it?
Malachi de AElfweald: Also, is there a subscription fee, or just a one-time cost to purchase?
Brenda Byrne: OS is preloaded
Brenda Byrne: there is a one-time cost to purchase the machines
Brenda Byrne: if I understand you correctly
Malachi de AElfweald: so, for example, it would come with... Solaris10? no subscription/support costs for it?
Brenda Byrne: Yes - they come with Solaris 10 - which is a free operating system anyway.
Brenda Byrne: But no, Solaris 10 customer support does not come provided with the machine
Malachi de AElfweald: available but not required?
Brenda Byrne: There is a fee for signing up to customer support for Solaris 10 (which is not required)
Brenda Byrne: Solaris 10 customer support fees start at $10 per month
Brenda Byrne: What kind of a machine are you thinking of trialing?
Malachi de AElfweald: kk. and is there a way to specify how we want it configured? ie: bootable RAID-Z
Brenda Byrne: yes
Malachi de AElfweald: ok. it all sounds pretty good. I guess at this point I just need to figure out which machine I want.
Malachi de AElfweald: is there anything else I should know?
Malachi de AElfweald: any requirements to qualify or anything?
Brenda Byrne: Nope...except that it is on behalf of your company, and is delivered to your company address
Brenda Byrne: What kind of applications would you test on it?
Malachi de AElfweald: That's fine.
Malachi de AElfweald: A year ago, a polish hacker wiped out our FreeBSD box. everything. multiple domains. I told the FBI, but they didn't even ask what the IP address was.
Malachi de AElfweald: So, I am looking at replacing that server with a new one.
Malachi de AElfweald: Expected usage is that it is going to be a Xen Dom0 and run each domain as a DomU.
Malachi de AElfweald: Primarily, I am a java developer - but that won't really matter since each domain will have its own OS
Malachi de AElfweald: Ok, I think I have all the information I need. Just need to try to decide which machine to go with.
Malachi de AElfweald: Thank you for your assistance.
Brenda Byrne: no problem.
Your session has ended. You may now close this window.
You have been connected to Brenda Byrne.
Brenda Byrne: Welcome to Sun Microsystems. How may I help you?
Malachi de AElfweald: Try 2.
Brenda Byrne: Sorry about you getting pushed off a moment ago
Malachi de AElfweald: Hi Brenda. I have a question about Try and Buy products.
Brenda Byrne: Sure - go ahead!
Malachi de AElfweald: I am looking at replacing my server with one to run Solaris (or OpenSolaris), Raid-Z and Xen
Brenda Byrne: ok
Malachi de AElfweald: I assume that the Try and Buy products would all be 100% compatible with Solaris and OpenSolaris, correct?
Brenda Byrne: yes, of course
Malachi de AElfweald: How does the Try and Buy program work?
Brenda Byrne: You can simply receive one of the listed Sun machines on trial, no questions asked, for 60 days, with shipping costs provided by Sun, and at the end of the 60 days, or prior to then, make a choice to keep the machine....or buy it!
Brenda Byrne: Would you be trialing a machine on behalf of a company, or in a personal capacity?
Malachi de AElfweald: Company
Malachi de AElfweald: And do they come with the OS preloaded, or do I install it?
Malachi de AElfweald: Also, is there a subscription fee, or just a one-time cost to purchase?
Brenda Byrne: OS is preloaded
Brenda Byrne: there is a one-time cost to purchase the machines
Brenda Byrne: if I understand you correctly
Malachi de AElfweald: so, for example, it would come with... Solaris10? no subscription/support costs for it?
Brenda Byrne: Yes - they come with Solaris 10 - which is a free operating system anyway.
Brenda Byrne: But no, Solaris 10 customer support does not come provided with the machine
Malachi de AElfweald: available but not required?
Brenda Byrne: There is a fee for signing up to customer support for Solaris 10 (which is not required)
Brenda Byrne: Solaris 10 customer support fees start at $10 per month
Brenda Byrne: What kind of a machine are you thinking of trialing?
Malachi de AElfweald: kk. and is there a way to specify how we want it configured? ie: bootable RAID-Z
Brenda Byrne: yes
Malachi de AElfweald: ok. it all sounds pretty good. I guess at this point I just need to figure out which machine I want.
Malachi de AElfweald: is there anything else I should know?
Malachi de AElfweald: any requirements to qualify or anything?
Brenda Byrne: Nope...except that it is on behalf of your company, and is delivered to your company address
Brenda Byrne: What kind of applications would you test on it?
Malachi de AElfweald: That's fine.
Malachi de AElfweald: A year ago, a polish hacker wiped out our FreeBSD box. everything. multiple domains. I told the FBI, but they didn't even ask what the IP address was.
Malachi de AElfweald: So, I am looking at replacing that server with a new one.
Malachi de AElfweald: Expected usage is that it is going to be a Xen Dom0 and run each domain as a DomU.
Malachi de AElfweald: Primarily, I am a java developer - but that won't really matter since each domain will have its own OS
Malachi de AElfweald: Ok, I think I have all the information I need. Just need to try to decide which machine to go with.
Malachi de AElfweald: Thank you for your assistance.
Brenda Byrne: no problem.
Your session has ended. You may now close this window.
14 January 2007
Try Sun products risk free for 60 days!
Thanks to Lee for providing this link... Looks like they have the Sun Fire X2100 M2 Server for $945 and the Sun Fire X2200 M2 Server for $1595 or even a Sun Ultra 40 M2 Workstation for $2295... oh, how to decide ;)
12 January 2007
FreeBSD + ZFS?
Well, according to this forum post, it looks like ZFS is stable on FreeBSD now. Would it be better to run RAID-Z on FreeBSD due to the licensing issues and OS familiarity?
Solaris + Xen
Here is the Xen forum at OpenSolaris' website.
Installation instructions for Solaris dom0/domU
And one persons' experiences and notes trying to do so
Installation instructions for Solaris dom0/domU
And one persons' experiences and notes trying to do so
Subscribe to:
Posts (Atom)