I miss OpenSolaris. At work, the only options are Windows and Linux. The other day, I was rm -rf'ing about 10 different copies of the Android tree and it reminded me how much quicker it was to wipe out a zfs filesystem than it was to rm -rf one "little" directory. After many hours of waiting for access to my system, I decided to check on the state of ZFS on Linux again.
Last year, I had
tried the FUSE port. This time I decided to try
zfsonlinux. This comment on their page had me excited pretty early:
There is a lot of very good ZFS documentation already out there.
Almost all of it should be applicable to this implementation because
we have tried to conform to the Solaris behavior as much as possible.
For installation, I followed the steps listed
here:
sudo add-apt-repository ppa:dajhorn/zfs
sudo apt-get update
sudo apt-get install ubuntu-zfs
Installation went pretty smooth. Trying to use it afterwards was a bit more tricky. The module wasn't loaded so I tried to follow the instructions to do an insmod:
malachi@onyx:~$ sudo insmod /lib/modules/2.6.38-10-generic/updates/dkms/zfs.ko
insmod: error inserting '/lib/modules/2.6.38-10-generic/updates/dkms/zfs.ko': -1 Unknown symbol in module
Well, that's not good. Luckily, there was a really simple workaround:
malachi@onyx:~$ sudo modprobe zfs
And voila. It was loaded until next reboot. I didn't have a spare drive in the laptop to convert to a root zfs drive, so I didn't worry about doing a grub setup... but I would like the module to auto-load on boot. Turns out, that's pretty simple. Simply add a line 'zfs' to /etc/modules. While you are at it, modify /etc/default/zfs to enable auto (un)mounting:
malachi@onyx:~$ cat /etc/default/zfs
# Automatically run `zfs mount -a` at system startup if set non-empty.
ZFS_MOUNT='yes'
#ZFS_MOUNT=''
# Automatically run `zfs unmount -a` at system shutdown if set non-empty.
ZFS_UNMOUNT='yes'
#ZFS_UMOUNT=''
Currently, the /dev/zfs is owned by root:root with permissions set to 600. What this means is that to even do 'zfs list' or 'zpool status' you have to be root. While I understand the logic there, I find it extremely annoying. It may not go anywhere, but I submitted an
enhancement request to allow admin group rw. I realize that there are ACL mechanisms, but I do not believe those apply to the /dev/zfs special character device.
So, with no spare disk, how will I test it? While I know that performance would suffer, I decided to try my testing with file-backed zfs. I didn't want to keep doing it by hand, so created a script:
malachi@onyx:~$ cat /mnt/zfs-disks/createZFSFile
#!/bin/bash
FILE=$1
GIGS=$2
dd if=/dev/zero of=/mnt/zfs-disks/$FILE.zfs bs=1G count=$GIGS
ls -lh /mnt/zfs-disks/$FILE.zfs
Then created a few backing disks....
malachi@onyx:~$ createZFSFile disk1 10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 114.092 s, 94.1 MB/s
-rw-r--r-- 1 malachi malachi 10G 2011-08-12 15:41 /mnt/zfs-disks/disk1.zfs
Repeat that for disk2,3,4... For this testing, and especially since it was file-backed, I decided to not use raidz2 like I normally would so that I could easily add more space to the pool if necessary.
malachi@onyx:~$ sudo zpool create pool /mnt/zfs-disks/disk1.zfs
I then created a filesystem in the pool to work from:
malachi@onyx:~$ sudo zfs create pool/android
malachi@onyx:~$ sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool 128K 9.78G 30K /pool
pool/android 30K 9.78G 30K /pool/android
Now to test adding some disks to the pool...
malachi@onyx:~$ sudo zpool add pool /mnt/zfs-disks/disk2.zfs
malachi@onyx:~$ sudo zpool add pool /mnt/zfs-disks/disk3.zfs
malachi@onyx:~$ sudo zpool add pool /mnt/zfs-disks/disk4.zfs
malachi@onyx:~$ sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool 140K 39.1G 31K /pool
pool/android 30K 39.1G 30K /pool/android
malachi@onyx:~$ sudo zpool status
pool: pool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
/mnt/zfs-disks/disk1.zfs ONLINE 0 0 0
/mnt/zfs-disks/disk2.zfs ONLINE 0 0 0
/mnt/zfs-disks/disk3.zfs ONLINE 0 0 0
/mnt/zfs-disks/disk4.zfs ONLINE 0 0 0
errors: No known data errors
That all looks good. Downloading Android, built it, and then tried some timings...
root@onyx:~# annotate-output zfs snapshot pool/android@freshbuild
21:55:14 I: Started zfs snapshot pool/android@freshbuild
21:55:14 I: Finished with exitcode 0
root@onyx:~# zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
pool 10.1G 29.1G 31K /pool
pool/android 10.1G 29.1G 10.1G /pool/android
pool/android@freshbuild 0 - 10.1G -
root@onyx:~# annotate-output zfs destroy pool/android@freshbuild
21:56:00 I: Started zfs destroy pool/android@freshbuild
21:56:00 I: Finished with exitcode 0
root@onyx:~# zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
pool 10.1G 29.1G 31K /pool
pool/android 10.1G 29.1G 10.1G /pool/android
root@onyx:~# annotate-output zfs destroy pool/android
21:59:54 I: Started zfs destroy pool/android
22:00:28 I: Finished with exitcode 0
root@onyx:~# zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
pool 1.52M 39.1G 30K /pool
root@onyx:~# annotate-output zfs create pool/android
22:01:05 I: Started zfs create pool/android
22:01:06 I: Finished with exitcode 0
root@onyx:~# zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
pool 1.55M 39.1G 30K /pool
pool/android 30K 39.1G 30K /pool/android
Overall, I am really happy with these timings... Especially since it was running on file-backed zfs.
During all of my testing (I tried it 3-4 times) there was one gotcha. During my first test, git locked up downloading Android. Not sure what happened, but I couldn't even kill -9 it. Even rebooting wouldn't work since it hung waiting for that process to die. Since I had to hard power-cycle it, I am not surprised that the zfs system got corrupted. When I rebooted, 'zfs list' and 'zpool status' locked up. I was able to kill -9 the zfs list; but not the zpool status. I deleted the /etc/zfs/zpool.cache and on reboot it no longer knew about the pool, but no longer locked up. I was able to recreate everything and did not see that error happen again. Maybe there was a better way to fix the problem.
I think for the next Linux system I build, I may try a root zfs partition using zfsonlinux. Even with the one (serious) issue, I was pretty happy with the results of the testing.
Thanks to
aarcane for
finding a solution that allows you to give users access to the zfs/zpool commands. Recapped here:
root@onyx:~# nano /etc/udev/rules.d/91-zfs-permissions.rules
#Use this to add a group and more permissive permissions for zfs
#so that you don't always need run it as root. beware, users not root
#can do nearly EVERYTHING, including, but not limited to destroying
#volumes and deleting datasets. they CANNOT mount datasets or create new
#volumes, export datasets via NFS, or other things that require root
#permissions outside of ZFS.
ACTION=="add", KERNEL=="zfs", MODE="0660", GROUP="zfs"
root@onyx:~# groupadd zfs
root@onyx:~# gpasswd -a username zfs
Now, reboot. Upon login, the specified user should be able to run
zfs list and
zpool status without using sudo.
