High Performance Samba Server on Freebsd

Problem

A midsize educational organisation has a requirement to upgrade the operating system on several hundred desktop computer from Windows XP to Windows 7. The timescale to complete the reinstallation is the summer recess. A master disk image has been prepared and tested. The installation process involves PXE booting the the machines into a Windows PE environment, then mounting a cifs file share where a Windows Imaging Format disk image is accessed. The WIM images and associated files are greater then 20 gigabytes in size, and will be accessed simultaneously by all client's being reimaged.

Solution

Legacy hardware was reinstalled with FreeBSD, using the ZFS filesystem, and Samba to provide a file sharing solution where constraining factors became the edge network and client PC's rather than the disk access on the image file server.

Overview

When similar reinstallation tasks had been undertaken on earlier occasions it had been found that simultaneous random disk read access had been a limiting factor. A common way to resolve issues of simultaneous access on file servers is to use multiple disks, and "stripe" the file system across them. In this way the disk read access can be increased approximately linearly with the number of disks. Storage Array Networks commonly have large numbers of disks and high capacity network interfaces to make sure that many random disk accesses can be supported. These enterprise products will come with an enterprise price tag though.

Another way to provide fast disk access is to make use of RAM disks. System RAM can be configured to appear as a disk device, providing fast access, but without any persistence after reboot.

The organisation had an old HP Proliant DL380 G5 with 48G RAM, 2x72G SATA disk drives in a hardware mirror configuration ( RAID 1 ) , and 4x 1Gbit network interfaces. This had previously been used as a VMWare host, but had reached end of life. ( It was 6 years old and out of hardware warranty ).

The relatively large RAM to available disk space lead us to choose to install FreeBSD 9, and use the ZFS file system, and export the file system using Samba for Windows clients. This gives us the ability to cache the current working set of files in RAM dynamically, giving high performance. Network bonding allows us to use all the available network interfaces.

ZFS

ZFS is well supported on FreeBSD, and has some amazing features to provide robust and performant disk access, including some smart caching mechanisms. By default ZFS will use up to 80% of the available system RAM for its ARC cache.

In the use case we are looking at the first client access will cause all the required image files to be cached. This will mean only minimal disk access is required, and disk access should not be a bottleneck.

Installing FreeBSD 9 with a ZFS root partition

Because of a limitation of the HP Proliant hardware raid, we cannot access the disks directly, but only as a single abstracted RAID 1 Mirror. This means we rely on the HP hardware to inform us of disk errors.

Installing to a ZFS root partition is not directly supported from the FreeBSD 9 installer ( I believe it is in FreeBSD 10 ). To circumvent that we use a modified installation method described here: Installing FreeBSD 9.0 (or later) Root on ZFS using GPT

Samba

Samba gives us compatibility with Windows. We can join the Active Directory organisation and control access to files and share through Windows groups. Using Samba to join a Windows domain, as a member server, is well documented at samba.org.

Some additional options for performance

socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
min receivefile size=16384
use sendfile=true
aio read size = 16384
aio write size = 16384

and here is the complete smb.conf

[global]
workgroup = CHSTFADMIN 
netbios name = wye 
wins server = 172.25.1.65
wins server = 172.25.1.67
security = DOMAIN
printcap name = cups
disable spoolss = Yes
show add printer wizard = No
idmap uid = 15000-20000
idmap gid = 15000-20000
winbind use default domain = Yes
printing = cups
server signing = mandatory
socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
min receivefile size=16384
use sendfile=true
aio read size = 16384
aio write size = 16384


[dist2013]
    path=/export/dist2013
    comment = College Win7 installer
    read only = yes 
    read list = @sgimageinstallers
    write list = @sgimagebuilders
    valid users = @sgimageinstallers @sgimagebuilders
    create mask = 0775
    directory mask = 0775 

NIC bonding

FreeBSD lets us bond interfaces using LACP to get loadbalancing and increased throughput. This is configured in the /etc/rc.conf

ifconfig_bce0="up"
ifconfig_bce1="up"
ifconfig_bce2="up"
ifconfig_bce3="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport bce0 laggport bce1 laggport bce2 laggport bce3 172.25.1.197/16" 

This balances the outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. The hash includes the Ethernet source and destination address and, if available, the VLAN tag, and the IPv4 or IPv6 source and destination address.

The switch ports also need configuring as a LACP group.

Results

Here are some munin based graphs that show performance over the last year. Peak usage was in July. The munin rrd statistics are averaged down over time, so it's likely that peak will have been higher.

Network throughput

We can see total network throughput here

And how this is distributed across the bonded interfaces.

Simultaneous users

The numbers of connected users

ZFS ARC Cache effiency

Cache hits approach 100%

Disk IO

Disk IO read approaches zero

Conclusion

Free Software can be combined with legacy hardware for the win!. The hardware is worth about £300 and is of the type commonly found in skips at medium size organisations. Free software extends it's life and provides utility exceeding proprietry solutions.