The Advanced Computing and Information Systems Lab where I work recently aquired new hardware to support the storage needs of iDigBio.We operate an open source Ceph cluster to provide object storage for media and datasets. iDigBio was too successful at mobilizing digital media about museum specimens... the original storage cluster is full!

The new hardware is racked and has received the base OS (Ubuntu 18.04 LTS). Ceph makes RAID obsolete, but to get the best utility out of the hardware we needed to flash each LSI controller with "IT" firmware. IT in this context means Initiator Target (also known as passthru). Ceph guides suggest IT/JBOD configuration in the storage controllers to maximize performance and to prevent crappy controller caches from interfering with Ceph's own redundancy.

Our servers from Supermicro include the LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 adapter. Each host also contains 11 storage drives at 12TB each (plus an SSD for the boot drive), 16 cores of Xeon Sky Lake processor, a whopping 256 GB of RAM, and 10 gig ethernet plugged into the University of Florida Campus Research Network (CRN).

Here are pictures of myself and Nicholas Rejack doing the firmware updates:


Photo by Grace Hong


Photo by Grace Hong

The new cluster contains over 900TB raw, getting us close to a Petabyte.

I am now working through SaltStack and systemd configurations, working up to the point where we can actually begin migrating objects from the old cluster to the new cluster.