Storage
There are presently comparatively few (but increasing) products available in the consumer market between add-on disk drives — and there are some very nice 400 gigabyte (GB) Universal Serial Bus (USB) disks and various Network Attached Storage (NAS) storage devices with capacities approaching two terabytes — and enterprise-class storage shelves, and host-attached Serial ATA (SATA) or Small Computer System Interconnect (SCSI, and usually in the Serial Attached SCSI (SAS) variant) controllers, or the associated NAS or Storage Area Network (SAN) Fibre Channel (FC) controllers.
Also becoming available in the consumer storage market is what is known as Reliable Array of Inexpensive Disk (RAID) Array technologies, which are software and/or hardware capabilities intended to allow your data to survive the failure of subset of the disk spindles configured in the RAID array. Disks are moving parts, and moving parts do fail. RAID arrays have been in common use among enterprise configurations, and are now widely available in the consumer market, both with controller-based RAID products and with NAS-based RAID Array products.
Commercial commodity single-spindle storage capacities have progressed to 750 gigabytes during 2006, and have historically and are likely to continue to climb. HoffmanLabs expects to see one terabyte (TB) disks by early 2007. (This terabyte volume capacity is also larger than the unpartitioned address space available within various commercial file systems, though that's certainly fodder for another discussion.)
Seagate provides the largest available single-spindle disk that HoffmanLabs is aware of to date, at 750 GB. Most vendors are offering multi-spindle NAS servers with integrated RAID capabilities. (Having your 750 GB spindle fail, and not have a copy, can be disconcerting. RAID helps avoid mechanical failures, of course, though does require increases the numbers of disks spindles. RAID does not prevent accidental deletions, nor volume structure corruptions.)
You might look at these storage capacities and wonder who might need such capacities, but realize that recording Digital Television (DTV) requires eight gigabytes per hour.
The major storage interconnects available to hobbyists (and to DTV viewers) are USB, SATA, SAS, and NAS over Gigabit Ethernet. USB is certainly widely available, while SATA and SAS are becoming widely available and have higher performance. NAS is certainly a possibility, but you do need to have sufficient throughput and sufficiently low latency via the Gigabit Ethernet and into the NAS server and into its storage.
An Introduction Big Storage
DTV recordings consume prodigious quantities of disk, at the rate of about eight gigabytes (8 GB) per hour. This sans compression, obviously.
This means you're looking at storage. Lots of storage.
On one local Microsoft Windows Media Center Edition (MCE; software which doesn't yet seem fully ready for production, HoffmanLabs might add), an internal 300 GB drive, and a removable 400 GB USB drive, and a stack of five external 400 GB USB drives. Yes, that's 2.7 terabytes (TB). And then there's the stack of DVD+R/RW disks beside the MCE Digital Video Recorder (DVR).
Now as for storage hardware, there are various NAS boxes, such as the Seagate Mirra Personal Server (nice, but rather undersized for DTV) and a gazillion product boxes running the Microsoft product name de jour for its storage-oriented software product. These boxes are basically Microsoft Windows with Disks, many of which are rackmount.
Seagate makes a number of well-regarded disk drives, but is a disk manufacturer and not yet what HoffmanLabs would consider a storage appliance manufacturer. That is, Seagate product information and products are entirely oriented toward its disks as individual products, and not at the sales of shelves and packages. Intel presently offers various iSCSI and storage products for small and medium business markets and at the home market with its Intel Entry Storage System SS4000-E, as well, and vendors such as Yellow Machine are aiming rather more squarely at the high-end home market. Buffalo, Iomega and many other vendors are also active in this market.
When looking at Network Attached Storage or host-attached storage, look at the capacity, at whether or not the product offers RAID capabilities (basically, if the product can survive the loss of a disk), whether or not the controller software is itself stored on the RAID media. Also look at the speed of the interconnect, as single-host USB isn't particularly speedy, nor is a multi-host 100 Mb Ethernet connection, nor is a typical 54 Mb wireless LAN (WLAN) IEEE 802.11g wireless connection.
RAID substantially reduces the total capacity of the storage, but having to perform backups of a 750 GB drive can be tedious, and the loss of even a part-filled 750 GB drive without a backup can ruin your whole day. RAID is a trade-off, but quite commonly a good one. RAID doesn't guard against accidental deletion or corruption of your files, just against failure.
USB is a popular interconnect choice, but it also ties the storage access to a particular system. 400 Mb USB might initially seem fast, but you can end up with a surprisingly slow interconnection with the access speeds of typical devices over the USB. Digital Video Recorders (DVRs) do involve moving large volumes of data around, too.
WLAN is often easy, but a wireless network can sometimes be a real mess to manage. The trade-off with wireless is in the management arena, where the client connectivity is somewhat easier and where the management -- particularly when something goes weird -- is more difficult. Did you know, for instance, that there are only a maximum of three channels available for a B/G-class wireless home network, and often just two channels? The channel assignments have substantial frequency overlaps, meaning that you can see interference from adjacent channels, with the various problems that are caused by interference. Do you know if your neighbor knows how to manage a wireless network, or if you or the neighbor unintentionally configures a WLAN device or a cordless phone that happens to stomp on your WLAN channel assignment? With a wireless network, it's always something.
A wired network has its own problems, particularly when there's a bad cable or connector in the mix. Wired also requires running the wires, which can be surface mounted in various available conduit (and varying degrees of ugliness), or run through the walls. Wire itself is cheap, running the wire isn't. If you are building or remodeling, by all means do include wiring and particularly wiring chases -- even if you have no plans to ever use the wire. The wire is cheap, and cheap to install when the walls are open. Several runs of Category 3 (Cat 3) and Cat 6 wire for telephone and networking, and consider a run or two of single- or multimode optical, and coax for satellite and broadcast HD television connections, and for home monitoring and home security.
HoffmanLabs would recommend the following hierarchy: USB-attached storage for the smallest of home networks, and for those installations that have no plans to expand to multiple systems. Beware the incremental development of a forest of USB devices, however. Wireless storage for small-scale networks, and those covering limited areas or areas where wires are unavailable or impractical. Wired networks are recommended for moderate- to large-scale networks, and networks that must cover larger areas or areas were wireless connections are impractical due to interference or distance, or when raw speed or improved security or improved resistance to interference is required.
As for storage, there are low-end, high-end, and out-of-your-price-range interconnections. Parallel ATA and ATAPI are the low end (and slowly exiting the market, being replaced by Serial ATA and ATAPI connections), Serial attached SCSI (SAS) and iSCSI in the high-end, and Fibre Channel is out of the price range and beyond the needs and requirements of SOHO markets and all but the richest of DTV users.
HoffmanLabs recommends against choosing general-purpose operating systems for use as storage controller operating systems — you really don't want to have a virus-infected storage controller, and HoffmanLabs has seen those. You also don't want to have to manage the controller itself, beyond its initial configuration. You want to have a controller that protects its operating system on the RAID storage media. (If you lose your controller operating system or your controller storage configuration, you can lose your data.) And double your required storage, to allow for RAID requirements. And you want an easy path for upgrades to the firmware, when (and not if) that is required.
For storage controller communications, look for the application protocols used by the operating systems you plan to utilize, SMB/CIFS for Windows systems, and NFS and such for Linux systems. Also look at the availability of clients for your operating system(s), such as clients for iSCSI. Most systems already include USB support, obviously.
External removal access to the disks in the storage device quite is convenient for incremental growth (i.e. adding disks) or for cases where disk replacement is necessary, but costs can obviously be lower when the disks are embedded inside the storage enclosure. External (removable) access can tie you to a vendor, and you'll want to have some idea of the availability of the enclosure as compared with the lifetime of the disks. Removable media can also be convenient (configured outside of the RAID array, obviously) as a way to increase near-line storage; you can have more disks than slots. USB avoids these issues, but typically leaves you with a forest of parts and devices and (for non-bus-powered devices) power cords and wall-warts (transformers), and inevitably with no free USB ports.
Information on storage devices of interest to OpenVMS Hobbyists is also available.