IT Storage: balancing performance, capacity & price

IT Storage: balancing performance, capacity & price

Every minute of every day   – YouTube receives 48 hours of uploaded video – Over 2 million search queries hit Google – Twitter users post 100,000 tweets – 571 new websites are created – Over 200,000,000 emails are created and sent     The size of digital data the world over is estimated to be nearly 1.2 zettabytes; that’s about 1.3 trillion gigabytes! In 1997 Michael Lesk (original Unix team) theorised that there was 12,000 petabytes of data stored globally. The web was thought to be increasing 10-fold annually.  Estimates today reckon 2.5 quintillion bytes of data is written daily and 90% of global data has been created in the last two years!  14.7 exabytes of new data is expected for this year!  The more we generate, the more we preserve and protect data with backup and replication, driving the demand for IT storage media even higher.  One of the biggest challenges continues to be our inability to predict how much we need and when the increased storage amounts are needed. Traditionally, and before virtualisation, storage was predictable.  Administrators could predict the communication (input/output) between systems and these were relatively sequential.  Now with the heavy workloads of virtual machines, the I/O processes become random (data could be anywhere on the disk).  The read/write activity on storage disk heads is increased which results in increased latency (the length of time taken to discover data). Virtualisation technology and cloud-based applications consolidate compute and storage but need high performance and high capacity storage to handle the high volume transactions of the number of concurrent users. Storage systems store data and metadata together....

A comparison of AoE to FC and iSCSI protocols

One of the first issues I have to contend with when talking about Coraid storage and its use of the ATA-over-Ethernet (AoE) protocol to transfer data, is the response “Ethernet? Oh, so it’s iSCSI then?”. No it isn’t…. AoE was built from the ground up as an open source data transfer protocol, specifically concerned with finding the most efficient way to transmit raw disk I/O commands over raw Ethernet, and keeping the overhead as low as possible to maximize the throughput. In many ways AoE is more akin to Fibre Channel (FC) than it is to iSCSI in that it is a non routable protocol designed for locally based storage rather than sending data over the Internet. Like FC, AoE can be made to route over the Internet when it needs to, such as in site-to-site DR applications, but the non routable nature of the protocol makes accidental exposure of data to non authorized networks that much harder. So in order to help differentiate the data transfer protocols upon which all your networked storage systems are based, this blog entry is here to help dispel some of the myths about AoE. The only real comparison of AoE and iSCSI is that they both use Ethernet as the transport medium. iSCSI uses TCP/IP at Layer 4 and AoE Layer 2, but after that things get very different. Data delivery the iSCSI way The diagram below shows how data is sent from a client to a disk device using the iSCSI protocol.     iSCSI is a connection based topology, as is FC, and therefore requires sequenced serial delivery of the...

A closer look at storage and virtual desktops

Virtual Desktop Infrastructure (VDI) is becoming a hot topic in the wake of server virtualisation. It was one of the areas that generated the most attention at the recent IPExpo 2010 in October and certainly looks to gain momentum in the following few years as companies complete their server virtualisation programmes and look to transfer that knowledge to the desktop. However, assuming that server virtualisation knowledge is enough to then go and virtualise the PCs in your environment is where you may well come unstuck. There is a killer named IOPS lurking in the shadows, which you ignore at your peril! Firstly we must understand the problem when tackling the problem of desktop virtualisation. The intial startup and login A Windows PC running on a local hard drive generally uses a IDE or SATA drive running at 5400 or 7200 rpm. Typically these disks can deliver 40 or 50 IOPS (Input Outputs Per Second), which is enough for a single user. When a PC starts it loads the basic OS and a number of services. Many of these services that exist specifically to optimise a physical disk environment, such as indexing and prefetching, and produce a lot of IOPS that may benefit a single disk system but when combined with many other virtual desktops can cripple a virtual desktop system. The amount of IOPS a Windows client produces varies according to the applications loaded and how heavy the use is, but generally 8 to 10 IOPS for a the average user and 14 to 20 IOPS for heavy users is not too far off the mark. However, what is...