Data Storage
When people think “high performance” on server and systems infrastructure, they think of things like RAM (memory), and CPU (processor). When building out this equipment, they will usually lavish boatloads of both, to ensure their systems and infrastructure are performant.
But, all too often, one of the most important and least acknowledged part of an infrastructure-build, and is also the culprit for performance bottleneck issues and the dreaded “lag” that end users complain about, is storage.
Many people tend to think you just need to have “enough” storage. A bunch of Terabytes to accommodate storing all of your company’s files, and that’s it.
However, while this may have been the case in architected solutions in the days of old, this is just a small role that storage plays in modern infrastructure designs. Two big factors play into the shift in architecture and infrastructure design, and the way storage shifted from being just strictly storage, to being a high demand resource that can pinch the performance of a system if not designed and implemented properly.
Virtualized Systems
This biggest factor is the emergence of virtualized systems, and the shift over to virtualized architectures. Suddenly, storage was not just storing files, it was serving up “volumes” for virtual machines to mount to. These virtual machines could be over-provisioned and forced to fight for resources on a volume that possibly was designed more for file storage than it was for virtual hosting.
High Disk I/O
If the storage RAID configuration is not optimized for performance and resiliency, then it is going to struggle to perform with an ever increasing demand from new virtual machines being mounted to it at a growing rate, and if the disks are all “slow” “spinning’ 5200 RPM drives, then the problems are exacerbated, usually in the form of high disk I/O.
Disk I/O is defined as the disk input/output operations (in KB/s). Put simply, it is the speed at which data is transferred between the hard drive (or volume) and the RAM (memory). Even in conventional systems, you can see how this relationship makes storage (hard disk(s)) a vital piece of your hardware.
Read/Write Requests
When it gets bogged down with read/write requests, it will slow down your entire system. On individual PCs we have addressed this bottleneck with Solid State Drives (SSDs), which dramatically improves performance on that personal machine. Sadly, when it comes to architecting infrastructure, the storage solution is not nearly as straight forward or simple.
Things like SQL servers have become much more ubiquitous. While everyone knows SQL is a memory hog, not everyone realizes that all of those SQL database reads and writes translate to reads and writes on the disk, which means high I/O. High I/O means decreased performance on your storage, and if you are running virtual machines mounted to storage, that means slow performance for everything mounted to that storage.
At this point a pattern begins to emerge, slow storage has a cascading domino effect that impacts your organization in a very real, and very noticeable way.
So how do you address these major infrastructure storage problems/needs? Well, the solution is going to differ based on the need, the size of the organization, and the business model. For large enterprises managing dozens of virtual servers with high I/O need, an enterprise grade SAN is generally going to be the solution. Something expandable, resilient and performant, with a hybrid of slow spinning disks, fast spinning disks (10,000-15,000 RPMs), and SSDs, that can intelligently determine where to put data chunks based on how frequently those data chunks are referenced and needed.
If that type of system sounds expensive, that’s because it is. A Nimble SAN, or HP 3PAR gets in the six figures easily; but, that is also for intense need (think major oil companies chewing on massive amounts of geographic charting and graphing data on a daily basis for a national or global operation).
SMB Solutions
For small and medium sized businesses, it makes a lot more sense to just have performant disk arrays in the proposed server build, built in a RAID 6, with a hybrid of fast disk and SSDs. Usually you are going to be running less than a dozen virtual servers (VMs), and this should serve the needs of your organization without a problem.
However, the importance of properly building out the storage side of your new server build cannot be stressed enough. This is usually the most expensive part of the build (RAM and CPU are actually pretty cheap), which is why many less experienced consultancies will leave it off of a build quote, and more experienced ones will insist on including it, even though this necessity makes a quote, simply put, more expensive.
Additional Reading: CSRF Guide, The Important of Social Media Security, Social Media Security Tips, PCI Compliance, SOC 2, HIPAA Compliance, API Scanner
IT Solution Quotes
Discerning business owner, you want to learn to spot the difference between a quote that, while maybe lower, does not serve the best interest of your organization over the long run, vs a proposal that has risked appearing less competitive in the interest of ensuring your organization’s longevity and sustainability from an IT infrastructure stand point.
IT is a lot to navigate. One of the best places to start when getting quotes and going through the bid process, is to see the level of detail different organizations are providing. You will typically find that an organization that is coming in at a higher price point, is also providing a lot more information with that quote, so that you can see exactly why the quoted it that way. Those same organizations will generally be more than happy to answer any questions and discuss their reasoning, and why they came to the conclusions they did when designing a system for your organization.
Caveat Emptor
This will not always be the case, be cautious of quotes that are higher than others and short on details. In addition, quotes should always be itemized. If a quote just shows a lump sum, and generic terms like “hardware” and “labor”, without any definition or itemization, that is a red flag. Demand transparency in your quoting and bidding process. Anyone who cannot meet these basic standards should be disqualified, and you will save yourself (and your CPA), a lot of headache down the road.
SOW (Statement of Work/Scope of Work)
Finally, it always helps when vendors quoting you projects provide a SOW (Statement of Work/Scope of Work). This defines what the scope of the project is and will generally itemize out the labor hours so that you can see where that time is going. Drawing strict definitions of the work to be done is important for both your organization, and the vendor, so that expectations can be properly managed, and successfully met.
Ultimately, storage is one of those pieces of the puzzle that has long lived in the shadows and gone ignored. Sometimes because of past precedence and confusion over the way new and emerging technologies work, and, other times, because the cost of properly designing and building out storage can discourage people from building out the most performant solution.
IT Debt
But, make no mistake, not doing it right in the beginning, usually means spending twice as much to remediate the problem further down the road. That cost you deferred will return, and with interest (we refer to this as “IT Debt”, but that is a conversation for another time).
So, if it feels like your systems are sluggish and slow, or that the new infrastructure you purchased is not really performing up to the original expectations, perhaps a site survey and evaluation are in order. Contact a competent IT vendor and see if they can properly diagnose what ails you. Who knows? It may just be your storage was overlooked in the last design implementation.
