Knowing the time when you’ve outgrown shareware tools like Iometer and VDBench

There comes a time when freeware products just aren’t going to cut it

Here at Load DynamiX, we often hear comments like, “I understand the importance of performance testing of storage arrays, but I’ve been using shareware tools for years and they are doing an OK job. Why should I spend money on Load DynamiX?”

This is a valid question, as we know that money doesn’t grow on trees and IT budgets are always stretched to capacity. And quite honestly, do you really want to spend that already stretched budget on testing resources when you can just buy more storage hardware and over-provisioning to solve performance issues?

Well, of course, we here at Load DynamiX are always going to be in favor of you purchasing our solutions. I mean, that’s what we’re in business for, right? So we took that question and gave it to one of the storage industry’s foremost storage system testing experts for two reasons—1) To demonstrate the confidence we have in our solutions, and 2) to take the bias out of the equation. In December 2015, we asked Demartek to give an impartial evaluation to analyze where the shareware tools are most appropriate to use and to determine the tipping point of justifying the investment in a more robust solution like Load DynamiX.

Basically, the study reached some useful conclusions concerning the use of testing tools that employ synthetic workloads to profile and analyze storage performance. One of the key conclusions is that shareware tools are often “good enough” for smaller, one-off tests. But shareware tools become difficult to manage and lack important features when requirements scale and the workload becomes more complex. Here’s a more in-depth look at the Demartek analysis. In most cases, these issues increase in importance as your needs scale, and as your storage budget increases:

  • If you’re embarking on a quick, one-off test that doesn’t need much load generation horsepower and never needs to be repeated, shareware tools are fairly easy to get started with. However, when your scenarios get large, you may need to purchase dozens or even hundreds of vSphere licenses and dozens of servers to support the shareware. At this point, it would make sense to purchase a dedicated Workload Generation Appliance, such as Load DynamiX, which is designed to scale to exceed the performance of even the biggest scale-up storage arrays.
  • If you are doing a simple protocol test, shareware tools are an excellent resource. However, if you need to test a mixed application workload or a workload with hundreds of LUNs, the effort to build a realistic shareware model can easily take days with these tools. Again, if you’re only doing this once a year, a shareware tool will probably suffice. If you’re doing ongoing testing, tools that require an investment will include workload acquisition products that can import your array data or switch data allow you to automatically create a complex workload with a click of the mouse. Purchasing Load DynamiX can save man-months of work per year per workload and easily justify the investment.
  • If you are running a single test and you’re the only one doing this work, you don’t need much “test management.” If you’re running multiple tests, or others are using the system and want shared access to test beds, tests and reports, then with open source tools, you’ll need to invent your own test management system to make sure things (like your results) don’t get accidentally overwritten. At this point, you should evaluate the cost of test management tools along with the scripting man hours required in these complex test beds. Compare that cost to an investment in Load DynamiX and see where your ROI really comes in to play.
  • Simple test profiles, with standard read/write and random/sequential metrics, are fairly straightforward with shareware tools. However, if you are looking for the greatest realism, you will want the ability to granularly adjust dedupe and compression ratios, use features like seeded random and sequential data patterns, and account for both temporal and spatial data patterns. Realism is further enhanced if you can separately configure read and write components for block size, random/sequential mix, outstanding requests, and LUN range utilization, and match your particular MPIO strategy. To be able to do any of the above, you’ll need Load DynamiX.
  • With shareware tools, you can somewhat easily import output files to Excel for graphing. If you’ve had to use several physical or virtual servers to generate the load, this effort will be time consuming and error prone. If you’re only doing this once a year, that is most likely acceptable. But if you’re doing this every week/month, products like Load DynamiX have reporting tools with built-in graphs and reports, which are automatically generated as the tests run–saving you man-weeks of work in a given year and eliminating possible errors as compared to the two-step shareware process.
  • If you’re only testing a single array and it will never need to be compared with another array again, then shareware tools will probably be fine. If you need to repeat the test to achieve an apples-to-apples comparison at another data center or at a later date, then you’ll need to freeze your shareware/hardware test bed, and that will mean no bug fixes, security updates, or server virtualization upgrades, ever. That probably won’t go over well with your IT department. For consistent, repeatable results you will need to invest in dynamic testing tools. You can even send your workload to your storage vendor, and know that his test will run exactly the same on his Load DynamiX appliance for purposes of replicating your problems, which makes troubleshooting and other issues so much easier for everyone.
  • As with most shareware, you’re going to have to accept that support for these tools are limited. You don’t get what you don’t pay for. If your tests are not urgent, and you can afford time to ask for help on a user forum, you might be able to get the help you need with shareware in a few days. There are a lot of smart people out there willing to help. If your tests are urgent, or there is a large implication for how your company spends its CAPEX, you may find that having access to a paid support team, or professional services department at Load DynamiX, is by far the most critical component of your test plan.

In summary, what Demartek found was probably something we all know: shareware and open source tools have a valuable place as an entry level tool in IT organizations. They can be useful tools for a one-off project or specific job. But, specifically for business critical storage infrastructure, Demartek outlined some important inflection points for storage professionals to note as they begin to spend millions per year on storage, and are on the hook for making rock-solid recommendations for future purchases. If you haven’t seen the full report, just send me an e-mail requesting it.

Jim Bahn
Product Marketing