Summary
Benchmarking database storage has evolved beyond outdated tools like SLOB, with modern methodologies leveraging YCSB, FIO, and real-world simulations, while Pure Storage’s FlashBlade//S500 sets new performance standards with the STAC-M3 Benchmark for high-speed analytics.
Here at Pure Storage, our TPC-C tests have been pretty impressive. We regularly run our unaudited HammerOra harness at over 1.73 million tpm (in ARCHIVELOG mode!), which ranks 6th among the audited results.
The problem with HammerOra is that it doesn’t actually push the storage sub-system to its limits. The bottleneck in these tests is almost always a combination of CPU resources and concurrency. Maybe that’s why you don’t see any storage vendors sponsoring TPC benchmarks.
To demonstrate what our storage is actually capable of, we like to use the “Silly Little Oracle Benchmark” (SLOB) tool which was developed by Kevin Closson. Kevin was the Performance Architect in Oracle’s Exadata group, and he now works for EMC. Unlike HammerOra (and Swingbench), SLOB is specifically designed to drive a lot of physical I/O without any application contention. In other words, SLOB shows just how much Oracle-generated I/O your storage subsystem can do.
For example, on the read-intensive test (128 readers, 0 writers), the Pure Storage FlashArray can drive over 2GB/s of throughput at over 250,000 IOPS with sub-millisecond latency:
Naturally, latency is a concern. As the histogram shows, we average 1ms latency for the duration of the test run.
We will have more SLOB results and discussions soon.
If you are evaluating a Flash Storage solution, send us your AWR report and we’ll schedule a free consultation with our Oracle performance experts to talk about how Pure Storage’s FlashArray can improve the performance of your Oracle environment while actually reducing the cost of your storage architecture.
Updated Information:
Advancements in Storage Benchmarking Practices
Over the past decade, storage benchmarking has evolved significantly, with new tools and methodologies emerging to provide more accurate and comprehensive performance assessments.
1. Modern Benchmarking Tools:
Yahoo! Cloud Serving Benchmark (YCSB): Developed by Yahoo!, YCSB is a popular tool for evaluating the performance of different types of database systems, particularly NoSQL databases. It generates workloads that simulate real-world operations, allowing for customizable testing scenarios.
FIO (Flexible I/O Tester): FIO is a versatile benchmarking tool that allows users to simulate various I/O workloads, providing insights into the performance characteristics of storage systems under different conditions.
2. Best Practices in Benchmarking:
Standardized Workloads: Utilizing standardized workloads ensures consistency and comparability across different benchmarking tests. This approach helps in accurately assessing the performance of various storage solutions.
Transparent Reporting: Clearly documenting the hardware configurations, software settings, and testing methodologies used in benchmarks is crucial. Transparency allows for reproducibility and provides context for interpreting results.
Real-world Simulations: Incorporating real-world scenarios into benchmarking tests provides more relevant insights into how storage systems will perform under actual operating conditions. This approach enhances the practical applicability of benchmark results.
3. Pure Storage’s Benchmarking Achievements:
- STAC-M3 Benchmark: Pure Storage’s FlashBlade//S500 has achieved the STAC-M3 Benchmark, an industry standard for tick-analytics benchmarks, specifically designed for database software/hardware stacks managing large time series of market data. This accomplishment demonstrates the high performance and efficiency of Pure Storage solutions in handling complex, data-intensive workloads.
Conclusion
As storage technologies and database systems continue to evolve, staying updated with the latest benchmarking tools and best practices is essential for accurately assessing performance. By adopting modern benchmarking methodologies and leveraging advanced tools, organizations can ensure their storage solutions meet current performance standards and are well-suited to handle contemporary data workloads.
