Log analytics data comes from everywhere these days, including security forensics, IoT devices, and cloud metrics.
The wealth of machine log data has business potential that extends far beyond day-to-day IT performance insights. It can deliver improved customer experiences, shed light on the past, present, and future, and empower DevOps and security teams. All told, the operational efficiencies, cost reduction capabilities, and competitive advantage log data yields make it a goldmine.
But log data can also be notoriously complex. We’re learning that, for a decades-old practice, it requires thoroughly modern systems to get right.
Figure 1: Three pillars of a typical log analytics architecture
In “Drive Simplicity for Log Analytics at Scale”, Greg Crosswhite, Senior Solution Architect, Analytics, at Pure Storages, and Somu Rajarathinam, Technical Director at Pure Storage, discuss the challenges of log analytics and how Pure has helped many small and large companies optimize their log analytics at scale.
The Log Data Conundrum
Log data analysis is not new to IT organizations, which means it’s often placed squarely in the territory of legacy systems and practices. But while it’s not new in practice, the types of data it’s asked to handle is. And that’s exactly what’s tripping things up.
“Originally we were looking at HDFS and batch processes, with fewer resources, so it was very reasonable to say we should create systems that scale out and have the disc architecture be a part of the compute.”
—Greg Crosswhite, Senior Solution Architect for Analytics, Drive Simplicity for Log Analytics at Scale
Modern log data is diverse, disparate, and unwieldy. We’re not only talking telemetry and data points from every machine in your organization, every second of the day; today’s log data includes data from streaming sources, cloud environments, containers, and virtual machines. Feeding this into legacy systems, then asking modern questions from it, can exacerbate the problem.
The Many Opportunities for Modern Log Data
It’s easy to see how log analytics at scale can be an uphill battle—but there’s inherent value in it you can’t ignore.
With the right tools to ingest, clean, and analyze log data and the right end-to-end strategy, organizations can derive serious value. Use cases for log data applications are expanding all the time, including
- Cybersecurity: Help security operations improve forensic analysis, predict threats, detect anomalies, and more.
- DevOps and software development: Empower teams to reduce duplicate test efforts and use historical data to detect and resolve potential issues during development.
- Automation and Industrial Internet of Things (IoT)
The webinar dives into why legacy direct attached storage (DAS) won’t work for today’s log analytics needs and why Pure helps you optimize your log analytics.
Adopting an Adaptable Framework (and Frame of Mind)
No matter what your log data looks like, getting it right requires the right architecture. It’s paramount that you’re able to access scalable amounts of data—in real-time—from new and changing sources.
That means revising log data architectures to keep complexity in check, and optimizing for real-time workloads, VMs, and containers. Infrastructures need to support more concurrency, increasingly disaggregated architectures, and explosive growth. Here’s where leaning on legacy storage that’s better-suited for batched, sequential workloads can cause problems.
The right infrastructure is critical, but it’s half the battle. Solution architects also need strategies to identify, ask, and answer the right questions. The report provides nine helpful “guideposts” to shape your strategy. For example:
- How long does log data need to be retained? As log data grows, so do the costs for storing and managing it. But compliance and corporate audit rules may dictate how long archival data must sit on ice, so organizations should look for lower-cost ways to store.
- What role will changing business realities have? If log analytics are a goldmine of information about a business, then evolving business needs will change the questions you ask of it. Everything can and should affect infrastructure planning for log analytics.
No matter where you are in your log data strategy, learn why a better data storage solution is key to your success, and how to get it right.
