The wealth of machine log data has business potential that extends far beyond day-to-day IT performance insights. It can deliver improved customer experiences, shed light on the past, present and future, and empower DevOps and security teams. All told, the operational efficiencies, cost reduction capabilities, and competitive advantage log data yields make it a goldmine.

But log data can also be notoriously complex. We’re learning that, for a decades-old practice, it requires thoroughly modern systems to get right.

Pure Storage partnered with O’Reilly to develop “Understanding Log Analytics at Scale: Log Data, Analytics, & Management,” a comprehensive report on the current log data landscape, including use cases and opportunities, pitfalls to avoid, and toolsets to know. Read on to learn more.

The Log Data Conundrum

Log data analysis is not new to IT organizations, which means it’s often placed squarely in the territory of legacy systems and practices. But while it’s not new in practice, the types of data it’s asked to handle is. And that’s exactly what’s tripping things up.

“The overwhelming majority of log data offers little value… [and] conventional data analytics are ill-suited to handle the variety, velocity, and volume of log data.”
—Matt Gillespie, author, Understanding Log Analytics at Scale

Modern log data is diverse, disparate, and unwieldy. We’re not only talking telemetry and data points from every machine in your organization, every second of the day. Today’s log data includes data from streaming sources, cloud environments, containers, and virtual machines. Feeding this into legacy systems, then asking modern questions from it, can exacerbate the problem.

The Many Opportunities for Modern Log Data

It’s easy to see how log analytics at scale can be an uphill battle—but there’s inherent value in it you can’t ignore.

With the right tools to ingest, clean, and analyze log data and the right end-to-end strategy, organizations can derive serious value. Use cases for log data applications are expanding all the time, including

  • Cybersecurity: Help security operations improve forensic analysis, predict threats, detect anomalies, and more.
  • DevOps and software development: Empower teams to reduce duplicate test efforts and use historical data to detect and resolve potential issues during development.
  • Automation and Industrial Internet of Things (IoT)

The report dives deep into all the ways modern log can deliver value to organizations of all types and sizes—including yours.

Adopting an Adaptable Framework (and Frame of Mind)

No matter what your log data looks like, getting it right requires the right architecture. It’s paramount that you’re able to access scalable amounts of data—in real-time—from new and changing sources.

“Hyper-distributed applications are demanding heretofore unheard-of levels of concurrency. The architecture as a whole must be adaptable to the ongoing—and accelerating—growth of data stores.”
—Understanding Log Analytics at Scale

That means revising log data architectures to keep complexity in check, and optimizing for real-time workloads, VMs, and containers. Infrastructures need to support more concurrency, increasingly disaggregated architectures, and explosive growth. Here’s where leaning on legacy storage that’s better-suited for batched, sequential workloads can cause problems.

The right infrastructure is critical, but it’s half the battle. Solution architects also need strategies to identify, ask, and answer the right questions. The report provides nine helpful “guideposts” to shape your strategy. For example:

  • How long does log data need to be retained? As log data grows, so do the costs for storing and managing it. But compliance and corporate audit rules may dictate how long archival data must sit on ice, so organizations should look for lower-cost ways to store.
  • What role will changing business realities have? If log analytics are a goldmine of information about a business, then evolving business needs will change the questions you ask of it. Everything can and should affect infrastructure planning for log analytics.

No matter where you are in your log data strategy, learn why a better data storage solution is key to your success, and how to get it right.

Read the O’Reilly report on log analytics.
Learn more about solutions for modern analytics.