Ingest Processor for Observability Cloud

As infrastructure and application environments become exceedingly complex, the volume of data they generate continues to grow significantly. This increase in data volume and variety makes it challenging to gain actionable insights and can impact problem identification and troubleshooting efficiencies. Additionally, the cost of storing and accessing this data can skyrocket. Many data sources, particularly logs and events, provide critical visibility into system operations. However, in most cases, only a few details from these extensive logs are actually needed for effective monitoring and alerting.

Common Challenges:

• Increasing complexity of infrastructure and application environments.
• Significant growth in data volume generated by these environments.
• Challenges in gaining actionable insights from large volumes of data.
• High costs associated with storing and accessing extensive data.
• Logs and events provide critical visibility but often contain only a few essential details.

To address these challenges, Splunk Ingest Processor provides a powerful new feature: the ability to convert log events into metrics. Metrics are more efficient to store and process, allowing for faster identification of issues, thereby reducing Mean Time to Detection (MTTD). When retaining the original log or event is necessary, they can be stored in cheaper storage solutions such as S3, reducing the overall cost of data ingestion and computation required for searching them.

Solution:

• Convert log events into metrics where possible.
• Retain original logs or events in cheaper storage solutions if needed.
• Utilize federated search for accessing and analyzing retained logs.

Outcomes:

• Metrics are more efficient to store and process.
• Faster identification of problems, reducing Mean Time to Detection (MTTD).
• Lower overall data ingestion and computation costs.
• Enhanced monitoring efficiency and resource optimization.
• Maintain high visibility into system operations with reduced operational costs.

In this workshop you’ll have the opportunity to get hands on with Ingest Processor and Splunk Observability Cloud to see how it can be used to address the challenges outlined above.