Skip to content

Getting Started ↵

About Splunk for SAP LogServ

v0.0.5 release: AI Assistant LLM functionality intentionally disabled pending review

The v0.0.5 release ships with the AI Assistant’s LLM-driven path disabled at compile time pending internal review of the OWASP LLM Top 10 controls. The predefined-prompt path + Splunk MCP Server integration + 20 dashboards + Environment Topology view + audit log are all fully active. The free-form chat input is disabled, the model picker is hidden, and the Provider Credentials Settings tab is hidden. See Templates-only Build for the build mechanism. The LLM-driven path will be re-enabled in a future release once review concludes.

Introduction

SAP offers its customers ECS (fka RISE) with SAP S/4HANA Cloud Private Edition. This is an IaaS model (on a very basic level) from SAP’s vendor perspective, where SAP hosts customers’ SAP S/4HANA and other SAP systems in the customer’s choice of public cloud providers (AWS, Microsoft Azure, GCP, etc.), in accounts owned and managed by SAP itself. SAP LogServ provides logs from all SAP systems and layers (OS, database, etc.), and the logs can be integrated to be available to the customer’s security information and event management (SIEM) solution.

Splunk for SAP LogServ provides multiple mechanisms to access the logs from LogServ, ingest them into Splunk, and map the various log types to Splunk sourcetypes — plus a React-based UI App with 20 dashboards, a graph-based Environment Topology view, and an LLM-aware AI Assistant that lets analysts run pre-canned investigations without leaving Splunk.

Two Packages

The solution is delivered as two separately installable packages:

Package App ID Install On Role
Data TA splunk_ta_sap_logserv Deployment Server, Heavy Forwarders, Indexer (or single instance) Data collection, sourcetype routing, index-time filtering, DS automation, ships the indexes.conf for sap_logserv_logs + _ai_assistant_audit
LogServ App splunk_app_sap_logserv Search Head only (or single instance) Dashboards, AI Assistant, Environment Topology view, search-time extractions

The Data TA ingests log data, routes it to the right sourcetype, and ships the indexes.conf for both the SAP data index (sap_logserv_logs) and the AI Assistant audit index (_ai_assistant_audit) — Splunk auto-creates them when the Data TA loads on an indexer, no separate Index App required. The LogServ App provides the analytics layer the user interacts with. Both index names are configurable via search macros (sap_logserv_idx_macro, sap_logserv_audit_idx_macro).

For details on which package goes where, see Architecture.

Key Features

  • Index-time filtering — control which log types are indexed and drop stale data, all configured through Splunk Web with zero license cost for filtered events.
  • Deployment Server automation — automatically stages filter configurations for distribution to Heavy Forwarders with a one-click deploy button.
  • 20 React-based dashboards — organized as one top-level Environment Health landing page plus four purpose-driven navigation groups: Applications (5 dashboards: ABAP runtime, work-process performance, HANA audit + trace), Integration (5 dashboards: SAP Services, Router, Cloud Connector, Web Dispatcher, Web/API Performance), Security (3 dashboards: Network Perimeter, Cross-Stack Authentication, Change & Configuration), and Platform (6 dashboards including the multi-tab Host Details view + multi-tab Data Pipeline Overview). Every dashboard ships with cross-dashboard drill-downs (time range preserved), per-dashboard auto-refresh picker, and built-in Download PNG export.
  • Environment Topology — interactive graph view of SAP systems, integration partners, and endpoints; built on @xyflow/react with self-derived IP→SID inventory, saved layouts (KV Store), and Live mode auto-refresh.
  • AI Assistant — Splunk-aware chat panel with two paths: a predefined-prompt browser (48 saved searches across sap_basis / security / operations packs, dispatched via the Splunk MCP Server with no LLM call), and a free-form prompt path that adds vendor-LLM synthesis on top. Three privacy tiers (Tier 0 air-gapped, Tier 1 default, Tier 2 admin opt-in). The privacy invariant is type-system-enforced: no event data from your Splunk instance is ever transmitted to any AI vendor.
  • OWASP LLM Top 10 (2025) compliance — every item in the top-10 has a matching control: prompt-injection sanitization, type-bounded data redaction, supply-chain SBOM, audit hash-chain, per-user rate limit, USD spend cap, SPL static-analysis guard, jailbreak pattern detection, PII redaction, and a tamper-evident audit log forwarder over HEC.
  • Templates-only build variant — a deployable variant of the LogServ App that disables the LLM-driven flow at compile time. The MCP path + canned prompts stay fully active so the solution can be demonstrated end-to-end without enabling any LLM provider.
  • Search-time field extractions — ~176 search-time directives (EXTRACT, EVAL, FIELDALIAS) across 31 SAP-specific sourcetypes.
Version 0.0.5.0-beta
Supported vendor products SAP LogServ for SAP ECS in Amazon Web Services (AWS)
Splunk platform versions 9.4.3 and later
CIM 5.1.1 and later

Environment Health


Host Details

Architecture

Two-Package Model

The SAP LogServ solution for Splunk is delivered as two separately installable packages:

Package App ID Purpose
Data TA splunk_ta_sap_logserv Data collection, index-time filtering, deployment server automation, configuration UI, ships the indexes.conf for sap_logserv_logs (SAP data) and _ai_assistant_audit (AI Assistant audit log)
LogServ App splunk_app_sap_logserv Dashboards, AI Assistant, Environment Topology view, search-time field extractions, macros

The Data TA handles everything that happens at index time: ingesting data from S3, routing events to the correct sourcetype, applying index-time filters, and defining the two indexes the solution writes to. It includes Python scripts, REST handlers, a configuration UI built with Splunk’s UCC framework, and default/indexes.conf so Splunk auto-creates sap_logserv_logs and _ai_assistant_audit on first install. Both index names are macro-configurable via sap_logserv_idx_macro (SAP data) and sap_logserv_audit_idx_macro (audit log); customers who rename either index update the matching macro definition. The _ai_assistant_audit index is required for the AI Assistant’s audit log to function — without it, audit events have no destination index.

The LogServ App handles everything that happens at search time: field extractions, field aliases, computed fields, and the dashboards you use to visualize and analyze the data. It contains no Python code and no data collection components.

Why two packages?

The split follows Splunk best practices for distributed deployments. The Data TA runs at the ingest tier (Heavy Forwarders for distributed deployments, plus the Indexer where the bundled indexes.conf provisions storage) and the LogServ App runs on the Search Head where users interact with dashboards. Keeping these tiers separate means search-only logic (field extractions, dashboards, the AI Assistant chat panel) doesn’t bloat the forwarders, and ingest logic (sourcetype routing, REST handlers, UCC config UI) doesn’t leak into the Search Head.

Install Matrix

Where you install each package depends on your Splunk topology:

Topology Data TA LogServ App
Single instance Same instance Same instance
DS + HFs + on-prem SH Deployment Server + each HF + Indexer Search Head only
DS + HFs + Splunk Cloud Deployment Server + each HF (Splunk Cloud admin handles the indexer tier — Data TA installed there provides the index defs) Splunk Cloud SH only

Important

  • The Data TA is never installed directly on Heavy Forwarders when using a Deployment Server – the DS distributes it automatically.
  • The LogServ App is never installed on Heavy Forwarders or the Deployment Server.
  • On Splunk Cloud, the customer’s Splunk Cloud admin handles the indexer tier separately. The Data TA installed on that indexer provides the bundled index definitions.
  • For single-instance deployments, both packages are installed on the same instance and Splunk merges their configurations at runtime.

Data Flow

The diagram below shows how SAP LogServ data flows from the SAP ECS environment into Splunk:

  SAP ECS Environment
        |
        v
  SAP LogServ S3 Bucket (SAP-managed)
        |
        v  (S3 event notifications via SQS)
  Customer's AWS Account
  +-----------------------------------------------+
  |  Destination S3 Bucket  -->  SQS Queue        |
  |         (S3 events trigger SQS messages)      |
  +-----------------------------------------------+
        |
        v  (Splunk AWS Add-on reads from SQS)
  Splunk Heavy Forwarders
  +-----------------------------------------------+
  |  1. Ingest NDJSON from S3 via SQS             |
  |  2. Route to sourcetype (TRANSFORMS)          |
  |  3. Apply index-time filters (nullQueue)      |
  |  4. Forward to indexer                        |
  +-----------------------------------------------+
        |
        v
  Splunk Indexer
  +-----------------------------------------------+
  |  Stores events in sap_logserv_logs index      |
  +-----------------------------------------------+
        |
        v
  Splunk Search Head
  +-----------------------------------------------+
  |  LogServ App: dashboards + field extractions  |
  +-----------------------------------------------+

Index-Time Filtering

The Data TA provides built-in index-time filtering that lets you control which log types are indexed. Filtering happens on the Heavy Forwarders using TRANSFORMS-based queue routing:

  • Include patterns – Only ingest log types that match the pattern (e.g., linux/* to include only Linux logs)
  • Exclude patterns – Drop specific log types (e.g., linux/cron to exclude cron logs)
  • Days in past – Drop data older than a specified number of days based on the S3 object path date

Filtered events are routed to nullQueue and never consume Splunk license. Filter settings are configured on the Deployment Server and pushed to Heavy Forwarders automatically.

See Configuring Filters for detailed setup instructions.

Sourcetype Routing

All SAP LogServ data arrives in a single generic format. The Data TA examines each event’s metadata during index-time parsing and routes it to the appropriate Splunk sourcetype. Routing is defined in transforms.conf using regex-based sourcetype assignment.

Two routing strategies are used:

  • Source-path matching – For log types with unique source field values (e.g., /var/log/messages for Linux syslog, /var/log/squid/access.log for Squid proxy)
  • Classification field matching – For SAP application logs that share similar source paths, routing matches the clz_dir and clz_subdir fields in the NDJSON envelope. When the same clz_subdir value appears under multiple clz_dir paths (e.g., audit exists under both abap/ and scc/), compound lookahead regexes match both fields simultaneously to avoid collisions.

For the complete list of supported log types and their sourcetype mappings, see Supported Log Types.

v0.0.5.0 React App Architecture

The LogServ App was rewritten as a React application in v0.0.5.0. The Data TA architecture is unchanged from v0.0.4.x — only the UI tier changed.

Stack:

  • Build pipeline: webpack-based bundle build atop @splunk/webpack-configs. Each Splunk app page resolves to a single React bundle.
  • UI primitives: @splunk/react-ui (forms, tables, modals), @splunk/visualizations (charts), @xyflow/react (Topology graph), styled-components for theming.
  • State management: React context for cross-cutting concerns (AI Assistant, time range, refresh ticker); useState / useReducer for component-local state. No Redux.
  • Data fetching: custom useSearch hook wraps @splunk/search-job for SPL dispatches; results expose rows / loading / error to consuming components.
  • Routing: React Router 7 with HashRouter so URLs survive Splunk Web’s app-namespace routing; query strings carry time-range hydration (?earliest=...&latest=...).

Build-time feature flags:

The build supports compile-time variants. The first such flag is TEMPLATES_ONLY: when set, the resulting bundle has the AI Assistant’s free-form / LLM-driven flow disabled at compile time, NOT runtime — there is no runtime setting that could re-enable it. See the Templates-only Build page for the user-facing implications.

Static-asset cache busting:

Splunk Web caches the React bundle’s asset URL by an integer [install] build field in app.conf. Every meaningful code change bumps this number; without bumping, browsers serve stale bytes after deploy. The 3-part SemVer in [id] version is independent of the build number and changes only on user-facing version bumps.

AI Assistant Architecture

The AI Assistant is a chat-style panel embedded in the React UI App that lets analysts run pre-canned investigations + free-form prompts against their Splunk data. It has two distinct paths and a strong privacy invariant.

Privacy invariant — type-system-enforced, not policy-enforced:

User question  -->  AI vendor  -->  AI picks tools  -->  MCP server  -->  Splunk
                       |                                       |
                       |  <----  Hidden<MCPToolResult>  <------+
                       |
                       v  (sanitize chokepoint: count + timing only — Tier 1)
                       |  (or aggregated metadata — Tier 2)
                       v
                  AI synthesizes narrative reply

Tool results from the Splunk MCP Server are typed Hidden<MCPToolResult> in TypeScript. The compiler refuses to put a Hidden<T> value into the outbound vendor payload — the only way to convert it is via sanitize(hidden, summarizer), which forces the caller to provide a non-data summary. The summarizer is gated by the active privacy tier:

  • Tier 0 (Ollama, future) — air-gapped local LLM; no vendor traffic at all.
  • Tier 1 (default) — summary is count + execution_time only. AI sees no values.
  • Tier 2 (admin opt-in) — summary adds aggregated metadata (per-column cardinality, top-N values + counts for categorical, min/max/avg/sum for numeric, time range when _time is present). Still no raw rows.

Two paths:

  • Predefined prompts (no LLM call): the user opens the prompt browser and clicks one of the 48 cataloged prompts. The orchestrator dispatches the saved search via the Splunk MCP Server, renders the result tile in the right pane, and appends a static interpretation + suggested-next-steps card. No vendor LLM is invoked. This is the path used in the templates-only build.
  • Free-form prompts (LLM-driven): the user types a natural-language question. The orchestrator sends the system primer + user message + tool definitions to the active vendor (Anthropic / OpenAI / Azure OpenAI / AWS Bedrock). The vendor picks tools, the orchestrator dispatches them in parallel via MCP, the vendor sees only the privacy-tier summary, and the vendor synthesizes a narrative response.

Audit log:

Every AI Assistant action — both paths — produces audit events into a dedicated _ai_assistant_audit index. Categories include local_only (canned-prompt dispatches), vendor_tier1 / vendor_tier2 (LLM calls with token counts + USD cost estimate), security_blocked_spl, user_prompt_jailbreak_flag, session_tool_cap_hit, daily_spend_cap_hit, audit_forwarder_failure, plus three legal-acknowledgement categories. The Audit Log tab in Settings provides an in-app browser; an optional HEC forwarder can stream events to a separate Splunk / SIEM destination for tamper-evidence.

Splunk MCP Server prerequisite:

The AI Assistant requires Splunk MCP Server (Splunkbase App 7931) v1.1.0 or later, installed on the same search head as the LogServ App. Cookie auth from the same Splunk Web session works by default on HTTP-only Splunk; an optional bearer token can be configured for OAuth-strict environments. See Splunk MCP Setup for end-to-end configuration, troubleshooting, and the auto-mint roadmap.

Environment Topology Architecture

The Environment Topology view (also accessible via URL slug integration-topology for backward compatibility) is a graph-based visualization of SAP systems, integration partners, and endpoints across the SAP landscape. It is implemented as a React component on top of @xyflow/react.

Data sources:

The view assembles its node + edge inventory from a union of six SPL searches against the existing sourcetypes (no new ingest required):

  • sap:abap:gateway — RFC peer/local IPs (P=<peer> / L=<local> fields)
  • sap:abap:icm — ICM peer/local IPs
  • sap:hana:tracelogs — HANA host + tenant SID extracted from the source path (/usr/sap/<HANA_SID>/HDB<inst>/<host>/trace/DB_<TENANT_SID>/)
  • sap:saprouter — peer hostname extracted from the parens after host <ip>/<service> (<resolved.host>)
  • linux_messages_syslog with osquery cpu_brand events — host inventory (CPU, RAM, OS, region, AZ, instance ID)
  • Default Splunk host field — fallback for hosts not surfaced in the above

Self-derived IP→SID inventory:

The “which IP belongs to which SID” mapping is derived from a multi-source union SPL with a mvcount(sids)=1 filter — a host whose multiple sourcetypes all agree on a single SAP SID is unambiguously attributed; otherwise it’s surfaced as “unknown”. Resolution depends on what your data exposes: unique hostname/IP appearances across multiple SAP sourcetypes (HANA tracelogs, ABAP gateway L=, ICM peer fields, saprouter peer hostnames) attribute cleanly, while shared NAT IPs and external partners typically remain unknown. Additional inventory sources can be added by appending another union arm — the inventory framework is extensible per-customer without new ingest.

Saved layouts:

User-arranged graph layouts are persisted via Splunk KV Store collection logserv_topology_layouts. The schema (currently v4) carries node positions, panel state, viewport zoom + pan, enabled integration types, selected node, active right-sidebar tab, and snap mode. Layouts are per-user-named (an admin can save a default layout that other users see; users can save their own variants). Schema migration is in-memory: v1 / v2 / v3 records still load.

Live mode auto-refresh:

The toolbar’s Live mode toggle drives a 30-second auto-refresh that re-runs all SPL queries on the topology view; saved layouts are preserved across ticks. Coexists with the per-dashboard auto-refresh picker — currently both contribute additively to the refresh nonce; consolidation is planned for a future release.

Release Notes

Version 0.0.5.0-beta (latest)

AI Assistant LLM functionality intentionally disabled pending review

The v0.0.5 release ships with the AI Assistant’s LLM-driven path disabled at compile time pending internal review of the OWASP LLM Top 10 controls. Every customer running v0.0.5 runs the templates-only build variant — there is no separate “regular” build published in this release. What’s still active: the predefined-prompt path (48 canned prompts via the Splunk MCP Server), tool tiles in the right pane, drill-down chips, audit log, all 20 dashboards + Environment Topology view, per-dashboard auto-refresh picker, Download PNG. What’s disabled: free-form chat input, the model picker, the Power Mode toggle, the Provider Credentials Settings tab, and all vendor (Anthropic / OpenAI / Azure / Bedrock) traffic. The LLM-driven path will be re-enabled in a future release once review concludes — the type-system enforcement, privacy tiers, and OWASP Top 10 hardening are designed and implemented, just gated off via the build flag for now. See the AI Assistant → Templates-only Build docs page and the AI Assistant → OWASP LLM Top 10 Compliance page for the full picture.

Compatibility
Splunk platform versions 9.4.3 and later
CIM 5.1.1 and later
Supported OS for data collection Platform independent
Vendor products SAP LogServ for SAP ECS in Amazon Web Services (AWS)
AI Assistant prerequisite Splunk MCP Server (Splunkbase App 7931) v1.1.0 or later, on the search head where the LogServ App is installed

Major architecture change

The LogServ App is fully rewritten as a React-based application. Dashboard Studio v2 is no longer used for any of the 20 dashboards. The app now ships as a single React bundle built on @splunk/react-ui, @splunk/visualizations, and @xyflow/react. The Data TA architecture is unchanged from v0.0.4.x — only the UI App tier has been rewritten.

Implications for upgraders:

  • Search-time field extractions are unchanged — your existing custom searches, alerts, and reports against sap_logserv_logs continue to work without modification.
  • Dashboard URLs have changed — old DS v2 deep links (/app/splunk_app_sap_logserv/<view>?form.global_time...) are replaced with React Router hash routes (/app/splunk_app_sap_logserv/home#/<route>?earliest=...&latest=...). Time-range query params are preserved.
  • Splunk 9.4.3+ remains the minimum version. No new floor.
  • No data re-ingest required — the upgrade is UI-only.

New features

  1. AI Assistant — Splunk-aware chat panel with two paths:

    • Predefined prompts (no LLM call): browse 48 saved searches across three packs (sap_basis 13, security 14, operations 13) plus a context-aware Dashboard Focused tab that auto-filters to prompts relevant to the current dashboard. Each prompt dispatches via the Splunk MCP Server and renders a tile in the right pane with a static interpretation + suggested-next-steps card. No vendor LLM is involved in this path.
    • Free-form prompts (LLM-driven): the same MCP tool path is available to one of four AI providers (Anthropic, OpenAI, Azure OpenAI, AWS Bedrock); the LLM picks tools, the orchestrator dispatches, and the LLM synthesizes a narrative response. Critical privacy invariant — enforced by the TypeScript type system at build time, not by policy: no event data from your Splunk instance is ever transmitted to any AI vendor. The compiler refuses to put any tool-result value into the outbound payload — there is no runtime check, no flag to flip.

  2. Three privacy tiers for the free-form path, admin-selectable in Settings:

    • Tier 0 — Ollama-based local-only (future release).
    • Tier 1 (default) — cloud LLM as SPL generator. Tool result summary fed back is only count + timing. The AI sees no row data and no aggregates.
    • Tier 2 (admin opt-in) — adds aggregated metadata: cardinality, per-column top-N values + counts, min/max/avg/sum (numeric), and time range. Still no raw rows.

  3. Environment Topology — graph-based view of SAP systems, integration partners, and endpoints. Built on @xyflow/react with a force-directed initial layout, self-derived IP→SID inventory drawn from multiple SAP sourcetypes (gateway L=, HANA tracelogs, ICM peer fields, saprouter peer hostnames), per-node sidebar tabs (Programs, Calls/Hr, Errors, Hosts), Live mode auto-refresh, and named saved layouts persisted via Splunk KV Store (schema v4 — viewport zoom + pan + enabled-types + selected-node + active-tab + snap-mode).

  4. Drill-down chips — every tool result tile in the AI Assistant’s right pane carries a ↗ Dashboard chip (when a related OOTB dashboard is mapped) and a ↗ Run SPL chip that opens Splunk’s Search app with the dispatched SPL pre-populated and the dispatch’s exact earliest/latest pre-applied. Same chips render alongside [→ saved_search] citations in the chat narrative on the left pane. Dashboards themselves also got drill-downs: ~70 KPIs / charts / tables / table rows across 19 dashboards open contextual cross-cutting searches with current time range preserved.

  5. Per-dashboard auto-refresh picker — every dashboard’s title row now carries a Refresh picker (Never / 30s / 1m / 5m / 15m / 30m / 1hr) with per-user-per-dashboard cadence persisted to a new KV Store collection (logserv_dashboard_refresh). All charts and KPIs re-run on each tick via a shared context nonce.

  6. OWASP LLM Top 10 (2025) compliance — every item has a matching control. Highlights: prompt-injection sanitization with role-marker + jailbreak-pattern filtering; type-bounded data redaction; SBOM (1416 components, CycloneDX 1.4) shipped with every build; tamper-evident audit log with optional HEC forwarder; per-user rate limit (configurable, default 30/hr); USD spend cap; SPL static-analysis guard blocking write/delete/alert operators; PII redaction for email / user(name) / *_ip / mac / account (hostname opt-in); session tool-call cap; jailbreak pattern detection on user input. See OWASP LLM Top 10 Compliance for the full controls list per item.

  7. Templates-only build variant — a deployable variant of the LogServ App that disables the LLM-driven flow at compile time. The MCP path + 48 canned prompts + tool tiles + drill-down chips + audit log all stay fully active so the solution can be demonstrated end-to-end without enabling any LLM provider. UI cues: chat input disabled with explanatory placeholder; Send button disabled; model picker hidden; Power Mode toggle hidden; Provider Credentials Settings tab hidden; cyan info-tone banner explains the build mode. Defense in depth: the LLM dispatch entry point bails immediately with a system notice if reached at runtime.

  8. Power Mode — role-gated ✦ Power toggle in the AI Assistant chat input. Admin assigns a list of Splunk roles (via services/authorization/roles) that may see the toggle; when on, every prompt forces a saved-search dispatch before LLM synthesis (forced-RAG). State persists per-tab in sessionStorage. Audit events tag the toggle state for SOC pivot analysis.

  9. TIME-WINDOW REASONING primer rules — the AI Assistant’s system primer (Tier 1 + Tier 2) now teaches the LLM to: (a) identify the dispatch window before claiming severity, (b) normalize cumulative count to events/hour or events/day before ranking, (c) for any finding ranked [severity:high] or [severity:critical], dispatch ONE additional verify query with earliest=-24h latest=now BEFORE writing the narrative, and (d) state the window precisely in narrative (“X events in the last 24h” vs. “X cumulative over the search’s rolling window”). The result: the AI now self-corrects in one turn instead of needing a follow-up prompt to re-rank cumulative-noise findings.

  10. HostDetails multi-host filter + 3-tab layout — the Host Details dashboard’s host picker is now a Multiselect with filter input + Select-All-Matches semantics. Multi-host scope is reflected in URL (?hosts=h1,h2,h3) with localStorage persistence. SPL builders splice a host IN (...) clause when 2+ hosts are selected. Three tabs: Overview (5 KPIs + charts + Host Inventory + Severity Timeline), Role Activity (7 role-specific panels with hideWhenNoData), Sourcetype Mapping (Sankey of source → sourcetype).

  11. Data Pipeline Overview dashboard-wide host filter — Multiselect + Top-N picker lifted from the chart-level actions slot to the dashboard’s title row. Filter scope expanded from one chart to all 4 KPIs + 4 panels + the Sourcetype Mapping linked graph on the second tab.

  12. Path B sourcetype migration — the legacy [set_srctype_for_syslog] transform has been split into four dedicated routing transforms producing four new sourcetypes: linux:cron, linux:warn, linux:sudolog, linux:slapd. This clears the AppInspect pretrained-sourcetype warning and avoids field-extraction collisions with Splunk_TA_nix’s built-in [syslog] stanza. Existing sourcetype=syslog data ages out per index retention; dashboards OR both old + new during the transition.

  13. Branded LS app icons — orange “LS” mark on a dark rounded-square frame. All three apps (UI App, Data TA, Index App) ship the same icon set at 36×36 + 72×72 in regular + Alt variants.

  14. Splunk-pattern legal acknowledgement — two compile-time legal/liability modals gate the master enabled toggle and the audit-forwarder-disabled save (matching Splunk’s splunk_instrumentation optInVersion framework). User identity, Splunk-stamped IP, timestamp, and a SHA-256 of the disclaimer revision are recorded in the audit log so subsequent acknowledgement reviews can prove which revision was acknowledged.

Enhancements

  1. 20 React-rewritten dashboards plus the new Environment Topology view — every one of the 20 v0.0.4.2 dashboards is a fresh React implementation, and the Environment Topology view is a new graph-based surface unique to v0.0.5. All dashboards use the unified dark theme (#0d1117 page background, #141b2d panel fill, #0877a6 panel outline) and ship the per-dashboard auto-refresh picker.
  2. Saved-Layout schema v4 — the topology view’s saved layouts now persist viewport (zoom + pan), enabled integration types, selected node, active right-sidebar tab, and snap-mode in addition to the v3 node + panel positions. Schema migration is in-memory: v1 / v2 / v3 records still load.
  3. Multiselect + Top-N picker as a reusable title-row pattern — labelless inline cluster matching the visual idiom across HostDetails and Data Pipeline Overview.
  4. AI Assistant prompt browser tab persistence — the last selected pack tab is remembered across modal-open events, persisted per-tab via sessionStorage. Persists only when the user actually picked a prompt, not on casual tab-flipping.
  5. Static guidance card per canned prompt — each predefined prompt’s intent-map entry includes an interpretation paragraph + bulleted nextSteps. Surfaced as a “How to read this result” card after the tool tile lands. Skipped on the AI-driven path (the LLM writes its own commentary). 126 next-step entries split: 64 plain · 57 canned-prompt links · 5 custom-SPL links.
  6. Dashboard Focused prompt browser tab — first-position tab in the prompt browser that filters the 48 prompts down to those mapped to the current dashboard. Auto-hides when no prompts match. Pack-origin chips on each card so users can find the prompt back in its home pack.
  7. Audit Log Settings tab — read-only browser of the _ai_assistant_audit index with time-range / category / user / limit filters; per-row JSON expand. Inline disclaimer covers the tamper-resistance threat model and recommends HEC-forwarder mitigation. 12 audit categories with distinct gradient-fill chip colors.
  8. HEC audit forwarder — admin-configurable forwarding of audit events to a separate Splunk / SIEM / S3-with-Object-Lock destination. Browser-side dual-write at flush time. Failure events captured as a separate audit_forwarder_failure category so disabled / down forwarders are visible in the audit log itself.
  9. Visible<T> brand types — outbound-message types are tagged Visible and unwrap explicitly; the type system refuses to put a Hidden<MCPToolResult> into an outbound vendor payload, mechanically enforcing the privacy boundary.
  10. Dynamic timechart span — every time-series chart’s SPL passes a timechartSpan computed from the current time range so 30-day windows don’t render with 700 data points. Helper at utils/timechartSpan.ts.

Fixed issues

  1. Stale aggregate framing in AI Assistant top-N responses — the LLM previously cited cumulative aggregates (“4,799 failed authentications”) as if they were active rates, leading to misleading “lock the accounts today” recommendations. Build 171’s TIME-WINDOW REASONING primer rules now force a verify query before high-severity claims, and the same cumulative number gets correctly downgraded with explicit “stale long-window aggregate, not an active brute-force” framing.
  2. Splunk risky-command safeguard on nextSteps.spl — two intent-map deep-dive strings used | map maxsearches=1 search="..." which Splunk flags as risky. Rewrote to first-class subsearch syntax. Intent map version bumped v0.0.8 → v0.0.9.
  3. AZ field bleeding into next osquery section — the Host Inventory panel’s zone regex now stops at the #012 osquery section separator ([^,#]+ instead of [^,]+), so AZ values like ap-south-1a no longer carry trailing data from adjacent fields.
  4. MCP cookie auth on same-session HTTP-only Splunk — verified empirically that the Splunk MCP Server v1.1.0 accepts cookie auth from the same Splunk Web session that’s serving the React app, so the default mcp_server_url works on HTTP-only Splunk with no bearer token configured. The optional bearer token layers on top via Authorization: Bearer and is invalidated on 401 with one retry.
  5. Splunk services/authorization/roles endpoint — Multiselect for the Power Users field reads roles from the correct path; services/authentication/roles (a common typo) silently 404s and produces a stuck “Loading roles…” UI.
  6. Splunk Web static-asset cache busting — every meaningful code change bumps [install] build in app.conf so browsers don’t serve stale bytes after deploy.
  7. Webpack style-loader requirement — adding import '@xyflow/react/dist/style.css' exposed a latent webpack-config gap where CSS was being compiled but never reaching the DOM. Both style-loader AND css-loader are now in the webpack rules.

Restyled (visual conventions)

  1. 20 React dashboards with the unified dark-theme card style: #0d1117 page, #141b2d panel fill, #0877a6 panel outline, 3 px rounded corners, 5 px inset, 12 px panel gaps. Equivalent to the v0.0.4.2 DS v2 look but rebuilt natively in styled-components.
  2. Severity dots — chat findings render with a colored dot (yellow → orange → red → dark-red for low → medium → high → critical) using a radial gradient so they read as glossy beads matching the donut-chart palette aesthetic.
  3. Win11-style 8-dot loading spinner — replaces the prior cyan-arc indicator in AI Assistant streaming + tool-executing states. CSS-only via single keyframe + per-dot --angle variable + staggered animation-delay. Reused in the Topology canvas loading overlay (extracted to a shared Spinner component).
  4. Cyan-light dotted-underline citation links — the AI’s [→ saved_search] citations render as clickable scroll-to-tile spans; sibling ↗ Dashboard and ↗ Run SPL chips use the same visual idiom.
  5. Compact Multiselect with Select-All-Matches — HostDetails + Data Pipeline Overview both use @splunk/react-ui/Multiselect with compact + filter + selectAllAppearance="checkbox" so typing into the filter narrows the dropdown and the Select All control auto-renames to “Select all matches”.
  6. Glossy severity-dot gradients — radial-gradient(circle at 35% 30%, ...) so dots read as 3D beads not flat circles.
  7. Audit-log filter chips with per-category gradients — 12 categories each get a distinct 3-stop linear gradient with mid-stop ~35–45% luminance for white-text readability, dim-when-unchecked via layered translucent-black wash so the text stays readable.

Known issues

  1. Tier 0 (Ollama, air-gapped) is not yet shipped. Tier 0 currently returns “not yet implemented” if selected. Planned for a future release.
  2. Auto-mint MCP token roadmap is not yet shipped. Bearer tokens for the Splunk MCP Server still require manual paste in Settings → Splunk MCP. Planned for a future release.
  3. Splunk MCP TA gate is bypassed because the dependent TA isn’t yet identified on Splunkbase. The gate will be restored when a real TA is published.
  4. hideWhenNoData panel-disappearance behavior continues to apply on HostDetails Role Activity tab. Expected behavior, but empty tabs can feel sparse on hosts that only forward a single sourcetype.

Third-party software attributions

The v0.0.5.0 LogServ App ships with THIRD-PARTY-NOTICES.md at the root of the installed app directory (and at the root of the GitHub release source tree). The file lists all 1235 unique top-level npm packages bundled with the React app — names, versions, declared licenses, repository URLs, and full LICENSE / NOTICE / COPYING text where available. License posture: 1012 MIT, 64 ISC, 57 Apache-2.0, 46 BSD-3-Clause, 22 BSD-2-Clause, 11 @splunk/* (covered as a Splunk Extension under §1.C of Splunk General Terms), plus a long tail of permissive licenses. No GPL / AGPL / LGPL components. See Third-Party Software for the full license-distribution summary and refresh policy.

A CycloneDX 1.4 SBOM (SBOM.json) is also regenerated on every build and shipped inside the package alongside THIRD-PARTY-NOTICES.md.

Version 0.0.4.2-beta

Compatibility
Splunk platform versions 9.4.3 and later
CIM 5.1.1 and later
Supported OS for data collection Platform independent
Vendor products SAP LogServ for SAP ECS in Amazon Web Services (AWS)

New features

  1. 3 new SAP service sourcetypes — sap:sapstartsrv (SAP Start Service / Host Control Agent with auth and SSL/TLS negotiation fields), sap:saphostexec (SAP Host Agent execution logs), and sap:saprouter (SAP Router connection and trace logs). These cover the sap/sapstartsrv, sap/saphostexec, and sap/saprouter log types in the LogServ S3 bucket.
  2. 28 total sourcetype routing transforms with @logserv_filter annotations for index-time filter support.
  3. ~176 total search-time directives (EXTRACT, EVAL, FIELDALIAS) across all SAP-specific sourcetypes in the LogServ App.
  4. 15 new dashboards in the LogServ App, bringing the total to 20. Dashboards are organized into 4 purpose-driven navigation groups plus a top-level Environment Health landing page (reorganized from the previous 3-group structure so that the top menu is balanced and each group answers a specific class of question):
    • Top-level — Environment Health (default landing)
    • Applications (5 dashboards) — the SAP app runtime itself: ABAP Network & Security, ABAP Operations, Work Process Performance (new), HANA Audit, HANA Trace
    • Integration (5 dashboards) — how SAP talks to other systems: SAP Services, SAP Router (new), Cloud Connector, Web Dispatcher, Web and API Performance (new)
    • Security (3 dashboards) — cross-source synthesis for security posture and compliance: Network Perimeter (new), Cross-Stack Authentication (new), Change & Configuration Activity (new)
    • Platform (6 dashboards) — infrastructure, ingest, and forensics: Data Pipeline Overview, DNS Analytics, Linux System & Security, Windows Events, Proxy Analytics, Host Details
  5. 6 new dashboards from Phase 2 (added after the original 14):
    • Cross-Stack Authentication — unified authentication failure analysis across SAP, HANA, and Windows layers, with per-layer KPIs, source-IP aggregation, and per-layer recent-failure tables
    • SAP Router — SAP Router connection activity, error analysis, and network boundary monitoring (separated out of SAP Services to give router its own investigation surface)
    • Work Process Performance — SAP ABAP work process utilization with all 13 SAP-standard dev_w* trace category codes, dispatcher health, and function-level activity
    • Web and API Performance — Web Dispatcher four-stage request timing (dt1-dt4), response-time percentiles, TLS version and cipher-suite distributions, and a cross-source panel overlaying HTTP error rate against Cloud Connector auth failure rate
    • Network Perimeter — unified network-boundary view synthesizing firewall drops, proxy outbound traffic, and DNS resolution into one dashboard; includes firewall-drops-by-protocol, top outbound domains with byte volumes, and a cross-source Suspicious Activity Indicator table ranking internal hosts by combined beaconing-DNS + denied-proxy signal score
    • Change & Configuration Activity — compliance-focused audit trail unifying HANA user/role/privilege/DDL changes, Windows account and group modifications (15 canonical security EventCodes), and Linux sudo + useradd/usermod/userdel/passwd activity; includes source-prefixed operator identities, a category taxonomy, and two compliance-focused “Recent” tables (Privileged Changes + After-Hours Changes)
  6. Environment Health dashboard — Cross-cutting operations view with 6 KPIs, 6 category-specific error trend charts (ABAP, HANA, Security, Web/Network, Cloud Connector, OS/Infra), critical events table, host error matrix, and performance panels. Every panel drills down to the relevant detailed dashboard. Now set as the default landing page.
  7. Tabbed Data Pipeline Overview — Two tabs: “Overview” (5 KPIs + 14-column Sourcetype Summary table + Host Latest Activity) and “Linked Graph” (full-width source-to-sourcetype link graph). The Sourcetype Summary table includes Status (Fresh/Stale/Very Stale), Trend sparkline, % of Total, Avg/Day, Volume, App Errors, Hosts, Sources, Events (1h), First Seen, Last Seen, and Lag columns.
  8. HANA Audit security panels — Three new panels surface the rich sap:hana:audit field set: Risk-Tiered Event Timeline (stacked column by risk_level), After-Hours / Weekend Admin Activity (table filtered to admin users outside business hours), and High-Risk Events (table of is_critical=true events with SQL Statement column).
  9. KPI sparklines — ~75 KPIs across all 20 dashboards display an inline daily-trend sparkline below the headline number, using a single-source timechart + eventstats pattern. Five flavors: count-based, distinct-count, rate, formatted-volume, and per-day re-detection. One acknowledged exception: the Linux “Top Drop Source” KPI is a string value (<IP> (<count>)) with no sparkline.
  10. Click-through drilldowns — Most KPIs, table rows, and chart points open a filtered Splunk search. Clickable table cells carry a cyan accent so the drilldown affordance is visible.
  11. KPI single values added to DNS Analytics (Total Queries, Unique Clients, Beaconing Domains), HANA Audit (Total Events, Failed Operations, Active Users), and Web Dispatcher (Total Requests, Error Rate, Avg Response Time). Access Denied Events KPI added to Cloud Connector; Top Drop Source KPI added to Linux.
  12. Enhanced DNS Analytics — Top Queried Domains, Top Clients by Domain Diversity (DGA detection), Query Type Distribution, and Top DNS Resolvers table.
  13. Enhanced HANA Audit — Top Users by Activity, Activity by Hour of Day (after-hours detection), and Client IP Analysis.
  14. Enhanced Web Dispatcher — Request Volume Over Time, Top URIs by Request Count, and Recent Errors (4xx/5xx).
  15. Host Details — 3-tab expansion — The Host Details dashboard is now organized into three tabs. Overview shows a 5-KPI row (Total Events, Data Volume, Active Sourcetypes, Errors/Criticals, Auth Failures), the Host Event Count by Sourcetype timeline, a cross-source Severity Timeline, Host Inventory (CPU/RAM/EC2/OS/region from osquery), Recent Authentication Events + Recent Errors & Criticals cross-source tables, Top Sources, Activity by Hour of Day, and Data Freshness per sourcetype. Role Activity contains seven role-specific panels (HANA Audit Activity, ABAP Work Process Mix, Web Dispatcher Traffic by Status, SAP Router Peers, Windows Event Codes, Sudo Commands, DNS Top Queries) that auto-hide via hideWhenNoData when the selected host has no data for that component. Sourcetype Mapping houses the full-width Sankey chart that was previously inline.
  16. Cross-dashboard navigation — Every dashboard includes a Navigate to Dashboard dropdown with Go button that preserves the selected time range when switching between dashboards.
  17. In-dashboard documentation link (“More Info” button) — A cyan More Info button in the top-right of every dashboard’s toolbar row opens the corresponding online-documentation section in a new browser tab. The link targets the dashboard’s section within the appropriate category page (.../dashboards/applications/#<dashboard-slug>, etc.) so users can jump from a live dashboard to its narrative documentation in one click. For multi-tab dashboards (Data Pipeline Overview, Host Details) the button appears on every tab.

Enhancements (per-dashboard restructures)

  1. SAP Services — Removed the 4 router-related panels (now on the SAP Router dashboard); featured SSL Authentication Failure Sources full-width; replaced Event Volume by Service line chart with a stacked column chart showing Normal vs Errors per service.
  2. Windows Events — Removed Security Event Actions chart and Top Users table (now on Cross-Stack Authentication); featured Top Event Codes full-width with 7 enriched columns (Event Code, Description, Source log, Severity, Events, Hosts, Last Seen).
  3. Proxy Analytics — Replaced single-slice donuts (Content Types → Cache Action Distribution column; HTTP Methods → Top Clients by Domain Diversity bar). Added new bottom row: Top URL Domains by Bytes Out + Bandwidth Over Time by Domain.
  4. DNS Analytics — Replaced the uninterpretable Volume & Packet Size scatter plot with a Top DNS Resolvers table; restructured row 2 to 4 panels including Query Type Distribution donut moved up to pair with the trend chart.
  5. ABAP Operations — Work Process Categories donut widened to 836 px with bottom legend showing all 13 friendly category names (uses the shared wp_category_name props.conf EVAL).
  6. Cloud Connector — Renamed “Error Rate” → “HTTP Error Rate” to clarify scope; added Access Denied Events KPI (4th KPI in row).
  7. Linux System & Security — Added Top Drop Source KPI surfacing the highest single-source firewall drop count in <IP> (<count>) format (4th KPI in row).

Fixed issues

  1. DNS Analytics beaconing panels now use correct message_type="Query" case (was "QUERY").
  2. Web Dispatcher data source had hardcoded Unix timestamps; replaced with $global_time.earliest$/$global_time.latest$ tokens.
  3. Work Process Categories labels — The Work Process Categories panel on the ABAP Operations dashboard now displays meaningful names for all 13 standard SAP dev_w* trace component codes (A = ABAP Processor, B = Database Interface, C = Communication, D = Dispatcher, M = Memory Management, N = Network (NI), O = Enqueue / Lock, Q = RFC Queue, R = Roll Area, S = SQL / Statistics, T = Task Handler, X = RFC / CPIC, Y = Dynpro / Screen). Previously only A/B/C/M were mapped and the rest appeared as single-letter codes. The same wp_category_name mapping is now also used on the Work Process Performance dashboard.
  4. KPI panel alignment — KPI single-value widgets on all three-KPI dashboards are evenly spaced with the rightmost KPI outline aligned to the right edge of panels below.
  5. Right-edge symmetry — All rows on width=1920 dashboards now cap at R=1910; width=1600 dashboards cap at R=1590. Symmetric 10 px padding on both sides.
  6. HANA Trace component noise filter — Top Components, Component by Severity, and Source File Hotspots panels now filter out parsing artifacts (“INFO”, “of”, “service:”) that previously diluted real component data.
  7. Ingest Errors KPI on Data Pipeline Overview — Refined to exclude ExecProcessor noise (which wraps all scheduled-script output as ERROR-level regardless of the script’s actual log level). Filters to real Python ERRORs only.
  8. SSL Authentication Failure Sources panel (SAP Services) — Replaced the previous Sapstartsrv SSL/TLS Events panel which showed empty columns due to mismatched field extractions. Now aggregates by source IP using fields that actually exist in the data (auth_user, remote_ip, remote_port) and provides row drilldown to the full event set per IP.
  9. Empty-safe KPI pattern — All count-based and dc-based KPIs now display 0 instead of ### when the underlying search returns no events (uses a synthetic-row appendpipe wrap).

Restyled (visual conventions)

  1. Dashboard “card” style — All 20 dashboards use a unified visual treatment: #0d1117 page background, #141b2d panel fill, #0877a6 panel outline, rounded corners, 5 px inset between rect frame and inner viz.
  2. KPI typography standardized — majorFontSize: 36, explicit labelColor: #7b8ea8, labelFontSize: 13, semantic majorColor (#dc4e41 red for errors, white for neutral counts, orange for warnings, teal for positive signals). The Linux Top Drop Source KPI uses majorFontSize: 28 as an acknowledged exception for its long-text string display.
  3. Standard red consolidated — All red color variants (#e86c5d, #af575a, #ff3b30, #ff2d55) normalized to single hex #dc4e41.
  4. Tables — Hardcoded header background (#1e2a3d), zebra-stripe alternating rows (#0d1520 / transparent), fixed header. Cyan accent on clickable cells indicates drilldown affordance.
  5. 12 px panel gaps — Exact horizontal and vertical spacing between every panel border across all dashboards.
  6. Dashboard descriptions — Every dashboard now displays a 1-line description below its title.
  7. “Go >” navigation button — Standardized: 120×25 px at top-left of every dashboard with 10 px padding above and below; majorFontSize 16.
  8. “More Info” documentation button — Standardized: 140×25 px at top-right of every dashboard, aligned with the right edge of the canvas (10 px padding from the right; x = canvas_width − 150). Same cyan fill #0877a6, white text, majorFontSize 16 as the Go button. Opens the dashboard’s online-documentation section in a new browser tab via drilldown.customUrl with newTab: true.

Known issues

  1. The dashboards in the LogServ App use Dashboard Studio v2 format and require Splunk 9.4.3 or later.
  2. Several Host Details panels (Host Inventory, Recent Authentication Events, Recent Errors & Criticals, and all seven Role Activity panels) use hideWhenNoData and will disappear for hosts that lack the underlying sourcetype data. For example, a Windows host without osquery data will not show the Host Inventory panel; an ABAP-only host will not show the HANA Audit Activity panel on the Role Activity tab. This is the dashboard adapting to the selected host’s role — not a bug — but empty tabs can feel sparse on hosts that only forward a single sourcetype.

Third-party software attributions

Version 0.0.4.1-beta

Compatibility
Splunk platform versions 9.4.3 and later
CIM 5.1.1 and later
Supported OS for data collection Platform independent
Vendor products SAP LogServ for SAP ECS in Amazon Web Services (AWS)

New features

  1. 12 new SAP application sourcetypes — 9 SAP ABAP types (sap:abap:audit, sap:abap:dispatcher, sap:abap:enqueueserver, sap:abap:event, sap:abap:gateway, sap:abap:icm, sap:abap:messageserver, sap:abap:sapstartsrv, sap:abap:workprocess), 1 HANA trace type (sap:hana:tracelogs), and 2 SAP Cloud Connector types (sap:scc:audit, sap:scc:http_access).
  2. Compound lookahead routing — New routing pattern for log types where the same clz_subdir value appears under multiple clz_dir paths (e.g., audit exists under both abap/ and scc/). Uses regex lookahead to match both fields simultaneously.
  3. Search-time SID/instance extraction — ABAP and HANA sourcetypes extract sap_sid and sap_instance from the source metadata field using EXTRACT ... in source directives in the LogServ App.
  4. ~128 total search-time directives across all SAP-specific sourcetypes.

Fixed issues

Known issues

  1. The dashboards in the LogServ App use Dashboard Studio v2 format and require Splunk 9.4.3 or later.

Third-party software attributions

Version 0.0.3-beta

Compatibility
Splunk platform versions 9.4.3 and later
CIM 5.1.1 and later
Supported OS for data collection Platform independent
Vendor products SAP LogServ for SAP ECS in Amazon Web Services (AWS)

New features

  1. Two-package architecture — The solution is now split into two packages: the Data TA (splunk_ta_sap_logserv) for data collection and index-time processing, and the LogServ App (splunk_app_sap_logserv) for dashboards and search-time field extractions. See Architecture for details.
  2. Built-in index-time filtering — Configure include/exclude patterns and time-based filters directly through the Splunk Web UI. Filtered events never consume Splunk license. See Configuring Filters.
  3. AWS Lambda-based filtering — New deployment option that filters S3 event notifications in AWS before they reach Splunk, reducing S3 GET request costs and SQS message volume. Available via the AWS Remote S3 Filter Setup Walkthrough or the Connect to Filter Migration. Can be used alongside or independently of the native TA filtering.
  4. Deployment Server automation — When installed on a Deployment Server, the TA automatically stages filter configurations for distribution to Heavy Forwarders and provides a one-click “Deploy to Forwarders” button.
  5. Upgrade notifications — A system message banner alerts administrators when a TA upgrade adds support for new log types that are not covered by existing include filter patterns.
  6. Daily time filter refresh — A built-in scripted input automatically refreshes the time-based filter cutoff once per day to maintain accuracy of the rolling time window.
  7. SAP HANA Audit field extractions — 14 EXTRACT, 11 EVAL, and 16 FIELDALIAS directives for the sap:hana:audit sourcetype.
  8. SAP Web Dispatcher field extractions — 18 EXTRACT, 3 EVAL, and 6 FIELDALIAS directives for the sap:webdispatcher:access sourcetype.

Fixed issues

  1. Dashboards moved from Data TA to dedicated LogServ App package for proper distributed deployment support.

Known issues

  1. The dashboards in the LogServ App use Dashboard Studio v2 format and require Splunk 9.4.3 or later.

Third-party software attributions

Version 0.0.2-beta

Compatibility
Splunk platform versions 9.4.x, 10.0.x
CIM 5.1.1 and later
Supported OS for data collection Platform independent
Vendor products SAP LogServ for SAP ECS in Amazon Web Services (AWS)

New features

Fixed issues

  1. Drilldown on overview dashboard to host details dashboard had the wrong application name and displayed an error when clicking on the host name.
  2. Renamed the ‘logserv_web_dispatcher_access.xml’ dashboard to ‘logserv_web_dispatcher.xml’.
  3. Renamed the ‘sap_rise_host_details.xml’ dashboard to ‘logserv_host_details.xml’.
  4. Updated the ‘~/ui/nav/default.xml’ with updated dashboard names.

Known issues

  1. The dashboards included in this TA are Dashboard Studio dashboards that may not work with Splunk versions prior to 9.4.

Third-party software attributions

Version 0.0.1-beta

Compatibility
Splunk platform versions 9.4.x, 10.0.x
CIM 5.1.1 and later
Supported OS for data collection Platform independent
Vendor products SAP LogServ for SAP ECS in Amazon Web Services (AWS)

New features

Fixed issues

Known issues

  1. Drilldown on overview dashboard to host details dashboard has the wrong application name and displays an error when clicking on the host name.

  2. The dashboards included in this TA are Dashboard Studio dashboards that may not work with Splunk versions prior to 9.4.

Third-party software attributions

Third-Party Software Licenses

Splunk for SAP LogServ bundles open-source software from the npm ecosystem (the React-based LogServ UI App is a webpack-built single-page application). Each component carries its own license, and per those licenses we preserve attribution and bundle the full license text alongside the App.

Where to find the full notices

The complete list of bundled third-party packages — names, versions, declared licenses, and full license text — is delivered in two places:

  • Inside the App tarball — splunk_app_sap_logserv/THIRD-PARTY-NOTICES.md (the file is at the root of the installed app directory after extraction)
  • At the source-tree root — the same file lives at the root of the GitHub release source tarball

The file is generated deterministically from the resolved node_modules/ tree at build time, so its content matches exactly what shipped with the App version you have installed.

License-distribution summary (v0.0.5.0)

The v0.0.5.0 build includes 1235 unique top-level packages under node_modules/. The license breakdown:

License (SPDX as declared) Count Inclusion obligation
MIT 1012 Copyright + license text (preserved in THIRD-PARTY-NOTICES.md)
ISC 64 Copyright + license text
Apache-2.0 57 License text + NOTICE if present
BSD-3-Clause 46 Copyright + license text
BSD-2-Clause 22 Copyright + license text
SEE LICENSE IN LICENSE.md (Splunk libs) 11 Covered as a Splunk Extension under §1.C of Splunk General Terms
BlueOak-1.0.0 4 License text
CC0-1.0 / 0BSD / MIT-0 7 Public-domain or near-public-domain (no obligation)
BSD-3-Clause AND Apache-2.0 1 License text
MIT AND Zlib 1 License text
MIT OR Apache-2.0 1 Either license; we attribute MIT
MIT OR CC0-1.0 1 Either license; we attribute MIT
MPL-2.0 OR Apache-2.0 1 We do not modify the file; either license is satisfied
MIT OR SEE LICENSE IN FEEL-FREE.md 1 MIT chosen
MPL-2.0 (axe-core) 1 License text only (we do not modify the source)
Python-2.0 1 License text
CC-BY-4.0 / CC-BY-3.0 2 Attribution preserved
(no declared license field) 2 See note below

Notes

  • No GPL/AGPL/LGPL components are included in the App. The license posture is fully compatible with commercial redistribution under the standard Splunkbase model.
  • Two packages without a declared license field — @mapbox/jsonlint-lines-primitives@2.0.2 and uuid-v4@0.1.0 — are present as transitive dependencies. Both are de-facto open-source and have been on the public npm registry for years; reviewers preparing legal attestations should consult the upstream repositories listed in the per-component entries of THIRD-PARTY-NOTICES.md for any final disposition.
  • @splunk/* packages (11 total, declared as SEE LICENSE IN LICENSE.md) are covered as a Splunk Extension under §1.C of the Splunk General Terms — the standard mechanism for redistributing Splunk UI libraries inside a Splunkbase-distributed app.

Refresh policy

The THIRD-PARTY-NOTICES.md file is regenerated from the resolved node_modules/ tree at every release. v0.0.5.0 used a one-off generation pass; v0.1.1 onward auto-refreshes the file as part of the standard yarn build pipeline so the notices and the bundled JavaScript artifacts always agree on what was shipped.

Supported Log Types

Overview

SAP ECS environment logs are not a singular data source but a collection of OS-specific, SAP environment, database, and other application logs.

Due to the nature of this solution, the SAP LogServ packages are not standalone integrations. To take full advantage of their capabilities (like CIM mapping), you need to install additional TAs as specified in the Prerequisites.

For a streamlined data ingestion process, all selected logs are ingested under one sourcetype: sap_logserv_logs. They are then assigned to a final sourcetype during parsing/indexing on the Heavy Forwarder (or Indexer in single-instance mode), based on the source field.

All events are in NDJSON format with metadata (like _time, host, source, etc.) and the _raw field containing the event contents. To limit index size, only the _raw field is ingested from each event – metadata fields are either mapped to Splunk’s native metadata fields or dropped. However clz_dir and clz_subdir fields are preserved to maintain backtracking capabilities. These fields correspond to the directory tree of the original data in S3.

LogServ S3 Path Structure

The log files in the SAP LogServ S3 bucket follow this path pattern:

logserv/<clz_dir>/<clz_subdir>/<YYYY>/<MM>/<DD>/<filename>.json.gz

For example:

logserv/linux/messages/2025/09/15/messages-abc123.json.gz
logserv/hana/hanaaudit/2025/10/01/hana-xyz789.json.gz
logserv/dns/binddns/2025/11/20/dns-def456.json.gz

The clz_dir/clz_subdir values are used by the index-time filter to match include/exclude patterns. See Configuring Filters for details.

Sourcetype Mapping

SAP HANA Audit (LogServ App)

The LogServ App provides search-time field extractions for SAP HANA audit events, including 14 EXTRACT, 11 EVAL, and 16 FIELDALIAS directives.

Source field value Sourcetype assigned Filter path
hana audit log sap:hana:audit hana/hanaaudit

SAP Web Dispatcher (LogServ App)

The LogServ App provides search-time field extractions for SAP Web Dispatcher access logs, including 18 EXTRACT, 3 EVAL, and 6 FIELDALIAS directives.

Source field value Sourcetype assigned Filter path
web dispatcher access log sap:webdispatcher:access webdispatcher/accesslog

SAP ABAP Application Logs (LogServ App)

The LogServ App provides search-time field extractions for 9 SAP ABAP application log types. Each sourcetype includes sap_sid and sap_instance extraction from the source metadata field, plus type-specific field extractions.

Source field value Sourcetype assigned Filter path
ABAP security audit log sap:abap:audit abap/audit
ABAP dispatcher log sap:abap:dispatcher abap/dispatcher
ABAP enqueue server log sap:abap:enqueueserver abap/enqueueserver
ABAP event log sap:abap:event abap/event
ABAP gateway log sap:abap:gateway abap/gateway
ABAP ICM (Internet Communication Manager) log sap:abap:icm abap/icm
ABAP message server log sap:abap:messageserver abap/messageserver
ABAP sapstartsrv log sap:abap:sapstartsrv abap/sapstartsrv
ABAP work process log sap:abap:workprocess abap/workprocess

SAP HANA Trace Logs (LogServ App)

The LogServ App provides search-time field extractions for HANA trace logs, including SID/instance extraction from the source path.

Source field value Sourcetype assigned Filter path
HANA trace log sap:hana:tracelogs hana/tracelogs

SAP Cloud Connector (LogServ App)

The LogServ App provides search-time field extractions for SAP Cloud Connector audit and HTTP access logs.

Source field value Sourcetype assigned Filter path
SCC audit log (CSV format) sap:scc:audit scc/audit
SCC HTTP access log sap:scc:http_access scc/tracelogs

SAP Service Logs (LogServ App)

The LogServ App provides search-time field extractions for SAP host-level service logs. These are infrastructure services that run at the host control level (/usr/sap/hostctrl/) rather than within a specific SAP instance.

Source field value Sourcetype assigned Filter path
SAP Start Service log (auth, SSL/TLS) sap:sapstartsrv sap/sapstartsrv
SAP Host Agent execution log sap:saphostexec sap/saphostexec
SAP Router connection and trace log sap:saprouter sap/saprouter

SAP Service Log Details

  • sap:sapstartsrv includes fields for OS authentication failures, SSL/TLS negotiation errors (protocol version, cipher suite, peer addresses), and webmethod invocation failures.
  • sap:saprouter covers both .log files (CONNECT/DISCONNECT/INVAL DATA events with connection IDs and host addresses) and .trc files (NiBuf/NiI error traces with peer/local addresses and return codes) as a single sourcetype.

Splunk Add-on for Unix and Linux

Source field value Sourcetype assigned Filter path
/lastlog lastlog linux/linux_secure
/var/log/cron syslog linux/cron
/var/log/firewall linux_secure linux/linux_secure
/var/log/kernel linux_secure linux/linux_secure
/var/log/localmessages linux_messages_syslog linux/localmessages
/var/log/messages linux_messages_syslog linux/messages
/var/log/pacemaker(.log) syslog linux/warn
/var/log/slapd.log syslog linux/slapd
/var/log/sssd(.log) linux_secure linux/linux_secure
/var/log/sudolog syslog linux/sudolog
/var/log/warn syslog linux/warn
/who who linux/linux_secure

Splunk Add-on for Microsoft Windows

Source field value Sourcetype assigned Filter path
WinEventLog:Application XmlWinEventLog windows/WinEventLog:Application
WinEventLog:(*.)Operational XmlWinEventLog windows/WinEventLog:Powershell
WinEventLog:Security XmlWinEventLog windows/WinEventLog:Security
WinEventLog:System XmlWinEventLog windows/WinEventLog:System

Splunk Add-on for Squid Proxy

Source field value Sourcetype assigned Filter path
/var/log/squid/access.log squid:access proxy/squid
/var/log/squid/cache.log squid:access proxy/squid
/var/log/squid/store.log squid:access proxy/squid

Splunk Add-on for ISC BIND

Source field value Sourcetype assigned Filter path
/var/lib/named/log/named/default.log isc:bind:query dns/binddns
/var/lib/named/log/named/general.log isc:bind:network dns/binddns
/var/lib/named/log/named/lame-servers.log isc:bind:lameserver dns/binddns
/var/lib/named/log/named/network.log isc:bind:network dns/binddns
/var/lib/named/log/named/notify.log isc:bind:transfer dns/binddns
/var/lib/named/log/named/queries.log isc:bind:query dns/binddns
/var/lib/named/log/named/resolver.log isc:bind:network dns/binddns
/var/lib/named/log/named/update.log isc:bind:transfer dns/binddns
/var/lib/named/log/named/xfer-out.log isc:bind:transfer dns/binddns

Filter Path Column

The Filter path column shows the clz_dir/clz_subdir value used in the index-time filter include/exclude patterns. A -- means the log type does not currently have a filter-eligible transform. See Configuring Filters for details.

Prerequisites Overview

Splunk for SAP LogServ ships as two separately installable packages with distinct prerequisites. Use this page to plan what you need before starting installation.

The Two Packages

Package App ID Role Where it installs
LogServ Data TA splunk_ta_sap_logserv Data collection from S3, index-time filtering, deployment server automation, ships the indexes.conf for the two indexes the solution writes to Single instance, OR Deployment Server + each Heavy Forwarder + Indexer
LogServ UI App splunk_app_sap_logserv Dashboards, AI Assistant chat panel, search-time field extractions Single instance, OR the Search Head only

For single-instance deployments, both packages install on the same instance. For distributed topologies, each package goes to its own tier — never SCP a Data TA file to the Search Head, and never SCP a UI App file to a Heavy Forwarder. The Data TA carries indexes.conf defining both sap_logserv_logs (SAP data) and _ai_assistant_audit (AI Assistant audit log); Splunk auto-creates them on indexer install, no separate Index App required.

Both indexes are macro-configurable — sap_logserv_idx_macro (SAP data, default index="sap_logserv_logs") and sap_logserv_audit_idx_macro (audit log, default index="_ai_assistant_audit"). Customers who rename either index update the matching macro definition under Settings → Advanced search → Search macros. See Renaming an index for the full procedure (READ + WRITE paths).

Common Prerequisites (both packages)

  • Splunk Enterprise 9.4.3 or later, or Splunk Cloud Platform

Splunk 9.4.3 is the minimum because the LogServ App’s React stack (@splunk/react-ui, @splunk/visualizations, @xyflow/react) requires the React component versions shipped with that release.

Package-Specific Prerequisites

Each package has its own additional prerequisites — install Splunkbase add-ons appropriate to that tier.

Decision Tree

Your situation What you need
Single Splunk instance running the full LogServ solution Both prerequisite sets — Data TA + App
Distributed Splunk with on-prem Search Head Data TA prereqs on DS + each HF + the indexer; App prereqs on the SH
Distributed Splunk with Splunk Cloud Search Head Data TA prereqs on DS + each HF; App prereqs on the Splunk Cloud SH; Splunk Cloud admin handles the indexer tier (Data TA installed there provides the index defs)
Splunk Cloud Search Head only (no on-prem ingest tier) App prereqs only — your Splunk Cloud admin handles the data tier and the indexer tier separately

Next Steps

Quick Install Reference

A single matrix mapping every Splunkbase add-on, prerequisite, and LogServ component to the tier(s) where each gets installed. Use this as a pre-install checklist; for full install steps see the per-package pages linked from each row.

Package Matrix

Single-instance Splunk

For a single-instance Splunk deployment (one host playing every role), install every required app on that one host. The matrix below is for distributed topologies — column headings refer to specific tiers. SH = Search Head, IDX = Indexer, HFs = Heavy Forwarders, DS = Deployment Server.

App Splunkbase Required? SH IDX HFs DS
LogServ Data TA (splunk_ta_sap_logserv) this repo required — ✓ (indexes.conf) ✓ (via DS) ✓ (filter UI)
LogServ App (splunk_app_sap_logserv) this repo required ✓ — — —
Splunk Add-on for Unix and Linux 833 required (CIM) ✓ ✓ ✓ —
Splunk Add-on for Microsoft Windows 742 required (CIM) ✓ ✓ ✓ —
Splunk Add-on for Squid Proxy 2965 required (CIM) ✓ ✓ ✓ —
Splunk Add-on for ISC BIND 2876 required (CIM) ✓ ✓ ✓ —
Splunk Add-on for AWS 1876 required if SAP ECS in AWS — — ✓ (S3 inputs) —
Splunk MCP Server 7931 v1.1.0+ required for AI Assistant ✓ — — —
Splunk AI Assistant 200 recommended companion to 7931 ✓ — — —

Notes

  • Indexer rationale. The Data TA goes on the indexer because it bundles indexes.conf. See Why does the Data TA need to go on the Indexer? for the trade-off + opt-out path.
  • CIM add-ons (Unix/Linux, Windows, Squid, ISC BIND). Install on every tier where the Data TA installs so sourcetype definitions resolve consistently. Splunkbase’s AppInspect rules require these as declared dependencies.
  • AWS Add-on (1876). Only needed when SAP ECS data lives in AWS S3. The TA owns the SQS-based S3 inputs that pull data from the dest bucket; the LogServ Data TA then sourcetype-routes events as they’re parsed on HFs. The actual index = sap_logserv_logs setting that sends events to the right place lives in this TA’s S3 input config — not in the LogServ Data TA.
  • MCP Server (7931). Required for the AI Assistant’s predefined-prompt path even when the LLM-driven path is disabled. Without it, the AI Assistant chat panel can’t dispatch saved searches.
  • Splunk AI Assistant (200). The LogServ App uses only the core splunk_run_saved_search and splunk_run_query MCP tools (which work standalone against 7931), but App 200 follows Splunk’s documented co-install pattern and unlocks saia_*-prefixed MCP tools that may be used in future LogServ releases.

Per-Topology Checklists

Single Splunk instance

Install all required + recommended apps on the same instance. Splunk auto-creates both indexes (sap_logserv_logs + _ai_assistant_audit) when the Data TA loads on first start.

Distributed (DS + HFs + on-prem SH+IDX)

Tier Install
Search Head LogServ App, MCP Server (7931), Splunk AI Assistant (200), CIM add-ons (Unix/Linux, Windows, Squid, ISC BIND)
Indexer LogServ Data TA (provides indexes.conf for both indexes), CIM add-ons
Deployment Server LogServ Data TA (manages filter UI + pushes Data TA to HFs), CIM add-ons
Heavy Forwarders Receive LogServ Data TA via the DS automatically. Install the AWS add-on (1876) directly + CIM add-ons.

Distributed (DS + HFs + Splunk Cloud SH)

Tier Install
Splunk Cloud Search Head LogServ App, MCP Server (7931), Splunk AI Assistant (200), CIM add-ons
Splunk Cloud Indexer tier Splunk Cloud admin handles. Either (a) install the LogServ Data TA there to use the bundled index defs, OR (b) the Cloud admin manually creates sap_logserv_logs and _ai_assistant_audit via the Splunk Cloud UI — see Why does the Data TA need to go on the Indexer?.
Deployment Server LogServ Data TA, CIM add-ons
Heavy Forwarders Receive LogServ Data TA via the DS. Install AWS add-on (1876) directly + CIM add-ons.

Next Steps

Ended: Getting Started

LogServ Data TA ↵

Data TA Prerequisites

This page covers the prerequisites for the LogServ Data TA (splunk_ta_sap_logserv) — the data-collection + index-time-filtering side. For the LogServ App’s prerequisites (the AI Assistant’s MCP Server dependency), see LogServ App Prerequisites.

Splunk Platform Requirements

  • Splunk Enterprise 9.4.3 or later, or Splunk Cloud Platform

Required Splunk Add-ons

The SAP LogServ packages depend on several additional Splunk Technical Add-ons for sourcetype definitions and CIM mapping. Install these from Splunkbase before proceeding:

SAP ECS running in Amazon Web Services (AWS)

If you have SAP ECS running in Amazon Web Services (AWS) you need to install this additional Splunk Technical Add-on as well.

Additional configuration instructions for the Splunk Add-on for Amazon Web Services (AWS) are provided in the Setup walkthroughs after the prerequisite steps have been completed so just install for now.

Next Steps

Next steps:

  1. Install the Data TA
  2. Install the LogServ App

Installing the Data TA

This page covers installing the Data TA (splunk_ta_sap_logserv). For the LogServ App installation, see Installing the LogServ App.

High Level Steps

Below are the high level steps for installing the Data TA. Follow them in order.

Steps 4 and 5 are alternative paths — complete the one that matches your Splunk environment.

  1. Create a default events index
  2. Download the Data TA
  3. Identify where to install the Data TA based on your topology
  4. Install the Data TA in Splunk Cloud (if applicable)
  5. Install the Data TA in Splunk Enterprise (if applicable)


1. Indexes (auto-created on install)

The Data TA ships with default/indexes.conf defining two indexes that Splunk auto-creates the first time the Data TA loads on an indexer:

Index Purpose Default name Macro
SAP data index Receives every event the Data TA forwards (logs ingested from S3 and routed to the appropriate sourcetype) sap_logserv_logs sap_logserv_idx_macro
AI Assistant audit index Receives every audit event the AI Assistant writes — canned-prompt dispatches, free-form vendor calls (when LLM path is enabled), security blocks, privacy-tier elevations, legal acknowledgements _ai_assistant_audit sap_logserv_audit_idx_macro

No customer-side index provisioning is required for the default install.

Note

Both the Data TA and the LogServ App include a macro named sap_logserv_idx_macro that resolves to index="sap_logserv_logs". The LogServ App also includes sap_logserv_audit_idx_macro for the audit index. If you use a different index name, follow the Renaming an index procedure below.

Renaming an Index

Both indexes are macro-configurable, so customers who need different names (e.g., a corporate naming convention) don’t have to fork the app — they update the macros (and, for the audit index, one config field).

To rename the SAP data index
  1. Pick a new name (e.g., splunk_audit_my_org_sap).
  2. Create the index under that new name. Either:
    • Add a custom local/indexes.conf to the Data TA with a stanza for your new name ([my_new_index_name] plus the same homePath / coldPath / thawedPath settings), OR
    • Create the index manually through Splunk Web’s Settings → Indexes → New Index UI. (See Splunk Cloud or Splunk Enterprise docs.)
  3. Update the macro definition. Open Settings → Advanced search → Search macros, find sap_logserv_idx_macro, and edit the definition from index="sap_logserv_logs" to index="my_new_index_name".
  4. Redirect the ingest pipeline to the new index name. The actual index = ... setting that determines where ingested events land lives in the Splunk_TA_aws add-on’s S3 input config (the SQS-based S3 inputs that own the data ingest path), NOT in this Data TA. Update each filtr2_logserv_s3_* input’s index field to point at the new name. See AWS Remote S3 Filter Setup Walkthrough for where these inputs are configured.

The Data TA’s default [sap_logserv_logs] stanza will still create that index unless you remove or override it via your custom local/indexes.conf. If your environment doesn’t need the default, that’s harmless; if it bothers you, override the stanza locally.

To rename the AI Assistant audit index
  1. Pick a new name (consider keeping the underscore prefix — Splunk uses underscore-prefixed names for internal / operational indexes, and excludes them from default-index searches).
  2. Create the index under that name (same options as above — local indexes.conf override, OR Splunk Web Settings UI).
  3. Update the macro definition. Open Settings → Advanced search → Search macros, find sap_logserv_audit_idx_macro, and edit the definition from index="_ai_assistant_audit" to index="<your_new_name>". This controls READS — the in-app Audit Log Viewer + any user-written queries will resolve the macro to your renamed index.
  4. Update the LogServ App config. Open Settings → AI Assistant → General → Audit & Telemetry, set the Audit index name field to your renamed index, and Save Defaults. This controls WRITES — the AuditWriter posts events to the configured index name.

The conf field controls writes; the macro controls reads. They MUST point at the same index, but Splunk doesn’t auto-sync them — keep them aligned manually whenever you rename.


2. Download the Data TA

Download splunk_ta_sap_logserv-0.0.5.0.tar.gz from the GitHub repository.

v0.0.4.3 changes — Path B Linux sourcetype migration

The v0.0.4.3 Data TA replaces the legacy [set_srctype_for_syslog] transform (which routed cron + warn + sudolog + slapd into Splunk’s pretrained syslog sourcetype) with four dedicated transforms producing four new sourcetypes: linux:cron, linux:warn, linux:sudolog, linux:slapd. This clears Splunkbase precert’s pretrained-sourcetype warning and avoids field-extraction collisions with Splunk_TA_nix’s built-in [syslog] stanza. Existing data with sourcetype=syslog ages out per index retention; the LogServ App’s dashboards OR both old + new sourcetypes during the transition.


3. Where to install

Refer to the Architecture page for the full install matrix. In summary:

Your Topology Install the Data TA On
Single instance The single Splunk instance
Deployment Server + HFs + on-prem Indexer Deployment Server (manages filter rules + distributes to HFs); the Indexer (provides default/indexes.conf for sap_logserv_logs + _ai_assistant_audit); HFs receive the TA automatically from the DS
Splunk Cloud Indexer Splunk Cloud admin manages the indexer tier — Data TA installed there provides the bundled index defs

Warning

If you are using a Deployment Server to manage Heavy Forwarders, install the TA on the Deployment Server only. Do not install the TA directly on the Heavy Forwarders — the DS will distribute it automatically when you configure filters. See Configuring Filters for details.

Why does the Data TA need to go on the Indexer?

Splunk only accepts events into an index that’s defined on the indexer that’s receiving them. The Data TA bundles default/indexes.conf (defining sap_logserv_logs + _ai_assistant_audit) — that file is what tells the indexer those indexes exist. Without it, Heavy Forwarders would forward events tagged index=sap_logserv_logs and the indexer would reject them with “no such index” errors.

Honest trade-off: the Data TA is ~9 MiB but only its indexes.conf actually does anything on a pure indexer (no Python is invoked there, no REST handlers fire, no transforms run on already-cooked events). The Python / REST / UCC / transforms code is dead weight on that tier. We chose this setup deliberately as a “two apps to install” simplification over a previous three-app split (Data TA + UI App + a separate Index App that was just indexes.conf + icons).

Opt-out path for customers who want a clean indexer: define both indexes manually on the indexer — either through Splunk Web (Settings → Indexes → New Index for sap_logserv_logs and _ai_assistant_audit with the same homePath/coldPath/thawedPath settings) or via your own etc/system/local/indexes.conf — and then don’t install the Data TA on the indexer. Splunk Cloud customers typically take this path because their Cloud admin team provisions indexes via the Cloud UI rather than installing customer apps on the indexer tier. The Data TA still goes on the DS + HFs as usual.


4. Install in Splunk Cloud

Install the Data TA to your instance of Splunk Cloud using the instructions below:

If you are using separate forwarders in conjunction with Splunk Cloud, be sure to deploy the add-on to your forwarders as well.

Note

The app installation workflow available to you in Splunk Web depends on your Splunk Cloud Platform Experience: Victoria or Classic. To find your Splunk Cloud Platform Experience, in Splunk Web, click Support & Services > About.

Classic Experience

Victoria Experience


5. Install in Splunk Enterprise

Install the Data TA to your instance of Splunk Enterprise:

5.a From the Splunk Web home screen, click the gear icon next to Apps.

5.b Click Install app from file.

5.c Locate the downloaded splunk_ta_sap_logserv-0.0.5.0.tar.gz file and click Upload.

5.d If Splunk Enterprise prompts you to restart, do so.

5.e Verify that the add-on appears in the list of apps and add-ons. You can also find it on the server at $SPLUNK_HOME/etc/apps/splunk_ta_sap_logserv.


6. Macros and Deployment Server

When the Data TA is pushed from a Deployment Server out to Heavy Forwarders, the bundled macros.conf travels with it — but HFs don’t run user searches, so any macro change is operationally inert on that tier. Macros only resolve at search time on the Search Head. The Data TA carries sap_logserv_idx_macro mainly so DS-admin diagnostic searches on the deployment server itself can resolve the macro.

What this means in practice:

Scenario Where the change happens DS involved?
Customer renames the data index SH only — override sap_logserv_idx_macro in the LogServ App’s local/macros.conf (READ), plus update the Splunk_TA_aws S3 input’s index field (WRITE). See Renaming an index above. No
Customer renames the audit index SH only — Settings → AI Assistant → General → Audit index name (WRITE), plus override sap_logserv_audit_idx_macro in the LogServ App’s local/macros.conf (READ). See Renaming an index above. No
Want a custom diagnostic macro present on every HF Edit etc/deployment-apps/splunk_ta_sap_logserv/local/macros.conf on the DS → trigger a scoped DS reload → HFs pull on next polling cycle. Operational effect: none — HFs don’t resolve macros. The macro is present but unused on the HF tier. Yes (cosmetic)

What the DS does push usefully to HFs from this Data TA: filter rules (which sourcetypes to keep, which to drop, days-in-past window, filter enable/disable) — managed via the Configuration tab in Splunk Web on the DS. See Configuring Filters.


Next Steps

  1. Install the LogServ App on your Search Head
  2. Install the Splunk MCP Server on your Search Head if you want to use the AI Assistant
  3. Complete the AWS Setup Walkthrough to configure data collection
  4. Configure index-time filters to control which log types are indexed

Configuring Filters

Overview

The Splunk TA for SAP LogServ provides two complementary approaches to filtering LogServ data. You can use either one independently or combine them for defense-in-depth filtering.

Native TA Index-Time Filtering AWS Lambda-Based Filtering
Where it runs Inside Splunk at index time In AWS Lambda, before data reaches Splunk
Configured via Splunk Web UI (Configuration → Filters) Lambda environment variables or migration config
Works with All deployment scenarios (Connect, Filter, Copy) S3 Filter and Connect-to-Filter migration only
What it filters Raw NDJSON events via TRANSFORMS queue routing S3 event notifications via Lambda function
License impact Filtered events consume zero Splunk license Filtered notifications never reach Splunk
Pattern syntax clz_dir/clz_subdir fnmatch patterns clz_dir/clz_subdir fnmatch patterns (same)
Time filtering Days in the Past (epoch regex, refreshed daily) Days in the Past (Lambda evaluation)
AWS resources needed None Lambda function, local SQS queue, DLQ
Deployment Server support ✅ Auto-distributes to Heavy Forwarders N/A (AWS-side only)
Upgrade notifications ✅ Alerts when new log types aren’t covered ❌ Not available


Which Approach Should I Use?

Choosing a filtering approach

Use Native TA filtering (recommended for most users) if you want the simplest setup — it works with every deployment scenario, requires no AWS resources beyond what you already have, and is managed entirely from Splunk Web. It also provides upgrade notifications when new log types are added to the TA.

Use Lambda-based filtering if you want to reduce the volume of SQS messages that Splunk processes. Because the Lambda filters S3 event notifications before they reach Splunk, unwanted objects are never downloaded from S3 at all. This can reduce S3 GET request costs and Splunk ingestion overhead in very high-volume environments.

Use both together for defense-in-depth. The Lambda filters at the AWS pipeline level, and the native TA filters catch anything that slips through at the Splunk level. Since both use the same clz_dir/clz_subdir pattern syntax, you can keep the configurations aligned.

The remainder of this page covers Native TA Index-Time Filtering in detail. For Lambda-based filtering setup, see the AWS Lambda-Based Filtering section at the bottom of this page or the AWS Remote S3 Filter Setup Walkthrough.


Native TA Index-Time Filtering

What It Does

Starting with version 0.0.3, the TA includes built-in index-time filtering configured entirely through the Splunk Web UI — no manual editing of configuration files is required.

How It Works
  • Include Filters — Only events matching at least one include pattern are eligible for indexing. Everything else is dropped before indexing (zero license cost).
  • Exclude Filters — Events matching any exclude pattern are dropped, even if they also match an include pattern. Excludes override includes.
  • Time Filter (Days in the Past) — Events with a _time older than the configured number of days are dropped. This prevents backfill of very old log data during initial setup or recovery scenarios.

All filtering happens at index time via TRANSFORMS-based queue routing, so filtered events never consume Splunk license.


Deployment Architectures

The TA supports two deployment models. The steps differ slightly depending on which one you use.

Single Instance / Search Head

The TA is installed directly on the Splunk instance that indexes data. Filter changes take effect immediately after clicking Save (no restart required).

Deployment Server with Heavy Forwarders

The TA is installed on the Deployment Server (DS). The DS distributes the TA and its filter configurations to Heavy Forwarders (HFs) that perform the actual data ingestion. This is the recommended architecture for production environments.

In this model:

  • You install and configure the TA on the DS only
  • The DS automatically stages configurations for distribution
  • You use the built-in Deploy to Forwarders button to push changes to HFs
  • HFs receive the TA and filter configs on their next phone-home interval

Note

Splunk Cloud cannot act as a Deployment Server. If you are using Splunk Cloud, you will need a separate on-premises Deployment Server to manage your Heavy Forwarders.


Open the Configuration > Filters Tab

  1. In Splunk Web, open the Splunk TA for SAP LogServ app
  2. Go to Configuration > Filters
Example

image

Set Your Filter Options

Enable Filtering

Check the Enable Filtering checkbox to activate index-time filtering. When disabled, all events are indexed without any filtering.

Include Filters

Comma-separated patterns specifying which log types to include. Patterns use the format clz_dir/clz_subdir with fnmatch-style wildcards:

Pattern Meaning
*/* Include all log types (default)
hana/hanaaudit Include only HANA audit logs
linux/* Include all Linux log types
linux/messages, hana/* Include Linux messages and all HANA logs
dns/* Include all DNS log types

Rules:

  • Each pattern must be in dir/subdir format or a standalone *
  • A standalone * is equivalent to */* (include everything)
  • Valid characters: letters, numbers, *, ?, ., -, :, _
  • This field cannot be empty when filtering is enabled — use */* to include everything
Supported Log Types Reference

See the Supported Log Types section below for a complete list of clz_dir/clz_subdir values you can use in your filter patterns.

Exclude Filters

Comma-separated patterns for log types to exclude. Same pattern format as includes. Events matching any exclude pattern are dropped even if they also match an include pattern.

Example: To include all Linux logs except cron and slapd:

  • Include: linux/*
  • Exclude: linux/cron, linux/slapd

Leave this field empty to exclude nothing.

Days in the Past

Whole number (0–3650) specifying the maximum age of events to index. Events with a _time older than this many days from today are dropped before indexing.

  • Set to 7 to only index events from the last 7 days
  • Set to 0 to disable time-based filtering
  • Default: 7
How the time filter works

The time filter uses a pre-computed epoch-based regex that is refreshed automatically once per day by a built-in scripted input. If the refresh fails to run for a day, the filter becomes slightly more restrictive (one extra day filtered) — the safer failure mode.


Save

Click Save. The TA validates your patterns and generates the necessary configuration files. If there are any validation errors (invalid characters, empty include field, etc.), the save is blocked and an error message is displayed.


Deployment Server: Additional Steps

If you are running on a single instance, you can skip this section — filter changes take effect immediately after Save.


What Happens Automatically on Save

When you save filter settings on a Deployment Server, the TA automatically:

  1. Copies the full TA package to /opt/splunk/etc/deployment-apps/splunk_ta_sap_logserv/
  2. Mirrors the generated filter configurations (transforms.conf, props.conf) to the deployment-apps copy
  3. Creates a server class named SAP_LogServ_HeavyForwarders in a disabled state (if it doesn’t already exist). The server class remains disabled until you configure client targeting in the next step.

After saving, the Filters tab displays:

  • A “⚠ Deployment Server Detected” banner
  • A “⚙ Server Class Setup Required” or “Client Targeting Needed” notice (first time only)
  • A “Deploy to Forwarders” button
Example

image


Configure the Server Class (First Time Only)

The auto-created server class needs client targeting to know which Heavy Forwarders should receive the TA:

  1. Go to Settings → Forwarder Management
  2. Find the SAP_LogServ_HeavyForwarders server class
  3. Click the three-dot menu → Edit agent assignment
  4. Add your Heavy Forwarder IP addresses, hostnames, or use * for all connected forwarders
  5. Save the agent assignment — this also enables the server class
Example

image

Client Targeting

Use IP addresses for client targeting, as Splunk matches against the client’s IP, hostname, DNS name, or GUID — not the instance name.

Return to the Filters tab — the setup notice should be gone (refresh the page in the browser if needed to see the change in the setup notice).

Example

image


Deploy to Forwarders

  1. On the Filters tab, click “Deploy to Forwarders” and confirm
  2. You should see the new updated banner on the filters screen confirming the deployment reload has been initiated
Example

image

  1. Wait for your Heavy Forwarders to phone home (typically 30–60 seconds depending on configuration)
  2. Verify deployment status in Settings → Forwarder Management — HFs should show as “Ok” under the server class, however it may take 5+ minutes to see the status on the forwarders change from Pending to Ok
Example

image


Updating Filters on a Deployment Server

Whenever you change filter settings:

  1. Make your changes on the Filters tab and click Save
  2. Click “Deploy to Forwarders” to distribute the updated configurations
  3. HFs will pick up the changes on their next phone-home interval

Note

Do not install the TA directly on Heavy Forwarders when using a Deployment Server. The DS manages the TA distribution. Installing locally on an HF can cause configuration conflicts.


Verifying Filters

Check Included Data Is Arriving

`sap_logserv_idx_macro` | stats count by clz_dir, clz_subdir

You should see data only for log types matching your include patterns.

Index macro

The sap_logserv_idx_macro macro expands to index="sap_logserv_logs" by default. If you changed the index name in your environment, update the macro definition under Settings → Advanced Search → Search Macros or substitute your index name directly in the search.


Check Excluded Data Is Not Present

If you configured exclude patterns, verify those log types are absent from search results.


Check Time Filtering

Search for events older than your configured cutoff:

`sap_logserv_idx_macro` earliest=-30d latest=-10d

If your Days in Past is less than 10, this search should return no results.


Supported Log Types

Log Type Reference

The table below lists all log types currently supported by the TA. The clz_dir and clz_subdir columns show the values used in filter patterns. Use these values when configuring your include and exclude filters.

clz_dir clz_subdir Splunk Sourcetype
abap audit sap:abap:audit
abap dispatcher sap:abap:dispatcher
abap enqueueserver sap:abap:enqueueserver
abap event sap:abap:event
abap gateway sap:abap:gateway
abap icm sap:abap:icm
abap messageserver sap:abap:messageserver
abap sapstartsrv sap:abap:sapstartsrv
abap workprocess sap:abap:workprocess
dns binddns isc:bind:query, isc:bind:lameserver, isc:bind:network, isc:bind:transfer
hana hanaaudit sap:hana:audit
hana tracelogs sap:hana:tracelogs
linux cron syslog
linux localmessages linux_messages_syslog
linux messages linux_messages_syslog
linux secure linux_secure, lastlog, who
linux slapd syslog
linux sudolog syslog
linux warn syslog
proxy squid squid:access
sap saphostexec sap:saphostexec
sap saprouter sap:saprouter
sap sapstartsrv sap:sapstartsrv
scc audit sap:scc:audit
scc tracelogs sap:scc:http_access
webdispatcher accesslog sap:webdispatcher:access
windows WinEventLog:Application XmlWinEventLog
windows WinEventLog:Powershell XmlWinEventLog
windows WinEventLog:Security XmlWinEventLog
windows WinEventLog:System XmlWinEventLog
Filter pattern examples using this table
  • To include all DNS logs: dns/* or dns/binddns
  • To include all Linux logs except cron: Include linux/*, Exclude linux/cron
  • To include only HANA audit and web dispatcher logs: hana/hanaaudit, webdispatcher/accesslog
  • To include all Windows event logs: windows/*
  • To include all ABAP application logs: abap/*
  • To include specific ABAP types: abap/icm, abap/gateway
  • To include all SAP Cloud Connector logs: scc/*
  • To include all SAP service logs: sap/*


Upgrade Notifications

How Upgrade Notifications Work

When the TA is upgraded to a new version that supports additional log types, a system message banner appears across all Splunk Web pages if your include filter patterns do not cover the newly supported types. This prevents new log types from being silently dropped.

Example

image

To resolve the notification:

  1. Open Configuration → Filters in the TA
  2. Review and update your include patterns to cover the new log types (or use */* to include everything)
  3. Save

The banner clears automatically once all supported types are covered.


Troubleshooting

Filters Not Taking Effect (Single Instance)

  • Verify filtering is enabled: Check the Enable Filtering checkbox
  • Check local/transforms.conf in the app directory for the generated filter stanzas
  • Check _internal for errors:
index=_internal sourcetype=splunkd component=PersistentScript splunk_ta_sap_logserv


Filters Not Reaching Heavy Forwarders

  • Verify the server class exists and has client targeting configured in Settings → Forwarder Management
  • Verify deployment-apps has the filter configs:
cat /opt/splunk/etc/deployment-apps/splunk_ta_sap_logserv/local/transforms.conf
  • Click Deploy to Forwarders and wait for phone-home
  • Check HF configs:
cat /opt/splunk/etc/apps/splunk_ta_sap_logserv/local/transforms.conf


Deployment Server Not Detected

  • The TA detects deployment servers by checking server roles and for connected deployment clients
  • Verify your Heavy Forwarders are configured as deployment clients pointing to the DS
  • Check with:
curl -sk -u admin:<password> \
  "https://localhost:8089/services/deployment/server/clients?output_mode=json&count=1"


Validation Error on Save

  • Include patterns must not be empty when filtering is enabled — use */* to include all
  • Patterns must use dir/subdir format with only valid characters (letters, numbers, *, ?, ., -, :, _)
  • Days in the Past must be a whole number between 0 and 3650


Time Filter Seems Off by a Day

The time filter epoch cutoff is refreshed once per day by a scripted input. After changing the Days in Past value, the cutoff regex is regenerated immediately on Save. If the daily refresh fails, the filter becomes slightly more restrictive (safer failure mode).


AWS Lambda-Based Filtering

Overview

The Lambda-based filtering approach filters S3 event notifications in AWS before they reach Splunk. A Lambda function sits between the cross-account SQS queue (in the SAP ECS account) and a local SQS queue (in your Secondary account). It evaluates each S3 event notification against include/exclude path patterns and a time-based cutoff, forwarding only matching notifications to the local queue that Splunk polls.

When to use Lambda-based filtering
  • You want to reduce the number of S3 GET requests Splunk makes (cost savings in high-volume environments)
  • You want to reduce SQS message volume before it reaches Splunk
  • You want filtering to happen at the AWS pipeline level as an additional layer


How It Works

SAP ECS Account                    Your Secondary Account
+--------------+    +-----------+    +------------+    +-----------+
| S3 Bucket    |--->| SQS Queue |--->|   Lambda   |--->| Local SQS |--->  Splunk HF
| (LogServ)    |    | (cross)   |    |  (filter)  |    |  Queue    |
+--------------+    +-----------+    +------------+    +-----------+
                                       | Drops non-
                                       | matching
                                       v notifications

The Lambda function evaluates the S3 object key in each notification to extract clz_dir and clz_subdir values, then applies the same fnmatch pattern syntax used by the native TA filtering.


Setup Options

There are two ways to deploy Lambda-based filtering:

New Deployment (S3 Filter Setup)

If you are setting up from scratch, follow the AWS Remote S3 Filter Setup Walkthrough. The CloudFormation template creates all required AWS resources (Lambda, local SQS queue, DLQ, IAM permissions) in a single deployment.

Migration from Existing S3 Connect Deployment

If you already have a working S3 Connect deployment and want to add Lambda-based filtering, follow the AWS Remote S3 Connect to Filter Migration. The Python migration script adds the Lambda resources to your existing IAM infrastructure without recreating it.


Lambda Filter Settings

The Lambda function uses three environment variables for its filter configuration:

Variable Description Example
DAYS_IN_THE_PAST Drop notifications for objects older than this many days 7
INCLUDE_FILTERS Comma-separated fnmatch patterns for paths to include linux/*,hana/*
EXCLUDE_FILTERS Comma-separated fnmatch patterns for paths to exclude linux/cron,linux/slapd

These use the same clz_dir/clz_subdir pattern format as the native TA filters. If you are using both approaches, keep the patterns aligned for consistent behavior.


Comparing the Two Approaches Side by Side

Consider a scenario where you want to ingest only hana/* and linux/messages logs from the last 7 days:

Native TA filtering only (simplest)
  • Setup: Configure the Filters tab with include hana/*, linux/messages and days in past 7
  • What happens: All S3 objects are downloaded by Splunk, but unwanted events are dropped at index time via TRANSFORMS before consuming license
  • Pros: No extra AWS resources, managed from Splunk Web, upgrade notifications, works with any deployment scenario
  • Cons: Splunk still downloads and parses all S3 objects before filtering
Lambda filtering only
  • Setup: Configure Lambda with the same include/exclude/days patterns
  • What happens: The Lambda drops S3 event notifications for unwanted objects before Splunk ever sees them. Splunk only downloads matching objects from S3
  • Pros: Reduces S3 GET request costs, lower Splunk ingestion overhead
  • Cons: Requires Lambda + local SQS + DLQ in AWS, no upgrade notifications, filter changes require updating Lambda environment variables
Both together (defense-in-depth)
  • Setup: Same patterns configured in both Lambda and the Filters tab
  • What happens: Lambda filters at the AWS level first, then native TA filtering catches anything that slips through at the Splunk level
  • Pros: Maximum coverage, safety net if either filter is misconfigured
  • Cons: Two places to maintain filter configurations


Next Steps

Return to the Setup Walkthroughs Overview for AWS data pipeline configuration, or see the Developer Reference for technical internals.

Setup Walkthroughs Overview

Introduction

Once the prerequisites and the installation of the Splunk TA for SAP LogServ have been completed, use the provided setup walkthroughs to complete the setup based on the cloud provider where your SAP ECS environment is running in and your preferred deployment scenario.

Note

Starting with version 0.0.3, the TA includes built-in index-time filtering that works with all deployment scenarios below. After completing the AWS setup, see Configuring Filters to control which log types are indexed and drop stale data directly from Splunk Web — no Lambda-based filtering required.

Amazon Web Services (AWS)

All AWS deployment scenarios achieve the end goal of ingesting LogServ logs into Splunk. However, there are some differences in functionality. All AWS deployment scenarios involve two distinct AWS accounts.

All deployment scenarios for AWS require the use of an AWS Secondary account (a different AWS account than the one SAP ECS is running in) due to the requirement from SAP for a cross-account IAM Role to access the AWS SAP ECS account where the LogServ logs reside. See the diagram below for reference.

For brevity and clarity, the AWS account at the top of the diagram will be referred to as the SAP ECS account and the second AWS account on the bottom of the diagram will be referred to as the Secondary account from this point onward.

image

The table below lists the AWS Resources required for each deployment scenario:

AWS Resources AWS Remote S3 Connect Setup AWS Remote S3 Filter Setup AWS Remote S3 Copy Setup
S3 Bucket (SAP ECS account) ✅ Required ✅ Required ✅ Required
SQS Queue (SAP ECS account) ✅ Required ✅ Required ✅ Required
S3 Bucket (Secondary account) ❌ Not Required ❌ Not Required ✅ Required
SQS Queue (Secondary account) ❌ Not Required ✅ Required ✅ Required
SQS Queue DLQ (Secondary account) ❌ Not Required ✅ Required ✅ Required
IAM Policy (Secondary account) ✅ Required ✅ Required ✅ Required
IAM Role (Secondary account) ✅ Required ✅ Required ✅ Required
IAM User (Secondary account) ✅ Required ✅ Required ✅ Required
Lambda Function (Secondary account) ❌ Not Required ✅ Required ✅ Required
Lambda Log Group (Secondary account) ❌ Not Required ✅ Required ✅ Required

If you want to have a secondary copy of the logs from the S3 bucket in the SAP ECS account, the AWS Remote S3 Copy Setup is the recommended approach. Utilizing that approach ensures you have your own copy of all the logs from the S3 bucket in the SAP ECS account, in a second S3 bucket in your secondary AWS account.

The table below lists the different features supported by each deployment scenario:

Feature AWS Remote S3 Connect Setup AWS Remote S3 Filter Setup AWS Remote S3 Copy Setup
Secondary Copy of Logs ❌ Not Supported ❌ Not Supported ✅ Supported
AWS Lambda-based Filtering ❌ Not Supported ✅ Supported ❌ Not yet, coming soon
Native TA Index-Time Filtering ✅ Supported ✅ Supported ✅ Supported
Native TA Filtering vs. AWS Lambda-based Filtering

Starting with version 0.0.3, the TA provides native index-time filtering that works with all deployment scenarios. This filtering happens inside Splunk at index time and is configured entirely through the Splunk Web UI.

The AWS Lambda-based filtering (available in the S3 Filter Setup) filters S3 event notifications before they reach Splunk, reducing the number of SQS messages processed. Both approaches can be used independently or together for defense-in-depth filtering.

See Configuring Filters for details on the native TA filtering.

       AWS Remote S3 Connect Setup Walkthrough

Note

This deployment scenario uses an IAM User with a configured Access Key and a cross-account IAM Role to directly access LogServ resources in the AWS SAP ECS account where the LogServ logs reside without the need to copy logs to a secondary S3 bucket.

image

       AWS Remote S3 Filter Setup Walkthrough

Note

This deployment scenario uses an IAM User with a configured Access Key and a cross-account IAM Role to directly access LogServ resources in the AWS SAP ECS account where the LogServ logs reside without the need to copy logs to a secondary S3 bucket. It also provides the mechanism to filter logs by times stamp and types of logs via parameters on the Lambda function.

  • No secondary S3 Bucket needed
  • Secondary SQS Queue needed
  • Log filtering options supported
  • CloudFormation Template provided

image

       AWS Remote S3 Copy Setup Walkthrough

Note

This deployment scenario uses an IAM User with a configured Access Key and a cross-account IAM Role along with a secondary S3 bucket and SQS queue. Use this deployment scenario if you want a copy of all the LogServ logs in your own S3 Bucket.

  • Greater control of data + retention
  • Requires secondary S3 Bucket
  • Requires secondary SQS Queue
  • CloudFormation Template provided

image

AWS Remote S3 Connect Setup Walkthrough

Introduction

The deployment diagram below shows two different AWS accounts. For brevity and clarity, the AWS account at the top of the diagram will be referred to as the SAP ECS account and the second AWS account in the middle of the diagram will be referred to as the Secondary account from this point onward.

Ensure you have completed all of the steps in the Prerequisites and the Installation of the Splunk TA for SAP LogServ before you continue with the steps in this setup walkthrough.

Your SAP ECS LogServ subscription for AWS (e.g. in your SAP ECS account) should have one S3 Bucket in that account where LogServ will store your LogServ logs. It should also have one SQS Queue in your SAP ECS account that receives notifications when new logs are added to the S3 Bucket in that account.

You will need obtain the AWS ARN for the SQS Queue and the S3 Bucket that resides in your SAP ECS account prior to performing the steps in this setup walkthrough.

Take note of the AWS Region in your SAP ECS account where the S3 Bucket and SQS Queue are located as you will need to deploy the CloudFormation template provided in that same region in your Secondary account.

image

Tip

If you want to ingest historical logs from the S3 bucket in the SAP ECS account, the AWS Remote S3 Copy Setup is the recommended approach. Utilizing that approach provides you the ability to copy historical logs from the S3 bucket in the SAP ECS account to a secondary S3 bucket, so they will be ingested into Splunk, with the flexibility to switch over to this remote S3 connect deployment approach afterward.

If you have no interest in obtaining the historical logs then use this remote S3 connect deployment approach.


High Level Steps

Below are the high level steps for this setup process listed in the order they should be followed.

Please ensure the user you log in with in your AWS Secondary account has the appropriate permissions to perform all the steps outlined below.

  1. Obtain the ARNs for the SQS Queue and the S3 Bucket in your SAP ECS account
  2. Deploy the AWS CloudFormation Template provided for this remote S3 connect deployment approach in your Secondary account
  3. Contact SAP LogServ support and request an update to the access policies for the SQS Queue and S3 Bucket residing in your SAP ECS account
  4. Create an Access Key for the new IAM User that was created with the CloudFormation template in your Secondary account
  5. Configure your AWS Secondary account in the Splunk Add-on for Amazon Web Services (AWS)
  6. Configure the new cross-account IAM Role from your AWS Secondary account in the Splunk Add-on for Amazon Web Services (AWS).
  7. Configure a new SQS-Based S3 Input in the Splunk Add-on for Amazon Web Services (AWS).
  8. Confirm that LogServ logs are being ingested into Splunk


1. Obtain ARNs from SAP ECS Account

Before you can deploy the CloudFormation template in your Secondary account, you need to obtain the AWS ARN for two resources that reside in your SAP ECS account:

  • The S3 Bucket that stores your LogServ logs
  • The SQS Queue that receives notifications when new logs are added to the S3 Bucket

You will also need to note the AWS Region where these resources are located, because the CloudFormation template in the next section must be deployed into that same region in your Secondary account.

What does an ARN look like?

AWS Amazon Resource Names (ARNs) follow a predictable format. Examples:

  • S3 Bucket ARN: arn:aws:s3:::sap-hec-clz-ap-south-1-hec53-xsd
  • SQS Queue ARN: arn:aws:sqs:ap-south-1:121212121212:sap-hec-clz-ap-south-1-hec53-xsd-logserv

How to obtain the ARNs:

1.a If you do not have console access to your SAP ECS account, contact SAP LogServ Support to obtain the ARN for the S3 Bucket, the ARN for the SQS Queue, and the AWS Region where these resources reside.

1.b If you do have console access to your SAP ECS account, obtain the ARNs yourself:

- For the **S3 Bucket**: navigate to the S3 console, find the bucket used for LogServ logs, click on the bucket name, and copy the ARN from the **_Properties_** tab.
- For the **SQS Queue**: navigate to the SQS console, find the queue that receives S3 notifications, click on the queue name, and copy the ARN from the **_Details_** section.

1.c Save both ARNs and the AWS Region in a secure location — you will reference them during the next section (Deploy CloudFormation Template) and again when configuring the SQS-Based S3 Input.


2. Deploy CloudFormation Template

Before you deploy the CloudFormation template, take note of the AWS Region in your SAP ECS account where the S3 Bucket and SQS Queue are located as you will need to deploy the CloudFormation template provided in that same region in your Secondary account.

What does the CloudFormation template do?

Creates the following resources:

  • An IAM User - Default name is splunk_logserv_user
  • An IAM Policy named splunk-logserv-ta-policy
  • An IAM Role named splunk-logserv-ta-role
  • An Inline IAM Policy on the IAM User to assume the IAM Role created above

2.a Navigate to the CloudFormation console (ensure the region you are in matches the region in your SAP ECS account)

2.b Click on the Create stack button and select With new resources (standard)

Example

image

2.c Upload the AWS CloudFormation Template file provided for this remote S3 connect deployment approach, then click the Next button

Example

image

2.d Enter a name for the CloudFormation Stack - splunk-logserv-remote-s3-connect

Example

image

2.e Enter just the name (not the ARN) of the S3 Bucket in your SAP ECS account in the CrossAccountS3Bucket parameter - If your ARN looks like this arn:aws:s3:::sap-hec-clz-ap-south-1-hec53-xsd then just use the name like this ap-hec-clz-ap-south-1-hec53-xsd

Example

image

2.f Enter the complete ARN of the SQS Queue in your SAP ECS account in the CrossAccountSQSQueueArn parameter

Example

image

2.g Choose and enter a name for the IAM User to be created in your Secondary account in the NewIAMUserName parameter, then click on the Next button

Example

image

2.h Scroll down to the bottom of the page and check the I acknowledge that AWS CloudFormation might create IAM resources with custom names checkbox, then click on the Next button

Example

image

2.i Scroll down to the bottom of the page and click on the Submit button

Example

image

2.j Ensure the deployment of the CloudFormation template completes successfully

Example

image


3. Contact SAP LogServ Support

Once the deployment of the CloudFormation templates completes successfully, you will need to provide SAP LogServ Support with the ARN of the IAM Role named splunk-logserv-ta-role that was created by the template.

The ARN for the IAM Role should look like the one below but with your 12-digit AWS account Id of your Secondary account.

arn:aws:iam::secondary-account-id:role/splunk-logserv-ta-role

Example access policies for the SQS Queue and S3 Bucket residing in your SAP ECS account are provided as a reference to provide clarity on the specific permissions that need to be applied by the SAP LogServ Support team.


4. Create Access Key for IAM User

4.a Navigate to the IAM console in your Secondary account and search for the IAM User name you used when deploying the CloudFormation template. Click on the name of the IAM User to see the user details.

Example

image

4.b Click on the Security credentials tab in the middle of the screen. Scroll down and click on the Create access key button.

Example

image

4.c Select the Local code use case, check the Confirmation checkbox and click on the Next button.

Example

image

4.d Enter a description tag value if desired and click on the Create access key button.

Example

image

4.e Copy the values for both the Access key and the Secret access key and save them in a secure place as you will need them in the upcoming steps. Now click on the Done button.

Example

image


5. Configure Secondary Account (AWS Add-on)

Please ensure the user you log in with in your Splunk instance has the appropriate permissions to perform all the steps outlined below.

5.a Login to your Splunk console then find and open the Splunk Add-on for AWS App

Example

image

5.b Click on the Configuration tab, then click on the Account tab, then click on the Add button

Example

image

5.c Choose and enter a descriptive name for the account in the Name field. Enter the Access Key and the Secret Key you created for the IAM User in the respective fields. Leave the Region Category set to Global. Click on the Add button.

Example

image


6. Configure IAM Role (AWS Add-on)

6.a Click on the IAM Role tab to the right of the Account tab, then click on the Add button

Example

image

6.b Choose and enter a descriptive name for the role in the Name field. Enter the IAM Role ARN in the IAM Role ARN field, then click the Add button. The ARN for the IAM Role should look like the one below but with your 12-digit AWS account Id of your Secondary account.

- arn:aws:iam::**_secondary-account-id_**:role/splunk-logserv-ta-role
Example

image


7. Configure SQS-Based S3 Input (AWS Add-on)

7.a Click on the Inputs tab. Click on the Create New Input button. Select the Custom Data Type option at the bottom of the drop-down, then select the SQS-Based S3 (Recommended) option.

Example

image

7.b Fill out the first three fields in the SQS-Based S3 Input (Name, AWS Account, Assume Role)

- Choose and enter a descriptive name for the input
- Select the AWS Account you configured previously
- Select the IAM Role you configured previously
Example

image

7.c Fill out the next three fields in the SQS-Based S3 Input (Force using DLQ, AWS Region, Use Private Endpoints)

- Leave the **_Force using DLQ (Recommended)_** checkbox **__checked__**
- Select the **_AWS Region_** where you deployed the CloudFormation template previously
- Leave the **_Use Private Endpoints_** checkbox **__unchecked__**
Example

image

7.d Fill out the next three fields in the SQS-Based S3 Input (SQS Queue Name, SQS Batch Size, S3 File Decoder)

- Enter the **__URL__** of the SQS Queue in your **_SAP ECS account_**, **__not__** the ARN or just the name 
    - If the ARN for your SQS Queue in your **_SAP ECS account_** looks like this:
        - arn:aws:sqs:ap-south-1:121212121212:sap-hec-clz-ap-south-1-hec53-xsd-logserv
    - Then format it as a URL like this:
        - https://sqs.ap-south-1.amazonaws.com/121212121212/sap-hec-clz-ap-south-1-hec53-xsd-logserv
- Leave the **_SQS Batch Size_** set to 10
- Leave the **_S3 File Decoder_** set to Custom Logs
Example

image

7.e Fill out the next three fields in the SQS-Based S3 Input (Signature Validate All Events, Source Type, Index)

- **__Uncheck__** the **_Signature Validate All Events_** checkbox
- Enter the value of **_sap_logserv_logs_** in the **_Source Type_** field
- Enter the name of the Splunk index you want to use in the **_Index_** field
- Click on the **_Add_** button
Example

image


8. Confirm LogServ Logs are Being Ingested

After completing all the previous steps, verify that LogServ logs are successfully being ingested into Splunk.

The first events typically appear within 5-10 minutes of completing the SQS-Based S3 Input configuration. The input polls the SQS Queue at the configured interval (default 300 seconds), so allow a short lag before the first events arrive.

8.a Log in to your Splunk console and open the Search & Reporting app (or the SAP LogServ App if you have installed it)

8.b Run a basic search against the index you configured in the SQS-Based S3 Input to confirm events are flowing:

    index=<your_index_name> | stats count by sourcetype

If you are using the SAP LogServ App, you can use the provided index macro:

    `sap_logserv_idx_macro` | stats count by sourcetype

8.c You should see events from the LogServ sourcetypes your SAP LogServ subscription is forwarding. Depending on the log types enabled, expected sourcetypes may include (but are not limited to):

- `linux_messages_syslog`, `linux_secure`, `syslog` -- Linux OS events
- `isc:bind:query` -- DNS query events
- `squid:access` -- Proxy events
- `XmlWinEventLog` -- Windows events
- `sap:hana:audit`, `sap:hana:tracelogs` -- HANA database events
- `sap:abap:*` -- ABAP application events
- `sap:webdispatcher:access` -- Web Dispatcher events
- `sap:scc:audit`, `sap:scc:http_access` -- Cloud Connector events

8.d Confirm events are arriving with recent timestamps:

    index=<your_index_name> earliest=-1h | stats count by sourcetype, host

You should see recent events from multiple hosts.

8.e If no events appear after 15-20 minutes, troubleshoot as follows:

- **SQS Queue has no messages** -- Verify the SAP LogServ Support team has applied the updated access policies (from Section 3) to the SQS Queue and S3 Bucket in your **_SAP ECS account_**. Without those policies, your **_Secondary account_** cannot receive SQS notifications or read from the S3 Bucket.
- **Authentication or permission errors** -- In the Splunk Add-on for AWS, check the logs for errors: run `index=_internal source=*aws* log_level IN (ERROR,WARN) earliest=-1h | head 50`. Common causes are a mis-copied Access Key/Secret Key (Section 4), an incorrect IAM Role ARN (Section 6), or the IAM Role not yet propagated through AWS (wait 5 minutes after initial setup).
- **SQS URL format** -- Double-check the **_SQS Queue Name_** field in the SQS-Based S3 Input (Section 7) is the full URL (`https://sqs.<region>.amazonaws.com/<account-id>/<queue-name>`), not the ARN or bare queue name.
- **Wrong AWS Region** -- The CloudFormation template (Section 2) and the SQS-Based S3 Input (Section 7) must both be in the same AWS Region as the SQS Queue and S3 Bucket in your **_SAP ECS account_**.
Where to go next

Once you’ve confirmed ingestion is working, explore the dashboards in the LogServ UI App to see your LogServ data in action. The default landing page is the Environment Health dashboard, which provides a cross-cutting view of your entire SAP landscape.

AWS Remote S3 Filter Setup Walkthrough

Introduction

The deployment diagram below shows two different AWS accounts. For brevity and clarity, the AWS account at the top of the diagram will be referred to as the SAP ECS account and the second AWS account on the bottom left of the diagram will be referred to as the Secondary account from this point onward.

Ensure you have completed all of the steps in the Prerequisites and the Installation of the Splunk TA for SAP LogServ before you continue with the steps in this setup walkthrough.

Your SAP ECS LogServ subscription for AWS (e.g. in your SAP ECS account) should have one S3 Bucket in that account where LogServ will store your LogServ logs. It should also have one SQS Queue in your SAP ECS account that receives notifications when new logs are added to the S3 Bucket in that account.

You will need obtain the AWS ARN for the SQS Queue and the S3 Bucket that resides in your SAP ECS account prior to performing the steps in this setup walkthrough.

Take note of the AWS Region in your SAP ECS account where the S3 Bucket and SQS Queue are located as you will need to deploy the CloudFormation template provided in that same region in your Secondary account.

image


High Level Steps

Below are the high level steps for this setup process listed in the order they should be followed.

Please ensure the user you log in with in your AWS Secondary account has the appropriate permissions to perform all the steps outlined below.

  1. Obtain the ARNs for the SQS Queue and the S3 Bucket in your SAP ECS account
  2. Create a new S3 Bucket in the correct AWS Region in your Secondary account and upload the splunk-logserv-filter-lambda.zip file to the root of the bucket
  3. Deploy the AWS CloudFormation Template provided for this remote S3 filter deployment approach in your Secondary account
  4. Contact SAP LogServ support and request an update to the access policies for the SQS Queue and S3 Bucket residing in your SAP ECS account
  5. Create an Access Key for the new IAM User that was created with the CloudFormation template in your Secondary account
  6. Configure your AWS Secondary account in the Splunk Add-on for Amazon Web Services (AWS)
  7. Configure the new cross-account IAM Role from your AWS Secondary account in the Splunk Add-on for Amazon Web Services (AWS).
  8. Configure a new SQS-Based S3 Input in the Splunk Add-on for Amazon Web Services (AWS).
  9. Review the SQS Queue Trigger in the new Lambda function created with the CloudFormation template
  10. Confirm that LogServ logs are being ingested into Splunk


1. Obtain ARNs from SAP ECS Account

Before you can create the local S3 Bucket and deploy the CloudFormation template in your Secondary account, you need to obtain the AWS ARN for two resources that reside in your SAP ECS account:

  • The S3 Bucket that stores your LogServ logs
  • The SQS Queue that receives notifications when new logs are added to the S3 Bucket

You will also need to note the AWS Region where these resources are located, because the S3 Bucket and CloudFormation template in the next two sections must be created and deployed into that same region in your Secondary account.

What does an ARN look like?

AWS Amazon Resource Names (ARNs) follow a predictable format. Examples:

  • S3 Bucket ARN: arn:aws:s3:::sap-hec-clz-ap-south-1-hec53-xsd
  • SQS Queue ARN: arn:aws:sqs:ap-south-1:121212121212:sap-hec-clz-ap-south-1-hec53-xsd-logserv

How to obtain the ARNs:

1.a If you do not have console access to your SAP ECS account, contact SAP LogServ Support to obtain the ARN for the S3 Bucket, the ARN for the SQS Queue, and the AWS Region where these resources reside.

1.b If you do have console access to your SAP ECS account, obtain the ARNs yourself:

- For the **S3 Bucket**: navigate to the S3 console, find the bucket used for LogServ logs, click on the bucket name, and copy the ARN from the **_Properties_** tab.
- For the **SQS Queue**: navigate to the SQS console, find the queue that receives S3 notifications, click on the queue name, and copy the ARN from the **_Details_** section.

1.c Save both ARNs and the AWS Region in a secure location – you will reference them when creating the S3 Bucket (next section), deploying the CloudFormation Template, and configuring the SQS-Based S3 Input later in this walkthrough.


2. Create S3 Bucket with Lambda Function ZIP File

2.a Take note of the AWS Region in your SAP ECS account where the S3 Bucket and SQS Queue are located.

2.b Log into your AWS Secondary account and change to the region that matches the region in your SAP ECS account

2.c Choose a name for the new S3 bucket (splunk-logserv-lambda-binary is the default bucket name in the CloudFormation template used in the next section)

2.d Navigate to the S3 console and create a general purpose S3 bucket using all the default settings

2.e Upload the splunk-logserv-filter-lambda.zip file to the root of the S3 bucket

Example

image


3. Deploy CloudFormation Template

Before you deploy the CloudFormation template, take note of the AWS Region in your SAP ECS account where the S3 Bucket and SQS Queue are located as you will need to deploy the CloudFormation template provided in that same region in your Secondary account.

What does the CloudFormation template do?

Creates the following resources:

  • An IAM User - Default name is splunk_logserv_user
  • An IAM Policy named splunk-logserv-ta-policy
  • An IAM Role named splunk-logserv-ta-role
  • An Inline IAM Policy on the IAM User to assume the IAM Role created above
  • An SQS Queue - Default name is splunk-logserv-local-target-queue
  • A Dead Letter SQS Queue - Default name is splunk-logserv-local-target-queue-dlq
  • A Lambda Function - Default name is splunk-logserv-lambda-filter
  • A LogGroup used by the Lambda Function

3.a Navigate to the CloudFormation console (ensure the region you are in matches the region in your SAP ECS account)

3.b Click on the Create stack button and select With new resources (standard)

Example

image

3.c Upload the AWS CloudFormation Template file provided for this remote S3 filter deployment approach, then click the Next button

Example

image

3.d Enter a name for the CloudFormation Stack - splunk-logserv-remote-s3-filter

Example

image

3.e Enter just the name (not the ARN) of the S3 Bucket in your SAP ECS account in the CrossAccountS3Bucket parameter - If your ARN looks like this arn:aws:s3:::sap-hec-clz-ap-south-1-hec53-xsd then just use the name like this ap-hec-clz-ap-south-1-hec53-xsd

Example

image

3.f Enter the complete ARN of the SQS Queue in your SAP ECS account in the CrossAccountSQSQueueArn parameter

Example

image

3.g Enter the number of days in the past to filter messages on. Messages older than this will be discarded.

Example

image

3.h Enter comma-separated patterns for paths to exclude (e.g., ‘linux/cron,*/messages’). Format: <clz_dir>/<clz_subdir>. Leave empty to exclude nothing.

Example

image

3.i Enter comma-separated patterns for paths to include (e.g., ‘hana/hanaaudit,linux/*,webdispatcher/accesslogs’). Format: <clz_dir>/<clz_subdir>. Use ‘*/*’ to include all paths.

Example

image

3.j Choose and enter a name for the S3 Bucket in your Secondary account that you used to upload the Lambda Function ZIP File in the LambdaCodeBucket parameter

Example

image

3.k Choose and enter a name for the Lambda Function to be created in your Secondary account in the LambdaFunctionName parameter

Example

image

3.l Choose and enter a name for the SQS Queue to be created in your Secondary account in the LocalSQSQueueName parameter

Example

image

3.m Choose and enter a name for the IAM User to be created in your Secondary account in the NewIAMUserName parameter, then click on the Next button

Example

image

3.n Scroll down to the bottom of the page and check the I acknowledge that AWS CloudFormation might create IAM resources with custom names checkbox, then click on the Next button

Example

image

3.o Scroll down to the bottom of the page and click on the Submit button

Example

image

3.p Ensure the deployment of the CloudFormation template completes successfully

Example

image


4. Contact SAP LogServ Support

Once the deployment of the CloudFormation templates completes successfully, you will need to provide SAP LogServ Support with the ARN of the IAM Role named splunk-logserv-ta-role that was created by the template.

The ARN for the IAM Role should look like the one below but with your 12-digit AWS account Id of your Secondary account.

arn:aws:iam::secondary-account-id:role/splunk-logserv-ta-role

Example access policies for the SQS Queue and S3 Bucket residing in your SAP ECS account are provided as a reference to provide clarity on the specific permissions that need to be applied by the SAP LogServ Support team.


5. Create Access Key for IAM User

5.a Navigate to the IAM console in your Secondary account and search for the IAM User name you used when deploying the CloudFormation template. Click on the name of the IAM User to see the user details.

Example

image

5.b Click on the Security credentials tab in the middle of the screen. Scroll down and click on the Create access key button.

Example

image

5.c Select the Local code use case, check the Confirmation checkbox and click on the Next button.

Example

image

5.d Enter a description tag value if desired and click on the Create access key button.

Example

image

5.e Copy the values for both the Access key and the Secret access key and save them in a secure place as you will need them in the upcoming steps. Now click on the Done button.

Example

image


6. Configure Secondary Account (AWS Add-on)

Please ensure the user you log in with in your Splunk instance has the appropriate permissions to perform all the steps outlined below.

6.a Login to your Splunk console then find and open the Splunk Add-on for AWS App

Example

image

6.b Click on the Configuration tab, then click on the Account tab, then click on the Add button

Example

image

6.c Choose and enter a descriptive name for the account in the Name field. Enter the Access Key and the Secret Key you created for the IAM User in the respective fields. Leave the Region Category set to Global. Click on the Add button.

Example

image


7. Configure IAM Role (AWS Add-on)

7.a Click on the IAM Role tab to the right of the Account tab, then click on the Add button

Example

image

7.b Choose and enter a descriptive name for the role in the Name field. Enter the IAM Role ARN in the IAM Role ARN field, then click the Add button. The ARN for the IAM Role should look like the one below but with your 12-digit AWS account Id of your Secondary account.

- arn:aws:iam::**_secondary-account-id_**:role/splunk-logserv-ta-role
Example

image


8. Configure SQS-Based S3 Input (AWS Add-on)

8.a Click on the Inputs tab. Click on the Create New Input button. Select the Custom Data Type option at the bottom of the drop-down, then select the SQS-Based S3 (Recommended) option.

Example

image

8.b Fill out the first three fields in the SQS-Based S3 Input (Name, AWS Account, Assume Role)

- Choose and enter a descriptive name for the input
- Select the AWS Account you configured previously
- Select the IAM Role you configured previously
Example

image

8.c Fill out the next three fields in the SQS-Based S3 Input (Force using DLQ, AWS Region, Use Private Endpoints)

- Leave the **_Force using DLQ (Recommended)_** checkbox **__checked__**
- Select the **_AWS Region_** where you deployed the CloudFormation template previously
- Leave the **_Use Private Endpoints_** checkbox **__unchecked__**
Example

image

8.d Fill out the next three fields in the SQS-Based S3 Input (SQS Queue Name, SQS Batch Size, S3 File Decoder)

- Select the **_SQS Queue Name_** you used in step **12** when previously deploying the CloudFormation template
- Leave the **_SQS Batch Size_** set to 10
- Leave the **_S3 File Decoder_** set to Custom Logs
Example

image

8.e Fill out the next three fields in the SQS-Based S3 Input (Signature Validate All Events, Source Type, Index)

- **__Uncheck__** the **_Signature Validate All Events_** checkbox
- Enter the value of **_sap_logserv_logs_** in the **_Source Type_** field
- Enter the name of the Splunk index you want to use in the **_Index_** field
- Click on the **_Add_** button
Example

image


9. Review SQS Queue Trigger

9.a Navigate to the Lambda console in your Secondary account and ensure the region you are in matches the region in your SAP ECS account. Find the Lambda function that was created by the CloudFormation template and click on its name to view details.

Example

image

9.b If you do not see an existing SQS Trigger in the Function overiew diagram as seen in the example image below, then follow the steps in the Create SQS Queue Trigger section below, otherwise follow the steps in the Configure SQS Queue Trigger section below.

Example

image


Create SQS Queue Trigger

  1. Click on the Add trigger button on the left side of the Function overview diagram to create a new SQS Queue Trigger
Example

image

  1. Click on the dropdown and select SQS as the trigger source
Example

image

  1. Fill out the first three fields in the SQS Trigger (SQS queue ARN, Activate trigger, Enable metrics)

    • Enter the complete ARN of the SQS Queue in your SAP ECS account
    • Check the Activate trigger checkbox
    • Check the Enable metrics checkbox
Example

image

  1. Fill out the next three fields in the SQS Trigger (Batch size, Batch window, Maximum concurrency)

    • Set the Batch Size to 10
    • Set the Batch window to 5
    • Set the Maximum concurrency to 8
Example

image

  1. Fill out the last field in the SQS Trigger (Report batch item failures) and save the trigger

    • Check the Report batch item failures checkbox
    • Click on the Add button on the bottom right of the screen to save the trigger
Example

image


Configure SQS Queue Trigger

  1. Click on the SQS rectangle on the left side of the Function overview diagram to navigate to the SQS Queue Trigger configuration
Example

image

  1. Check the checkbox for the SQS trigger and then click on the Edit button
Example

image

  1. Validate the first three fields in the SQS Trigger (SQS queue ARN, Activate trigger, Enable metrics)

    • Ensure the complete ARN of the SQS Queue in your SAP ECS account is referenced here
    • Ensure the Activate trigger checkbox is checked
    • Ensure the Enable metrics checkbox is checked
Example

image

  1. Validate the next three fields in the SQS Trigger (Batch size, Batch window, Maximum concurrency)

    • Ensure the Batch Size is set to 10
    • Ensure the Batch window is set to 5
    • Ensure the Maximum concurrency is set to 8
Example

image

  1. Validate the last field in the SQS Trigger (Report batch item failures) and save the trigger

    • Ensure the Report batch item failures checkbox is checked
    • Click on the Save button on the bottom right of the screen to save the trigger
Example

image


10. Confirm LogServ Logs are Being Ingested

After completing all the previous steps, verify that LogServ logs are successfully being ingested into Splunk.

The first events typically appear within 5-10 minutes of completing the SQS-Based S3 Input configuration. The filter Lambda function receives SQS notifications from the SAP ECS account, drops any that fail your include/exclude/days filters, and re-emits the rest to the local SQS Queue in your Secondary account. The Splunk SQS-Based S3 Input then polls the local SQS Queue at the configured interval (default 300 seconds), so allow a short lag before the first events arrive.

10.a Log in to your Splunk console and open the Search & Reporting app (or the SAP LogServ App if you have installed it)

10.b Run a basic search against the index you configured in the SQS-Based S3 Input to confirm events are flowing:

    index=<your_index_name> | stats count by sourcetype

If you are using the SAP LogServ App, you can use the provided index macro:

    `sap_logserv_idx_macro` | stats count by sourcetype

10.c You should see events from the LogServ sourcetypes that match your include pattern and are not blocked by your exclude / days-in-past filters. Depending on the log types enabled, expected sourcetypes may include (but are not limited to):

- `linux_messages_syslog`, `linux_secure`, `syslog` -- Linux OS events
- `isc:bind:query` -- DNS query events
- `squid:access` -- Proxy events
- `XmlWinEventLog` -- Windows events
- `sap:hana:audit`, `sap:hana:tracelogs` -- HANA database events
- `sap:abap:*` -- ABAP application events
- `sap:webdispatcher:access` -- Web Dispatcher events
- `sap:scc:audit`, `sap:scc:http_access` -- Cloud Connector events

10.d Confirm events are arriving with recent timestamps:

    index=<your_index_name> earliest=-1h | stats count by sourcetype, host

You should see recent events from multiple hosts.

10.e If no events appear after 15-20 minutes, troubleshoot as follows:

- **Filter dropped everything** -- Review the **DaysInPast**, **IncludeFilters**, and **ExcludeFilters** parameters you entered in Section 3. A narrow include pattern, a short `days_in_past` window, or an over-broad exclude pattern can legitimately drop every message before it reaches your local SQS Queue. Check the Lambda function's CloudWatch Logs for filter-decision messages and adjust the CloudFormation stack parameters if needed.
- **Lambda function errors** -- In the AWS Console, open CloudWatch Logs for the filter Lambda created by the CloudFormation template (Section 3). Look for cross-account access errors reading from the SAP ECS S3 Bucket or pushing to the local SQS Queue. Common causes are missing access policies on the SAP ECS resources (Section 4) or an IAM Role propagation delay (wait 5 minutes after initial setup).
- **Local SQS Queue has no messages** -- Verify the SAP LogServ Support team has applied the updated access policies (from Section 4) to the SQS Queue and S3 Bucket in your **_SAP ECS account_**. Without those policies, the filter Lambda cannot receive SQS notifications from the SAP ECS account or read from the SAP ECS S3 Bucket. Also verify the SQS Queue Trigger on the Lambda function is active (Section 9).
- **Authentication or permission errors** -- In the Splunk Add-on for AWS, check the logs for errors: run `index=_internal source=*aws* log_level IN (ERROR,WARN) earliest=-1h | head 50`. Common causes are a mis-copied Access Key/Secret Key (Section 5), an incorrect IAM Role ARN (Section 7), or the IAM Role not yet propagated through AWS (wait 5 minutes after initial setup).
- **SQS URL format** -- Double-check the **_SQS Queue Name_** field in the SQS-Based S3 Input (Section 8) points to the **_local_** SQS Queue that was created by the CloudFormation template in your **_Secondary account_** (not the SAP ECS SQS Queue).
- **Wrong AWS Region** -- The CloudFormation template (Section 3) and the SQS-Based S3 Input (Section 8) must both be in the same AWS Region as the SQS Queue and S3 Bucket in your **_SAP ECS account_**.
Where to go next

Once you’ve confirmed ingestion is working, explore the dashboards in the LogServ UI App to see your LogServ data in action. The default landing page is the Environment Health dashboard, which provides a cross-cutting view of your entire SAP landscape.

AWS Remote S3 Copy Setup Walkthrough

Introduction

The deployment diagram below shows two different AWS accounts. For brevity and clarity, the AWS account at the top of the diagram will be referred to as the SAP ECS account and the second AWS account on the bottom left of the diagram will be referred to as the Secondary account from this point onward.

Ensure you have completed all of the steps in the Prerequisites and the Installation of the Splunk TA for SAP LogServ before you continue with the steps in this setup walkthrough.

Your SAP ECS LogServ subscription for AWS (e.g. in your SAP ECS account) should have one S3 Bucket in that account where LogServ will store your LogServ logs. It should also have one SQS Queue in your SAP ECS account that receives notifications when new logs are added to the S3 Bucket in that account.

You will need obtain the AWS ARN for the SQS Queue and the S3 Bucket that resides in your SAP ECS account prior to performing the steps in this setup walkthrough.

Take note of the AWS Region in your SAP ECS account where the S3 Bucket and SQS Queue are located as you will need to deploy the CloudFormation template provided in that same region in your Secondary account.

image


High Level Steps

Below are the high level steps for this setup process listed in the order they should be followed.

Please ensure the user you log in with in your AWS Secondary account has the appropriate permissions to perform all the steps outlined below.

  1. Obtain the ARNs for the SQS Queue and the S3 Bucket in your SAP ECS account
  2. Create a new S3 Bucket in the correct AWS Region in your Secondary account and upload the splunk-logserv-lambda-binary.zip file to the root of the bucket
  3. Deploy the AWS CloudFormation Template provided for this remote S3 connect deployment approach in your Secondary account
  4. Contact SAP LogServ support and request an update to the access policies for the SQS Queue and S3 Bucket residing in your SAP ECS account
  5. Create an Access Key for the new IAM User that was created with the CloudFormation template in your Secondary account
  6. Configure your AWS Secondary account in the Splunk Add-on for Amazon Web Services (AWS)
  7. Configure the new cross-account IAM Role from your AWS Secondary account in the Splunk Add-on for Amazon Web Services (AWS).
  8. Configure a new SQS-Based S3 Input in the Splunk Add-on for Amazon Web Services (AWS).
  9. Review the SQS Queue Trigger in the new Lambda function created with the CloudFormation template
  10. Confirm that LogServ logs are being ingested into Splunk


1. Obtain ARNs from SAP ECS Account

Before you can create the local S3 Bucket and deploy the CloudFormation template in your Secondary account, you need to obtain the AWS ARN for two resources that reside in your SAP ECS account:

  • The S3 Bucket that stores your LogServ logs
  • The SQS Queue that receives notifications when new logs are added to the S3 Bucket

You will also need to note the AWS Region where these resources are located, because the S3 Bucket and CloudFormation template in the next two sections must be created and deployed into that same region in your Secondary account.

What does an ARN look like?

AWS Amazon Resource Names (ARNs) follow a predictable format. Examples:

  • S3 Bucket ARN: arn:aws:s3:::sap-hec-clz-ap-south-1-hec53-xsd
  • SQS Queue ARN: arn:aws:sqs:ap-south-1:121212121212:sap-hec-clz-ap-south-1-hec53-xsd-logserv

How to obtain the ARNs:

1.a If you do not have console access to your SAP ECS account, contact SAP LogServ Support to obtain the ARN for the S3 Bucket, the ARN for the SQS Queue, and the AWS Region where these resources reside.

1.b If you do have console access to your SAP ECS account, obtain the ARNs yourself:

- For the **S3 Bucket**: navigate to the S3 console, find the bucket used for LogServ logs, click on the bucket name, and copy the ARN from the **_Properties_** tab.
- For the **SQS Queue**: navigate to the SQS console, find the queue that receives S3 notifications, click on the queue name, and copy the ARN from the **_Details_** section.

1.c Save both ARNs and the AWS Region in a secure location – you will reference them when creating the S3 Bucket (next section), deploying the CloudFormation Template, and configuring the SQS-Based S3 Input later in this walkthrough.


2. Create S3 Bucket with Lambda Function ZIP File

2.a Take note of the AWS Region in your SAP ECS account where the S3 Bucket and SQS Queue are located.

2.b Log into your AWS Secondary account and change to the region that matches the region in your SAP ECS account

2.c Choose a name for the new S3 bucket (splunk-logserv-lambda-binary is the default bucket name in the CloudFormation template used in the next section)

2.d Navigate to the S3 console and create a general purpose S3 bucket using all the default settings

2.e Upload the splunk-logserv-lambda-binary.zip file to the root of the S3 bucket

Example

image


3. Deploy CloudFormation Template

Before you deploy the CloudFormation template, take note of the AWS Region in your SAP ECS account where the S3 Bucket and SQS Queue are located as you will need to deploy the CloudFormation template provided in that same region in your Secondary account.

What does the CloudFormation template do?

Creates the following resources:

  • An IAM User - Default name is splunk_logserv_user
  • An IAM Policy named splunk-logserv-ta-policy
  • An IAM Role named splunk-logserv-ta-role
  • An Inline IAM Policy on the IAM User to assume the IAM Role created above
  • An S3 Bucket with an Event Notification to the SQS Queue below- Default name is splunk-logserv-local-target-bucket
  • An SQS Queue - Default name is splunk-logserv-local-target-queue
  • A Dead Letter SQS Queue - Default name is splunk-logserv-local-target-queue-dlq
  • A Lambda Function - Default name is splunk-logserv-lambda-logforwarder
  • A LogGroup used by the Lambda Function

3.a Navigate to the CloudFormation console (ensure the region you are in matches the region in your SAP ECS account)

3.b Click on the Create stack button and select With new resources (standard)

Example

image

3.c Upload the AWS CloudFormation Template file provided for this remote S3 copy deployment approach, then click the Next button

Example

image

3.d Enter a name for the CloudFormation Stack - splunk-logserv-remote-s3-copy

Example

image

3.e Enter just the name (not the ARN) of the S3 Bucket in your SAP ECS account in the CrossAccountS3Bucket parameter - If your ARN looks like this arn:aws:s3:::sap-hec-clz-ap-south-1-hec53-xsd then just use the name like this ap-hec-clz-ap-south-1-hec53-xsd

Example

image

3.f Enter the complete ARN of the SQS Queue in your SAP ECS account in the CrossAccountSQSQueueArn parameter

Example

image

3.g Choose and enter a name for the S3 Bucket in your Secondary account that you used to upload the Lambda Function ZIP File in the LambdaCodeBucket parameter

Example

image

3.h Choose and enter a name for the Lambda Function to be created in your Secondary account in the LambdaFunctionName parameter

Example

image

3.i Choose and enter a name for the S3 Bucket to be created in your Secondary account in the LocalS3BucketName parameter

Example

image

3.j Choose and enter a name for the SQS Queue to be created in your Secondary account in the LocalSQSQueueName parameter

Example

image

3.k Choose and enter a name for the IAM User to be created in your Secondary account in the NewIAMUserName parameter, then click on the Next button

Example

image

3.l Scroll down to the bottom of the page and check the I acknowledge that AWS CloudFormation might create IAM resources with custom names checkbox, then click on the Next button

Example

image

3.m Scroll down to the bottom of the page and click on the Submit button

Example

image

3.n Ensure the deployment of the CloudFormation template completes successfully

Example

image


4. Contact SAP LogServ Support

Once the deployment of the CloudFormation templates completes successfully, you will need to provide SAP LogServ Support with the ARN of the IAM Role named splunk-logserv-ta-role that was created by the template.

The ARN for the IAM Role should look like the one below but with your 12-digit AWS account Id of your Secondary account.

arn:aws:iam::secondary-account-id:role/splunk-logserv-ta-role

Example access policies for the SQS Queue and S3 Bucket residing in your SAP ECS account are provided as a reference to provide clarity on the specific permissions that need to be applied by the SAP LogServ Support team.


5. Create Access Key for IAM User

5.a Navigate to the IAM console in your Secondary account and search for the IAM User name you used when deploying the CloudFormation template. Click on the name of the IAM User to see the user details.

Example

image

5.b Click on the Security credentials tab in the middle of the screen. Scroll down and click on the Create access key button.

Example

image

5.c Select the Local code use case, check the Confirmation checkbox and click on the Next button.

Example

image

5.d Enter a description tag value if desired and click on the Create access key button.

Example

image

5.e Copy the values for both the Access key and the Secret access key and save them in a secure place as you will need them in the upcoming steps. Now click on the Done button.

Example

image


6. Configure Secondary Account (AWS Add-on)

Please ensure the user you log in with in your Splunk instance has the appropriate permissions to perform all the steps outlined below.

6.a Login to your Splunk console then find and open the Splunk Add-on for AWS App

Example

image

6.b Click on the Configuration tab, then click on the Account tab, then click on the Add button

Example

image

6.c Choose and enter a descriptive name for the account in the Name field. Enter the Access Key and the Secret Key you created for the IAM User in the respective fields. Leave the Region Category set to Global. Click on the Add button.

Example

image


7. Configure IAM Role (AWS Add-on)

7.a Click on the IAM Role tab to the right of the Account tab, then click on the Add button

Example

image

7.b Choose and enter a descriptive name for the role in the Name field. Enter the IAM Role ARN in the IAM Role ARN field, then click the Add button. The ARN for the IAM Role should look like the one below but with your 12-digit AWS account Id of your Secondary account.

- arn:aws:iam::**_secondary-account-id_**:role/splunk-logserv-ta-role
Example

image


8. Configure SQS-Based S3 Input (AWS Add-on)

8.a Click on the Inputs tab. Click on the Create New Input button. Select the Custom Data Type option at the bottom of the drop-down, then select the SQS-Based S3 (Recommended) option.

Example

image

8.b Fill out the first three fields in the SQS-Based S3 Input (Name, AWS Account, Assume Role)

- Choose and enter a descriptive name for the input
- Select the AWS Account you configured previously
- Select the IAM Role you configured previously
Example

image

8.c Fill out the next three fields in the SQS-Based S3 Input (Force using DLQ, AWS Region, Use Private Endpoints)

- Leave the **_Force using DLQ (Recommended)_** checkbox **__checked__**
- Select the **_AWS Region_** where you deployed the CloudFormation template previously
- Leave the **_Use Private Endpoints_** checkbox **__unchecked__**
Example

image

8.d Fill out the next three fields in the SQS-Based S3 Input (SQS Queue Name, SQS Batch Size, S3 File Decoder)

- Select the **_SQS Queue Name_** you used in step **10** when previously deploying the CloudFormation template
- Leave the **_SQS Batch Size_** set to 10
- Leave the **_S3 File Decoder_** set to Custom Logs
Example

image

8.e Fill out the next three fields in the SQS-Based S3 Input (Signature Validate All Events, Source Type, Index)

- **__Uncheck__** the **_Signature Validate All Events_** checkbox
- Enter the value of **_sap_logserv_logs_** in the **_Source Type_** field
- Enter the name of the Splunk index you want to use in the **_Index_** field
- Click on the **_Add_** button
Example

image


9. Review SQS Queue Trigger

9.a Navigate to the Lambda console in your Secondary account and ensure the region you are in matches the region in your SAP ECS account. Find the Lambda function that was created by the CloudFormation template and click on its name to view details.

Example

image

9.b If you do not see an existing SQS Trigger in the Function overiew diagram as seen in the example image below, then follow the steps in the Create SQS Queue Trigger section below, otherwise follow the steps in the Configure SQS Queue Trigger section below.

Example

image


Create SQS Queue Trigger

  1. Click on the Add trigger button on the left side of the Function overview diagram to create a new SQS Queue Trigger
Example

image

  1. Click on the dropdown and select SQS as the trigger source
Example

image

  1. Fill out the first three fields in the SQS Trigger (SQS queue ARN, Activate trigger, Enable metrics)

    • Enter the complete ARN of the SQS Queue in your SAP ECS account
    • Check the Activate trigger checkbox
    • Check the Enable metrics checkbox
Example

image

  1. Fill out the next three fields in the SQS Trigger (Batch size, Batch window, Maximum concurrency)

    • Set the Batch Size to 10
    • Set the Batch window to 5
    • Set the Maximum concurrency to 8
Example

image

  1. Fill out the last field in the SQS Trigger (Report batch item failures) and save the trigger

    • Check the Report batch item failures checkbox
    • Click on the Add button on the bottom right of the screen to save the trigger
Example

image


Configure SQS Queue Trigger

  1. Click on the SQS rectangle on the left side of the Function overview diagram to navigate to the SQS Queue Trigger configuration
Example

image

  1. Check the checkbox for the SQS trigger and then click on the Edit button
Example

image

  1. Validate the first three fields in the SQS Trigger (SQS queue ARN, Activate trigger, Enable metrics)

    • Ensure the complete ARN of the SQS Queue in your SAP ECS account is referenced here
    • Ensure the Activate trigger checkbox is checked
    • Ensure the Enable metrics checkbox is checked
Example

image

  1. Validate the next three fields in the SQS Trigger (Batch size, Batch window, Maximum concurrency)

    • Ensure the Batch Size is set to 10
    • Ensure the Batch window is set to 5
    • Ensure the Maximum concurrency is set to 8
Example

image

  1. Validate the last field in the SQS Trigger (Report batch item failures) and save the trigger

    • Ensure the Report batch item failures checkbox is checked
    • Click on the Save button on the bottom right of the screen to save the trigger
Example

image


10. Confirm LogServ Logs are Being Ingested

After completing all the previous steps, verify that LogServ logs are successfully being ingested into Splunk.

The first events typically appear within 5-10 minutes of completing the SQS-Based S3 Input configuration. The Lambda function copies objects from the SAP ECS S3 Bucket to the local target S3 Bucket in your Secondary account when an SQS notification arrives, then the SQS-Based S3 Input polls the local SQS Queue at the configured interval (default 300 seconds), so allow a short lag before the first events arrive.

10.a Log in to your Splunk console and open the Search & Reporting app (or the SAP LogServ App if you have installed it)

10.b Run a basic search against the index you configured in the SQS-Based S3 Input to confirm events are flowing:

    index=<your_index_name> | stats count by sourcetype

If you are using the SAP LogServ App, you can use the provided index macro:

    `sap_logserv_idx_macro` | stats count by sourcetype

10.c You should see events from the LogServ sourcetypes your SAP LogServ subscription is forwarding. Depending on the log types enabled, expected sourcetypes may include (but are not limited to):

- `linux_messages_syslog`, `linux_secure`, `syslog` -- Linux OS events
- `isc:bind:query` -- DNS query events
- `squid:access` -- Proxy events
- `XmlWinEventLog` -- Windows events
- `sap:hana:audit`, `sap:hana:tracelogs` -- HANA database events
- `sap:abap:*` -- ABAP application events
- `sap:webdispatcher:access` -- Web Dispatcher events
- `sap:scc:audit`, `sap:scc:http_access` -- Cloud Connector events

10.d Confirm events are arriving with recent timestamps:

    index=<your_index_name> earliest=-1h | stats count by sourcetype, host

You should see recent events from multiple hosts.

10.e If no events appear after 15-20 minutes, troubleshoot as follows:

- **Lambda function errors** -- In the AWS Console, open CloudWatch Logs for the Lambda function created by the CloudFormation template (Section 3). Check for errors copying objects from the SAP ECS S3 Bucket to your local S3 Bucket. Common causes are missing access policies on the SAP ECS resources (Section 4) or an IAM Role propagation delay (wait 5 minutes after initial setup).
- **Local SQS Queue has no messages** -- Verify the SAP LogServ Support team has applied the updated access policies (from Section 4) to the SQS Queue and S3 Bucket in your **_SAP ECS account_**. Without those policies, the Lambda function cannot receive SQS notifications from the SAP ECS account or read from the SAP ECS S3 Bucket. Also verify the SQS Queue Trigger on the Lambda function is active (Section 9).
- **Authentication or permission errors** -- In the Splunk Add-on for AWS, check the logs for errors: run `index=_internal source=*aws* log_level IN (ERROR,WARN) earliest=-1h | head 50`. Common causes are a mis-copied Access Key/Secret Key (Section 5), an incorrect IAM Role ARN (Section 7), or the IAM Role not yet propagated through AWS (wait 5 minutes after initial setup).
- **SQS URL format** -- Double-check the **_SQS Queue Name_** field in the SQS-Based S3 Input (Section 8) points to the **_local_** SQS Queue that was created by the CloudFormation template in your **_Secondary account_** (not the SAP ECS SQS Queue).
- **Wrong AWS Region** -- The CloudFormation template (Section 3) and the SQS-Based S3 Input (Section 8) must both be in the same AWS Region as the SQS Queue and S3 Bucket in your **_SAP ECS account_**.
Where to go next

Once you’ve confirmed ingestion is working, explore the dashboards in the LogServ UI App to see your LogServ data in action. The default landing page is the Environment Health dashboard, which provides a cross-cutting view of your entire SAP landscape.

Ended: AWS Setup Walkthroughs

AWS Remote S3 Connect to Filter Migration Walkthrough

Introduction

This guide walks you through migrating from an existing splunk-logserv-remote-s3-connect deployment to the splunk-logserv-remote-s3-filter deployment without deleting or recreating your existing IAM User and IAM Role resources.

The migration adds Lambda-based S3 event filtering capabilities to your existing deployment, allowing you to filter S3 event notifications based on time and path patterns before they are processed by Splunk.

Also available: Native TA Index-Time Filtering

Starting with version 0.0.3, the TA also includes built-in index-time filtering that works inside Splunk itself. This can be used alongside or instead of Lambda-based filtering. After completing this migration (or with any deployment scenario), see Configuring Filters or the section at the bottom of this page.

This migration guide assumes you have already completed the AWS Remote S3 Connect Setup and have a working deployment in your Secondary account.


What the Migration Does

The Python migration script performs the following actions on your existing deployment:

Migration Actions

Updates to Existing Resources:

  • Updates the existing IAM Role trust policy to allow Lambda service
  • Attaches the AWSLambdaBasicExecutionRole managed policy to the existing IAM Role
  • Adds CloudWatch Logs permissions to the existing IAM policy
  • Adds local SQS queue permissions to the existing IAM policy

New Resources Created:

  • An SQS Queue - Default name is splunk-logserv-local-target-queue
  • A Dead Letter SQS Queue - Default name is splunk-logserv-local-target-queue-dlq
  • A Lambda Function - Default name is splunk-logserv-lambda-filter
  • A CloudWatch LogGroup used by the Lambda Function
  • An Event Source Mapping connecting the Lambda to the cross-account SQS Queue


Architecture After Migration

After migration, the architecture will match the splunk-logserv-remote-s3-filter deployment:

  • S3 event notifications from the SAP ECS account are received by the Lambda function
  • The Lambda function filters events based on time (days in past) and path patterns (include/exclude)
  • Matching events are forwarded to a local SQS queue in your Secondary account
  • Splunk polls the local SQS queue for filtered notifications


Prerequisites

Before starting the migration, ensure you have the following:

Existing Deployment:

  • A working splunk-logserv-remote-s3-connect deployment in your Secondary account
  • The ARN and name of the existing IAM Role (e.g., splunk-logserv-ta-role)
  • The ARN of the existing IAM User (e.g., arn:aws:iam::123456789012:user/splunk_logserv_user)
  • The name of the existing IAM policy attached to the role (e.g., splunk-logserv-ta-policy)

Cross-Account Information:

  • The ARN of the SQS Queue in your SAP ECS account
  • The name of the S3 Bucket in your SAP ECS account

Local Environment:

  • Python 3.9 or higher installed
  • boto3 library installed
  • AWS CLI configured with appropriate credentials/profile


High Level Steps

Below are the high level steps for the migration process listed in the order they should be followed.

Please ensure the user you log in with in your AWS Secondary account has the appropriate permissions to perform all the steps outlined below.

  1. Install Python and boto3 dependencies (if not already installed)
  2. Create a new S3 Bucket in your Secondary account and upload the Lambda function ZIP file
  3. Download the migration scripts and configuration file
  4. Configure the migration configuration file with your deployment details
  5. Run the migration script in dry-run mode, then execute it to perform the migration
  6. Update the Splunk AWS Add-on SQS-Based S3 Input to use the new local queue
  7. Verify the migration is successful


1. Install Python and boto3

The migration scripts require Python 3.9 or higher and the boto3 library.

Check Python Version

Open a command prompt or terminal and run the following command to check your Python version:

python --version

If Python is not installed or the version is below 3.9, download and install Python from the official Python website.

On Windows, ensure you check the Add Python to PATH option during installation.

Check boto3 Installation

Run the following command to check if boto3 is installed:

pip show boto3

If boto3 is installed, you will see version and location information. If not installed, you will see a warning message.

Install boto3

If boto3 is not installed, run the following command to install it:

pip install boto3


2. Create S3 Bucket with Lambda Function ZIP File

2.a Take note of the AWS Region in your SAP ECS account where the S3 Bucket and SQS Queue are located.

2.b Log into your AWS Secondary account and change to the region that matches the region in your SAP ECS account

2.c Choose a name for the new S3 bucket (splunk-logserv-lambda-binary is the default bucket name in the migration configuration file)

2.d Navigate to the S3 console and create a general purpose S3 bucket using all the default settings

2.e Upload the splunk-logserv-filter-lambda.zip file to the root of the S3 bucket


3. Download Migration Scripts

Download the following files from the GitHub repository to a local directory on your machine:

File Description
migrate-config.json Configuration file with all migration parameters
migrate-to-filter.py Python script that performs the migration
migrate-to-filter-rollback.py Python script that rolls back the migration


4. Configure Migration Settings

Open the migrate-config.json file in a text editor and update the values to match your deployment.

Existing Resources Configuration

Update the existing_resources section with the details from your current splunk-logserv-remote-s3-connect deployment:

"existing_resources": {
    "iam_user_arn": "arn:aws:iam::YOUR_ACCOUNT_ID:user/YOUR_IAM_USER_NAME",
    "iam_role_name": "YOUR_IAM_ROLE_NAME",
    "iam_role_arn": "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_IAM_ROLE_NAME",
    "iam_role_policy_name": "YOUR_IAM_POLICY_NAME"
}
Parameter Description Example
iam_user_arn ARN of the existing IAM User arn:aws:iam::112543817624:user/splunk_logserv_user
iam_role_name Name of the existing IAM Role splunk-logserv-ta-role
iam_role_arn ARN of the existing IAM Role arn:aws:iam::112543817624:role/splunk-logserv-ta-role
iam_role_policy_name Name of the existing inline policy on the IAM Role splunk-logserv-ta-policy

Cross-Account Configuration

Update the cross_account section with the details from your SAP ECS account:

"cross_account": {
    "sqs_queue_arn": "arn:aws:sqs:REGION:SAP_ACCOUNT_ID:QUEUE_NAME",
    "s3_bucket_name": "SAP_S3_BUCKET_NAME"
}
Parameter Description Example
sqs_queue_arn ARN of the SQS Queue in your SAP ECS account arn:aws:sqs:ap-south-1:395719258032:sap-logserv-queue
s3_bucket_name Name of the S3 Bucket in your SAP ECS account sap-logserv-bucket

Local SQS Configuration

Update the local_sqs section to configure the local SQS queue that will receive filtered notifications:

"local_sqs": {
    "queue_name": "splunk-logserv-local-target-queue",
    "visibility_timeout_seconds": 600,
    "message_retention_seconds": 1209600,
    "dlq_max_receive_count": 3
}
Parameter Description Default
queue_name Name for the local SQS queue splunk-logserv-local-target-queue
visibility_timeout_seconds Visibility timeout in seconds 600 (10 minutes)
message_retention_seconds Message retention period in seconds 1209600 (14 days)
dlq_max_receive_count Max receives before sending to DLQ 3

Lambda Function Configuration

Update the lambda_function section to configure the Lambda filter function:

"lambda_function": {
    "function_name": "splunk-logserv-lambda-filter",
    "code_s3_bucket": "splunk-logserv-lambda-binary",
    "code_s3_key": "splunk-logserv-filter-lambda.zip",
    "handler": "log-filterer-lambda.lambda_handler",
    "runtime": "python3.12",
    "timeout_seconds": 300,
    "memory_size_mb": 512
}
Parameter Description Default
function_name Name for the Lambda function splunk-logserv-lambda-filter
code_s3_bucket S3 bucket containing the Lambda ZIP file splunk-logserv-lambda-binary
code_s3_key S3 key (path) to the Lambda ZIP file splunk-logserv-filter-lambda.zip
handler Lambda handler function log-filterer-lambda.lambda_handler
runtime Lambda runtime python3.12
timeout_seconds Lambda timeout in seconds 300 (5 minutes)
memory_size_mb Lambda memory allocation in MB 512

Filter Settings Configuration

Update the filter_settings section to configure the S3 event filtering behavior:

"filter_settings": {
    "days_in_the_past": 7,
    "include_filters": "linux/*,hana/*",
    "exclude_filters": "linux/cron,linux/messages,linux/localmessages,linux/slapd,linux/sudolog,linux/warn"
}
Parameter Description Default
days_in_the_past Filter out messages older than this many days 7
include_filters Comma-separated fnmatch patterns for paths to include linux/*,hana/*
exclude_filters Comma-separated fnmatch patterns for paths to exclude linux/cron,linux/messages,...

Use */* for include_filters to include all paths. Leave exclude_filters empty to exclude nothing.

Event Source Mapping Configuration

Update the event_source_mapping section to configure the Lambda trigger:

"event_source_mapping": {
    "batch_size": 10,
    "maximum_batching_window_seconds": 5,
    "maximum_concurrency": 10
}
Parameter Description Default
batch_size Number of messages per Lambda invocation 10
maximum_batching_window_seconds Max time to wait for a full batch 5
maximum_concurrency Max concurrent Lambda executions 10


5. Run Migration Script

Command Line Arguments

The migration script supports the following command line arguments:

Argument Short Required Description
--config -c Yes Path to the migration configuration JSON file
--region -r Yes AWS region (must match your existing deployment region)
--profile -p No AWS CLI profile name (uses default credentials if not specified)
--dry-run -d No Preview changes without making any modifications

Dry Run Mode

Always run the migration script in dry-run mode first to preview the changes that will be made.

Windows Command Prompt:

python migrate-to-filter.py --config migrate-config.json --region us-east-1 --dry-run

Windows Command Prompt with AWS Profile:

python migrate-to-filter.py --config migrate-config.json --region us-east-1 --profile my-aws-profile --dry-run

Linux/macOS Terminal:

python3 migrate-to-filter.py --config migrate-config.json --region us-east-1 --dry-run

Linux/macOS Terminal with AWS Profile:

python3 migrate-to-filter.py --config migrate-config.json --region us-east-1 --profile my-aws-profile --dry-run

Review the dry-run output to ensure all steps are correct and the existing resources are found.

Execute Migration

Once you have verified the dry-run output, run the migration script without the --dry-run flag:

Windows Command Prompt:

python migrate-to-filter.py --config migrate-config.json --region us-east-1

Windows Command Prompt with AWS Profile:

python migrate-to-filter.py --config migrate-config.json --region us-east-1 --profile my-aws-profile

Linux/macOS Terminal:

python3 migrate-to-filter.py --config migrate-config.json --region us-east-1

Linux/macOS Terminal with AWS Profile:

python3 migrate-to-filter.py --config migrate-config.json --region us-east-1 --profile my-aws-profile

The script will display progress for each step and provide a summary upon completion, including the new local SQS queue URL that you will need for the next step.


6. Update Splunk AWS Add-on Configuration

After the migration completes successfully, you need to update your existing SQS-Based S3 Input in the Splunk Add-on for AWS to use the new local SQS queue.

6.a Log into your Splunk instance and open the Splunk Add-on for AWS

6.b Navigate to Inputs and find your existing SQS-Based S3 Input

6.c Click Edit on the input

6.d Update the SQS Queue Name field to use the new local queue URL from the migration output - Example: https://sqs.ap-south-1.amazonaws.com/112543817624/splunk-logserv-local-target-queue

6.e Click Save to apply the changes

The input will restart and begin polling the new local filtered queue.


7. Verify Migration

After completing the migration and updating the Splunk configuration:

7.a Check Lambda Logs - Navigate to CloudWatch Logs in the AWS Console and verify the Lambda function is receiving and processing messages

7.b Check Local SQS Queue - Navigate to SQS in the AWS Console and verify messages are appearing in the local queue

7.c Check Splunk - Run a search for recent LogServ logs to confirm data is being ingested: 

index=your_index sourcetype=sap_logserv_logs earliest=-1h


Rollback Migration

If you need to rollback the migration and return to the original splunk-logserv-remote-s3-connect configuration, use the rollback script.

Rollback Command Line Arguments

The rollback script supports the same command line arguments as the migration script:

Argument Short Required Description
--config -c Yes Path to the migration configuration JSON file
--region -r Yes AWS region
--profile -p No AWS CLI profile name (uses default credentials if not specified)
--dry-run -d No Preview changes without making any modifications

Rollback Dry Run

Always run the rollback script in dry-run mode first to preview what will be removed.

Windows Command Prompt:

python migrate-to-filter-rollback.py --config migrate-config.json --region us-east-1 --dry-run

Windows Command Prompt with AWS Profile:

python migrate-to-filter-rollback.py --config migrate-config.json --region us-east-1 --profile my-aws-profile --dry-run

Linux/macOS Terminal:

python3 migrate-to-filter-rollback.py --config migrate-config.json --region us-east-1 --dry-run

Linux/macOS Terminal with AWS Profile:

python3 migrate-to-filter-rollback.py --config migrate-config.json --region us-east-1 --profile my-aws-profile --dry-run

Execute Rollback

Once you have verified the dry-run output, run the rollback script without the --dry-run flag:

Windows Command Prompt:

python migrate-to-filter-rollback.py --config migrate-config.json --region us-east-1

Windows Command Prompt with AWS Profile:

python migrate-to-filter-rollback.py --config migrate-config.json --region us-east-1 --profile my-aws-profile

Linux/macOS Terminal:

python3 migrate-to-filter-rollback.py --config migrate-config.json --region us-east-1

Linux/macOS Terminal with AWS Profile:

python3 migrate-to-filter-rollback.py --config migrate-config.json --region us-east-1 --profile my-aws-profile
What does the rollback script do?

Removes Created Resources:

  • Deletes the Event Source Mapping
  • Deletes the Lambda function
  • Deletes the CloudWatch Log Group
  • Deletes the local SQS queue and Dead Letter Queue

Reverts IAM Changes:

  • Removes local SQS permissions from the existing IAM policy
  • Removes CloudWatch Logs permissions from the existing IAM policy
  • Detaches the AWSLambdaBasicExecutionRole managed policy
  • Reverts the IAM Role trust policy to remove Lambda service

After rollback, update your Splunk AWS Add-on SQS-Based S3 Input to use the original cross-account SQS queue URL from your SAP ECS account.


Troubleshooting

Lambda Not Receiving Messages

  • Verify the Event Source Mapping is enabled in the Lambda console
  • Verify cross-account permissions are correctly configured for your IAM Role
  • Check the IAM Role trust policy includes lambda.amazonaws.com

Lambda Errors in CloudWatch

  • Check for “Access Denied” errors which indicate IAM permission issues
  • Verify the Lambda code ZIP file was uploaded correctly to S3
  • Check the handler name in the configuration matches the Lambda code

Messages Not Appearing in Local Queue

  • Review the Lambda CloudWatch logs for filtering decisions
  • Verify the include_filters patterns match your expected paths
  • Check the days_in_the_past setting is not filtering out your messages

Splunk Not Ingesting

  • Verify the local SQS queue URL is correctly configured in the Splunk input
  • Check the Splunk AWS Add-on internal logs for connection errors
  • Verify messages are present in the local SQS queue


Configure Native TA Index-Time Filtering

Introduction

Starting with version 0.0.3, the Splunk TA for SAP LogServ includes built-in index-time filtering that works inside Splunk itself, independently of the Lambda-based S3 event filtering configured above. You can use both approaches together for defense-in-depth filtering, or use the native TA filtering on its own.

Native TA Filtering vs. Lambda-based Filtering
  • Lambda-based filtering (configured above) filters S3 event notifications before they reach Splunk, reducing the number of SQS messages processed by the Splunk AWS Add-on.
  • Native TA filtering (configured below) filters events inside Splunk at index time via TRANSFORMS-based queue routing. Filtered events never consume Splunk license.

Both use the same clz_dir/clz_subdir pattern syntax. Using both together means events are filtered at the AWS level first and then again at the Splunk level, providing a safety net if either filter configuration is incomplete.


Open the Filters Tab

  1. In Splunk Web, open the Splunk TA for SAP LogServ app
  2. Go to Configuration → Filters

image


Set Your Filter Options

Enable Filtering

Check the Enable Filtering checkbox to activate index-time filtering.

Include Filters

Comma-separated patterns specifying which log types to include. Patterns use the format clz_dir/clz_subdir with fnmatch-style wildcards:

Pattern Meaning
*/* Include all log types (default)
hana/hanaaudit Include only HANA audit logs
linux/* Include all Linux log types
linux/messages, hana/* Include Linux messages and all HANA logs

This field cannot be empty when filtering is enabled. Use */* to include everything.

Exclude Filters

Comma-separated patterns for log types to exclude. Events matching any exclude pattern are dropped even if they also match an include pattern. Leave empty to exclude nothing.

Days in the Past

Whole number (0–3650) specifying the maximum age of events to index. Set to 0 to disable time-based filtering. Default: 7.

Save

Click Save. The TA validates your patterns and generates the necessary configuration files.


Deployment Server: Deploy to Forwarders

If you are running on a Deployment Server with Heavy Forwarders, additional steps are required after saving filters.

After saving, the Filters tab will display a “⚠ Deployment Server Detected” banner with a “Deploy to Forwarders” button.

image

Configure the Server Class (First Time Only)

  1. Go to Settings → Forwarder Management
  2. Find the SAP_LogServ_HeavyForwarders server class
  3. Click the three-dot menu → Edit agent assignment
  4. Add your Heavy Forwarder IP addresses or hostnames
  5. Save the agent assignment

Deploy

  1. Return to the Filters tab and click “Deploy to Forwarders”
  2. Confirm the deployment
  3. Wait for HFs to phone home (typically 30–60 seconds)
  4. Verify deployment status in Settings → Forwarder Management


Verify Native Filtering

After deployment, verify that native filtering is operating correctly:

`sap_logserv_idx_macro` | stats count by clz_dir, clz_subdir

You should see data only for log types matching your include patterns. See Configuring Filters for the full verification and troubleshooting guide.

Ended: AWS Setup Migrations

Ended: LogServ Data TA

LogServ UI App ↵

LogServ App Prerequisites

This page covers the prerequisites for the LogServ App (splunk_app_sap_logserv) — the React-based UI App that ships dashboards + the AI Assistant chat panel. For the Data TA’s prerequisites (CIM add-ons), see Data TA Prerequisites. The SAP data + AI Assistant audit indexes are auto-created by the Data TA when it installs on the indexer — see Indexes (auto-created on install).

Splunk Platform Requirements

  • Splunk Enterprise 9.4.3 or later, or Splunk Cloud Platform
  • The LogServ App ships as a single React bundle and uses @splunk/react-ui + @splunk/visualizations + @xyflow/react. Splunk 9.4.3+ is the minimum for the React component versions used.

AI Assistant Prerequisites

The LogServ App includes a built-in AI Assistant chat panel that dispatches predefined prompts (saved searches) against your data via the Splunk MCP Server. To use the AI Assistant — even with the LLM-driven path disabled in v0.0.5 — install:

Install on the same Search Head as the LogServ App. Cookie auth from the same Splunk Web session works by default; no bearer token required for HTTP-only Splunk. See Splunk MCP Setup for full configuration.

Recommended companion app

The Splunk AI Assistant is not a strict prerequisite for the LogServ App’s AI Assistant — the LogServ App uses only the core splunk_run_saved_search and splunk_run_query MCP tools, which work standalone against the Splunk MCP Server. However, App 200 is the typical co-install for the Splunk MCP Server and unlocks additional saia_*-prefixed MCP tools (e.g., SPL optimization helpers) that may be used in future LogServ releases. If you’re already deploying the Splunk MCP Server, installing App 200 alongside it follows Splunk’s documented setup pattern and avoids friction if a future LogServ release calls those tools.

v0.0.5 release: AI provider credentials are NOT required

The v0.0.5 release ships with the LLM-driven path disabled at compile time pending internal review. The predefined-prompt path (which dispatches saved searches via the Splunk MCP Server) is the only AI Assistant path active in v0.0.5, and it does NOT call any LLM provider. You do not need to configure an Anthropic / OpenAI / Azure / Bedrock credential to use the AI Assistant in v0.0.5. When the LLM-driven path is re-enabled in a future release, the Settings → Provider Credentials tab will become visible and one provider credential will be required for the free-form chat input. See Templates-only Build.

Next Steps

Installing the LogServ App

This page covers installing the LogServ App (splunk_app_sap_logserv). For the Data TA installation, see Installing the Data TA.

v0.0.5 release: AI Assistant LLM functionality intentionally disabled pending review

The v0.0.5 release of the LogServ App ships with the AI Assistant’s LLM-driven path disabled at compile time pending internal review. The published v0.0.5 tarball is the templates-only build variant — there is no separate “regular” v0.0.5 build. The predefined-prompt path + Splunk MCP Server integration + tool tiles + drill-down chips + audit log + 20 dashboards + Environment Topology view are all fully active. The free-form chat input, the model picker, the Power Mode toggle, and the Provider Credentials Settings tab are all hidden. See Templates-only Build for the build mechanism.

About the LogServ App

The LogServ App provides:

  • 20 React-based dashboards plus the Environment Topology view, organized as one top-level Environment Health landing page + four purpose-driven groups (Applications, Integration, Security, Platform). Built on @splunk/react-ui + @splunk/visualizations + @xyflow/react. See Dashboards Overview.
  • Built-in AI Assistant chat panel — predefined prompts + Splunk MCP integration + audit log. (LLM-driven path disabled in v0.0.5; canned-prompt path active.) See AI Assistant Overview.
  • Search-time field extractions for all SAP-specific sourcetypes (~176 directives across EXTRACT, EVAL, FIELDALIAS).
  • The sap_logserv_idx_macro macro for searching the LogServ index.
  • Built-in Download PNG button on every dashboard for html2canvas-based full-canvas image export.
  • Per-dashboard auto-refresh picker (Never / 30s / 1m / 5m / 15m / 30m / 1hr) with per-user-per-dashboard cadence persisted via Splunk KV Store.

The LogServ App contains no Python code, no REST handlers, and no data collection components. It is a React-based app focused entirely on the search-time experience and AI Assistant chat surface.

High Level Steps

Below are the high level steps for installing the LogServ App. Follow them in order.

Steps 4 and 5 are alternative paths — complete the one that matches your Splunk environment.

  1. Identify where to install the LogServ App based on your topology
  2. Install the Splunk MCP Server prerequisite (for AI Assistant)
  3. Download the LogServ App
  4. Install the LogServ App in Splunk Cloud (if applicable)
  5. Install the LogServ App in Splunk Enterprise (if applicable)
  6. Verify the installation
  7. Update the index macro (if using a custom index name)


1. Where to install

Your Topology Install the LogServ App On
Single instance The single Splunk instance (alongside the Data TA)
Distributed with on-prem SH The Search Head only
Distributed with Splunk Cloud The Splunk Cloud Search Head only

Important

  • The LogServ App is never installed on Heavy Forwarders or the Deployment Server.
  • For single-instance deployments, install both the Data TA and the LogServ App on the same instance. Splunk merges their configurations at runtime.

2. Install the Splunk MCP Server prerequisite

The AI Assistant requires the Splunk MCP Server (Splunkbase App 7931) v1.1.0 or later, installed on the same Search Head as the LogServ App. Install it via Splunk Web (Apps → Install app from file) or via CLI:

/opt/splunk/bin/splunk install app /path/to/splunk-mcp-server.tar.gz

After install, restart Splunkd. Cookie auth from the same Splunk Web session works by default; no bearer token configuration required for HTTP-only Splunk. See Splunk MCP Setup for full configuration including the optional bearer token for OAuth-strict environments.

v0.0.5 release: no AI provider credentials needed

Even with the LLM-driven path disabled, the AI Assistant’s predefined-prompt path requires the Splunk MCP Server to dispatch saved searches. Install the MCP Server. Do not configure any AI provider credential (Anthropic / OpenAI / Azure / Bedrock) — they are not used in v0.0.5 and their Settings tab is hidden.

3. Download the LogServ App

Download splunk_app_sap_logserv-0.0.5.0.tar.gz from the GitHub repository.

The published v0.0.5 tarball is the templates-only build variant (LLM-driven path disabled at compile time pending review). There is no separate “regular” tarball published in v0.0.5.

4. Install in Splunk Cloud

Install the LogServ App to your Splunk Cloud Search Head:

Note

The app installation workflow available to you in Splunk Web depends on your Splunk Cloud Platform Experience: Victoria or Classic. To find your Splunk Cloud Platform Experience, in Splunk Web, click Support & Services > About.

Classic Experience

Victoria Experience

5. Install in Splunk Enterprise

Install the LogServ App to your Splunk Enterprise Search Head:

5.a From the Splunk Web home screen, click the gear icon next to Apps.

5.b Click Install app from file.

5.c Locate the downloaded splunk_app_sap_logserv-0.0.5.0.tar.gz file and click Upload.

5.d If Splunk Enterprise prompts you to restart, do so.

5.e Verify that the app appears in the list of apps. You can also find it on the server at $SPLUNK_HOME/etc/apps/splunk_app_sap_logserv.

6. Verify installation

After installation, navigate to the LogServ App in Splunk Web. You should see the navigation bar with:

  • Environment Health (default landing page — cross-cutting operations view)
  • Topology (graph-based Environment Topology view)
  • Applications dropdown (5 dashboards: ABAP Network & Security, ABAP Operations, Work Process Performance, HANA Audit, HANA Trace)
  • Integration dropdown (5 dashboards: SAP Services, SAP Router, Cloud Connector, Web Dispatcher, Web and API Performance)
  • Security dropdown (3 dashboards: Network Perimeter, Cross-Stack Authentication, Change & Configuration Activity)
  • Platform dropdown (6 dashboards: Data Pipeline Overview, DNS Analytics, Linux, Windows, Proxy, Host Details)
  • ✦ AI Assistant button in the top-right of the nav bar (clicking it opens the chat panel)

If the dashboards show no data, verify that:

6.a The Data TA is installed and collecting data on your Heavy Forwarders (or single instance)

6.b The sap_logserv_idx_macro resolves to the correct index name

6.c Events exist in the index: run `sap_logserv_idx_macro` | stats count by sourcetype in the Search app

6.d If the AI Assistant button shows a setup wizard instead of an empty chat panel, the Splunk MCP Server prerequisite isn’t healthy — re-check the install in Step 2.

7. Update the index macro

If you used a custom index name (not sap_logserv_logs), update the macro:

7.a In Splunk Web, go to Settings > Advanced search > Search macros

7.b Set the app context to Splunk App for SAP LogServ

7.c Find sap_logserv_idx_macro and update its definition to index=<your_index_name>

Next Steps

Dashboards Overview

The LogServ App includes twenty-two React-based dashboards plus an Environment Topology view, organized as one top-level landing page and four purpose-driven navigation groups. The app is built on @splunk/react-ui + @splunk/visualizations + @xyflow/react and ships as a single React bundle. Requires Splunk 9.4.3 or later.

The top menu is:

Environment Health (default landing) · Topology · Applications ▼ · Integration ▼ · Security ▼ · Platform ▼ · AI Assistant · Search

Use this page as an index — click any dashboard below to see its full purpose, panel list, and interpretation guide on the corresponding category page.

Full Inventory

Start Here

Dashboard Purpose Key Data Sources
Environment Health Cross-cutting operations view of errors, security failures, and performance across the entire SAP landscape. Default landing page. All sourcetypes

Topology

Dashboard Purpose Key Data Sources
Environment Topology Interactive graph view of SAP systems, integration partners, and endpoints. Force-directed initial layout, self-derived IP→SID inventory, named saved layouts via KV Store, Live mode auto-refresh. sap:abap:gateway, sap:abap:icm, sap:hana:tracelogs, sap:saprouter, plus host inventory from linux_messages_syslog osquery events

Applications (SAP application runtime)

Dashboard Purpose Key Data Sources
ABAP Network & Security ICM traffic analysis, gateway monitoring, and ABAP audit events sap:abap:icm, sap:abap:gateway, sap:abap:audit
ABAP Operations ABAP runtime health, dispatcher status, work process activity, and system uptime sap:abap:dispatcher, sap:abap:enqueueserver, sap:abap:event, sap:abap:messageserver, sap:abap:sapstartsrv, sap:abap:workprocess
Work Process Performance SAP ABAP work process utilization, dispatcher health, and function-level activity sap:abap:workprocess, sap:abap:dispatcher
HANA Audit SAP HANA database audit events, security monitoring, user activity, risk-tiered events, and after-hours admin activity sap:hana:audit
HANA Trace SAP HANA database trace logs, component health, and error analysis sap:hana:tracelogs

Integration (how SAP connects to other systems)

Dashboard Purpose Key Data Sources
SAP Services sapstartsrv authentication, SSL/TLS failure analysis, and host agent health sap:sapstartsrv, sap:saphostexec
SAP Router SAP Router connection activity, error analysis, and network boundary monitoring sap:saprouter
Cloud Connector SAP Cloud Connector HTTP traffic, audit events, and access denied events sap:scc:audit, sap:scc:http_access
Web Dispatcher HTTP traffic analysis, response times, status codes, and client patterns sap:webdispatcher:access
Web and API Performance Four-stage request timing, response-time percentiles, TLS posture, cross-source error correlation sap:webdispatcher:access, sap:scc:http_access

Security (cross-source synthesis + compliance)

Dashboard Purpose Key Data Sources
Network Perimeter Unified network-boundary view: firewall drops (inbound), proxy outbound traffic, DNS resolution, and cross-source suspicious-activity correlation linux_secure, squid:access, isc:bind:query
Cross-Stack Authentication Unified authentication failure analysis across SAP, HANA, and Windows layers sap:sapstartsrv, sap:hana:audit, XmlWinEventLog
Change & Configuration Activity Cross-stack audit trail: HANA user/role/privilege changes, Windows account and group modifications, Linux sudo and user-management activity, with compliance-focused privileged and after-hours views sap:hana:audit, XmlWinEventLog, linux_messages_syslog

Platform (underlying infrastructure, ingest, and forensics)

Dashboard Purpose Key Data Sources
Data Pipeline Overview Ingest pipeline view: 5 KPIs, Sourcetype Summary table, host activity, and source-to-sourcetype link graph. Two tabs (Overview + Linked Graph). Dashboard-wide host filter in title row scopes every panel + the linked graph. All sourcetypes
DNS Analytics DNS query analysis, top resolvers, beaconing detection, and client activity isc:bind:query, isc:bind:network, isc:bind:transfer
Linux System & Security Linux OS events, SAP application activity, and firewall monitoring (with Top Drop Source surface) linux_messages_syslog, linux:cron, linux:warn, linux:sudolog, linux:slapd, linux_secure
Windows Events Windows operational health — event severity trends, top event codes, service state changes, PowerShell activity XmlWinEventLog
Proxy Analytics Squid proxy traffic, top domains by bandwidth, cache action distribution, client diversity squid:access
Host Details Per-host drill-down with Overview, Role Activity, and Sourcetype Mapping tabs. Title-row Multiselect lets you filter to one host, multiple hosts (host IN (...)), or All Hosts; role-specific panels auto-hide for hosts without that data. All sourcetypes (host-filtered)

Searching LogServ data

All dashboards use the sap_logserv_idx_macro macro to query the LogServ index. You can use this same macro in your own searches: `sap_logserv_idx_macro` | stats count by sourcetype

Cross-dashboard navigation

Every dashboard includes a Navigate to Dashboard dropdown and Go button (top-left) that preserves your selected time range when switching between dashboards.

In-dashboard help — the More Info button

Every dashboard also includes a More Info button at the top-right of the toolbar row. Clicking it opens this online documentation in a new browser tab, jumping directly to the section for the dashboard you’re looking at. For multi-tab dashboards (Data Pipeline Overview and Host Details) the button lives on every tab and always links back to the same dashboard’s section in the docs.

Per-dashboard auto-refresh

Each dashboard’s title row carries a Refresh picker (Never / 30s / 1m / 5m / 15m / 30m / 1hr) next to the time-range picker. The selection is per-user-per-dashboard — your choice on Environment Health doesn’t carry over to HANA Audit. State persists across browser sessions via Splunk KV Store.

Visual Style

All dashboards share a consistent “framed dark cards” visual language so that patterns are easy to recognize as you move between views:

  • Dark page background (#0d1117) with slightly lighter navy panel cards (#141b2d) outlined in cyan (#0877a6) — each visualization sits inside its own framed card with consistent 12 px spacing
  • KPI typography — large numeric headline with a small muted label; number color carries semantic meaning (white for neutral counts, red for errors/failures/denials, orange for warnings, teal for healthy/positive signals)
  • Standard colors — a single standard red (#dc4e41) is used across all error/failure signals so that “red means something went wrong” is unambiguous
  • Tables use a fixed dark header row with alternating row striping; clickable cells are highlighted cyan to indicate a drilldown affordance
  • Charts strip non-essential ink (no axis titles, no data labels, no progress bars) so the data shape is what your eye lands on

The implementation uses styled-components on top of @splunk/react-ui primitives. Compared to the v0.0.4.x Dashboard Studio v2 era, search-time field extractions are unchanged — what changed is purely the rendering tier.

KPI Sparklines

Most KPI panels display a small inline sparkline directly below the headline number. The sparkline visualizes the daily trend across the dashboard’s current time range, so you get both the cumulative total and the shape of how that total accumulated in a single glance. There is no separate “up/down by N” trend value — the sparkline alone carries the trend signal.

Sparklines come in a few flavors depending on what the KPI measures:

  • Count-based KPIs (e.g., Total Events, Auth Failures) — sparkline shows daily event count
  • Distinct-count KPIs (e.g., Active Hosts, Unique Components) — sparkline shows daily distinct count
  • Rate KPIs (e.g., HTTP Error Rate, Web Error Rate %) — sparkline shows daily error percentage
  • Volume KPIs (e.g., Total Volume, Total Bandwidth) — sparkline shows daily MB; headline shows formatted KB/MB/GB
  • Empty-safe wrap — when a KPI’s search returns zero events, the KPI displays 0 (rather than ###) with a flat-zero sparkline

Click-Through Drilldowns

Most KPIs, table rows, and many chart points are clickable:

  • Clicking a KPI opens a contextual destination — typically the related drill-down dashboard with the current time range preserved, or Splunk’s Search app with a pre-built SPL query for cross-cutting KPIs (e.g., the Environment Health “Total Errors” KPI runs an 11-sourcetype OR search across the estate).
  • Clicking a table row opens a filtered view — Host Details for host columns, the relevant specialist dashboard for sourcetype columns, or Splunk’s Search app with the row’s context spliced into a SPL filter.
  • Clicking a chart point typically opens the underlying search for that time bucket or series.
  • Every drill-down opens in a new browser tab with noopener,noreferrer security flags. Time-range query params (?earliest=...&latest=...) are preserved across the navigation so the destination loads at the source dashboard’s window.

AI Assistant Drill-Down Chips

Tool-result tiles in the AI Assistant’s right pane carry two drill-down chips in their actions slot, alongside the Clear button:

  • ↗ Dashboard — opens the related OOTB dashboard (one chip per related dashboard for prompts mapped to multiple). Sourced from the intent map’s dashboard field.
  • ↗ Run SPL — opens Splunk’s universal Search app with the tool’s SPL pre-populated and the dispatch’s exact earliest/latest pre-applied. The same chip renders alongside [→ saved_search] citations in the chat narrative on the left pane.

These chips connect the AI Assistant’s investigation flow back into the dashboards: a top-N finding tile leads directly to the relevant dashboard, OR to a raw-search drill-down at the same time window the AI just queried.

Download PNG

Every dashboard’s title-row toolbar includes a Download PNG button. The capture uses html2canvas to render the full dashboard DOM (including off-screen content) to a PNG file, so the saved image always covers the entire dashboard length — not just what’s visible in the viewport. Useful for sharing in slide decks, embedding in incident reports, or capturing a dashboard’s state at a specific moment for compliance evidence.

Environment Health

Why This Dashboard Matters

The Environment Health dashboard is a single-pane-of-glass operations view that aggregates the most critical signals from across the entire SAP landscape. Instead of switching between individual dashboards to piece together overall health, administrators can use this dashboard to immediately identify active errors, security failures, and performance degradation. Every panel links to the relevant detailed dashboard for investigation, making this the recommended starting point for daily operations monitoring and incident triage.

Environment Health is the default landing page when you open the LogServ App — it sits above the four category dropdowns in the top menu.

Panels

  • Total Errors – Aggregate count of errors across all monitored sourcetypes. Click to open a detailed search showing errors by category, sourcetype, affected hosts, and last-seen time.
  • HANA Failed Ops – Count of non-successful HANA audit operations (login failures, permission denials, DDL errors). Click to drill down to HANA Audit dashboard.
  • Auth Failures – Combined count of sapstartsrv authentication failures and HANA audit connection failures. Click to drill down to SAP Services dashboard.
  • Firewall Drops – Count of Linux firewall (iptables) drop events across all monitored hosts. Click to drill down to Linux dashboard.
  • Web Error Rate % – Percentage of Web Dispatcher requests returning 4xx/5xx status codes. Click to drill down to Web Dispatcher dashboard.
  • Beaconing Domains – Count of DNS domains exhibiting periodic query patterns that may indicate malware or C2 communication. Click to drill down to DNS Analytics dashboard.
  • ABAP Error Trend – Stacked column chart of daily ABAP errors by sub-source (Dispatcher, ICM, Gateway). Click to drill down to ABAP Security dashboard.
  • HANA Error Trend – Stacked column chart of daily HANA errors (Audit Failures, Trace Errors). Click to drill down to HANA Audit dashboard.
  • Security Error Trend – Stacked column chart of daily security errors (Auth Failures, Firewall Drops). Click to drill down to SAP Services dashboard.
  • Web/Network Error Trend – Stacked column chart of daily web/network errors (WebDisp 4xx/5xx, Router Errors, Proxy Denied). Click to drill down to Web Dispatcher dashboard.
  • Cloud Connector Error Trend – Stacked column chart of daily Cloud Connector HTTP errors (4xx Client, 5xx Server). Click to drill down to Cloud Connector dashboard.
  • OS/Infra Error Trend – Stacked column chart of daily Windows events by severity (High, Medium). Click to drill down to Windows dashboard.
  • Recent Critical Events – Table of the 20 most recent critical events (HANA audit failures, ABAP dispatcher FATAL, HANA trace fatal, auth failures, Windows high severity). Click any row to drill down to Host Details for that host.
  • Top Affected Hosts – Table of hosts ranked by total error count with breakdowns by category (ABAP, HANA, Services, Firewall, Other). Click any host to drill down to Host Details.
  • Web Dispatcher Response Time – Line chart of daily average response time trend for web dispatcher requests. Click to drill down to Web Dispatcher dashboard.
  • ICM Status Codes – Stacked column chart of HTTP status code categories (2xx/3xx/4xx/5xx) over time. Click to drill down to ABAP Security dashboard.
  • Data Pipeline – Events/Day – Average daily event volume and daily trend line for monitoring pipeline health and detecting ingestion gaps. Click to drill down to Overview dashboard.

v0.0.5.0 Drill-Down Behavior

Every KPI card, chart panel, and table row on this dashboard is clickable and opens its drill-down destination in a new browser tab with the source dashboard’s currently-selected time range pre-applied (?earliest=...&latest=...). The destination’s TimeRangeProvider parses the URL and hydrates its initial time-range from those params on mount, so a click from “Last 7 days” lands you in the destination at the same window. Two destination patterns:

  • Cross-dashboard drill-downs — most KPIs and charts navigate to the relevant React dashboard (/applications/hana-audit, /integration/cloud-connector, etc.).
  • Splunk Search drill-downs — the Total Errors KPI runs a cross-cutting 11-sourcetype OR query that no single dashboard owns; it opens Splunk’s universal Search app with the SPL pre-filled. Same time-range carry-through.

The dashboard’s title-row toolbar carries a Refresh picker so you can have the page tick continuously (Never / 30s / 1m / 5m / 15m / 30m / 1hr) for use as an operations wallboard.

What to Look For

  • Rising error trend – An upward slope in any error category chart indicates a worsening condition. Correlate the timing with recent changes, deployments, or infrastructure events. Each chart drills directly to the relevant dashboard for investigation.
  • Auth failure spikes – Sudden increases in the Auth Failures KPI or Security Error Trend may indicate brute-force login attempts, expired credentials, or misconfigured service accounts. Cross-reference with the HANA Audit and SAP Services dashboards.
  • HANA audit failures – Non-zero HANA Failed Ops always warrant investigation. Failed operations may indicate unauthorized access attempts, privilege escalation, or application misconfigurations.
  • Cloud Connector errors – A spike in the Cloud Connector Error Trend (especially 5xx Server errors) may indicate backend system unavailability, network issues between the SCC and SAP BTP, or certificate expiration.
  • Pipeline volume drops – A sudden drop in the Events/Day trend may indicate an SQS queue backup, S3 input failure, HF outage, or filter misconfiguration. Check the Data Pipeline Overview for per-host visibility.
  • Response time degradation – An increasing trend in Web Dispatcher response time often precedes user-facing performance complaints. Investigate ICM status codes for correlated 5xx errors.
  • Host concentration – If the Top Affected Hosts table shows errors concentrated on one or two hosts, those systems may have a localized issue (disk full, service crash, misconfiguration) rather than an environment-wide problem.

Environment Health

Environment Topology

Why This View Matters

The Environment Topology view answers a question every SAP administrator has: “what’s actually talking to what?” Traditional dashboards show events, errors, and aggregates per source — but the shape of how SAP systems integrate with each other (RFC partners, HANA tenant relationships, web traffic boundaries, cross-zone connections) is hidden in those tabular surfaces. This view materializes that shape as an interactive graph drawn from your existing log data — no new ingest, no CMDB integration, no tagging effort required.

Use it to:

  • Validate that your topology matches the as-designed architecture (every system shows up; no rogue partners are talking inbound).
  • Investigate a single SID’s call surface — who calls it, what does it call, where are the errors concentrated?
  • Onboard a new analyst by giving them a one-page picture of the SAP landscape before they start digging into individual dashboards.
  • Spot orphaned IPs / hosts that don’t belong to any known SID — these are typically misconfigured load balancers, scanners, or unauthorized integration partners.

This view replaces the manual hand-drawn architecture diagrams that SAP admin teams typically maintain in slides or wikis. Because it’s data-driven, it stays current as systems are added or retired.

What’s on Screen

The view consists of three regions:

  • Left toolbar — a layout name chip showing the active saved layout, a Refresh button (re-runs all 6 underlying SPL queries while preserving the saved positions), a Live mode toggle (auto-refresh every 30 seconds), and a Save / Load Layout dropdown for named-layout management.
  • Center canvas — the force-directed graph itself, rendered with @xyflow/react. Nodes are SAP systems (SID-tagged), HANA tenant DBs (cylinder icon), and integration partners (DB / web / external endpoints). Edges show observed call relationships, with edge thickness reflecting call volume.
  • Right sidebar — when a node is selected, four tabs surface that node’s per-selection context:
    • Calls/Hr — bar chart of hourly call volume into the selected node, hardcoded to the last 24 hours regardless of the global time-range picker (this chart’s purpose is “what’s happening RIGHT NOW on this node,” not “what happened over the picker’s window”).
    • Programs — top ABAP programs / RFC functions invoked on the node, sourced from the gateway and ICM logs.
    • Errors — categorized error breakdown for the node (HTTP 4xx/5xx, severity ERROR / CRITICAL / FATAL, gateway error_function, ICM transactions). Aggregated by sourcetype + error_kind.
    • Hosts — the host(s) backing the SID, with first-seen / last-seen / dc(sourcetype) per host.
  • Right cluster (toolbar) — a checkbox list of integration types lets you filter the visible nodes (RFC partners, HANA tenants, web endpoints, etc.).

When no node is selected, the right sidebar’s “Systems” panel shows the inventory roll-up: how many nodes / edges / total calls; how many endpoints are unambiguously SID-attributed.

Where the Data Comes From

The view assembles its node + edge inventory from a union of six SPL searches against existing sourcetypes — no new ingest is required:

Source What it contributes
sap:abap:gateway RFC peer/local IPs (P=<peer> / L=<local> fields). The local IP is canonical for IP→SID inventory because it always belongs to this host’s SID.
sap:abap:icm ICM peer/local IPs for HTTP-side traffic.
sap:hana:tracelogs HANA host + tenant SID extracted from the source path (/usr/sap/<HANA_SID>/HDB<inst>/<host>/trace/DB_<TENANT_SID>/). Yields the rich HANA-side topology including multi-tenant relationships.
sap:saprouter Peer hostname extracted from the parens after host <ip>/<service> (<resolved.host>). Provides the inbound edge for external partners.
linux_messages_syslog (osquery cpu_brand subset) Host inventory: CPU model, RAM, OS, region, AZ, EC2 instance ID. Used to enrich the host-level facts on the Hosts tab.
Default Splunk host field (cross-source fallback) Picks up hosts that aren’t surfaced in the above.

Self-Derived IP→SID Inventory

The “which IP belongs to which SID” mapping isn’t read from a CMDB or an admin-maintained lookup — it’s self-derived from the same logs you’re already ingesting. The mechanism is a multi-source union SPL with a mvcount(sids)=1 filter: a host whose multiple sourcetypes all agree on a single SAP SID is unambiguously attributed; otherwise it’s marked “unknown.”

Coverage depends on what your data exposes — unique hostname/IP appearances across multiple SAP sourcetypes attribute cleanly, while shared NAT IPs and external partners typically appear as “unknown” nodes (their SAP affiliation isn’t observable from logs alone, or they legitimately don’t belong to any single SID).

The inventory is extensible per-customer without new ingest: appending another union arm to the inventory SPL adds a new attribution source. Common future arms: HANA audit svc_host rex, ABAP program self-attribution from gateway L= cross-pivot, customer-specific CMDB lookup.

Saved Layouts

User-arranged graph layouts are persisted via Splunk KV Store collection logserv_topology_layouts. The schema (currently v4) carries:

  • Node positions — where you’ve manually placed each node on the canvas
  • Panel state — sidebar tab + width
  • Viewport — zoom level + pan offset (x, y)
  • Enabled integration types — which checkboxes are ticked in the right toolbar
  • Selected node — which node was active when you saved
  • Active right-sidebar tab — Calls / Programs / Errors / Hosts
  • Snap mode — whether new nodes snap to a grid

Layouts are per-user-named — you can save your own variants (“focus on XCQ”, “all DBs”, “external partners only”) and switch between them from the Load Layout dropdown. An admin saving a layout named default sets the layout other users see on first load.

Schema migration is in-memory: v1 / v2 / v3 records still load (the loadCachedLayout helper promotes them with the new fields undefined). When you save, the record is rewritten as v4.

Live Mode

The Live mode toggle in the left toolbar drives a 30-second auto-refresh of all 6 underlying SPL queries while preserving the saved positions. Useful for:

  • Watching a known integration come online during a deploy.
  • Confirming an integration is healthy by seeing its edge weight increase tick-over-tick.
  • Operations wallboard use — leave the topology view up on a dashboard monitor and watch the call-graph update in near-real-time.

Live mode coexists with the per-dashboard auto-refresh picker — currently both contribute additively to the refresh nonce. Consolidation is planned for a future release.

What to Look For

  • Unknown nodes with high edge weight — an “unknown” SID with hundreds of calls/hr is either a misconfigured CMDB entry on our side, OR an unauthorized partner that nobody documented. Both warrant investigation.
  • Asymmetric edge counts — an edge showing many calls one direction but very few the other usually means one side is the caller (RFC client) and the other is the callee (RFC server). Use the Programs tab to confirm which.
  • Orphan SIDs — a node with no edges is a system that’s running but not integrated with anything else. May be a quiet test system, or a system whose integration partners aren’t monitored.
  • Tenants drifting from their HANA system — multi-tenant HANA produces edges from the parent system to each tenant. Missing tenant edges indicate either tenant outage or audit-log gap.
  • Unexpected external partners — saprouter peer_hostname entries that don’t match the documented partner list are the strongest signal of an unauthorized external integration.
  • Host counts on the Hosts tab — a SID showing 3 hosts when you expected 2 may indicate a stale forwarder / VM that should have been decommissioned.

Notes

  • The view’s URL slug is /integration-topology (historical — the user-visible label was changed from “Integration Topology” to “Environment Topology”, but the URL slug + code identifiers stayed stable so existing bookmarks + saved-layout records remain valid).
  • Layout persistence works across browser tabs and across Splunk Web sessions — your KV Store records survive Splunkd restarts.
  • Layouts are saved per Splunk user — every KV Store record is keyed by <username>::<slug>, so different users have completely separate records and can’t overwrite each other’s layouts. If the same user edits the topology in two browser tabs concurrently, the tabs race each other and last-tab-save wins (multi-tab edit-collision protection is planned for a future release).

Environment Topology

Applications

The Applications category covers dashboards that monitor the SAP application runtime layer itself — the ABAP application server and the HANA database engine. These are the workloads SAP customers run every day, and dashboards here answer questions about work processes, dispatcher health, database audit trails, and diagnostic trace output.

v0.0.5.0 React refinements (apply to every dashboard in this category)

Every panel, KPI card, chart, and table row is clickable and opens its drill-down destination in a new browser tab with the source dashboard’s currently-selected time range pre-applied via ?earliest=...&latest=.... The destination’s TimeRangeProvider parses the URL on mount and hydrates its initial range. Every dashboard’s title-row toolbar carries a per-dashboard Refresh picker (Never / 30s / 1m / 5m / 15m / 30m / 1hr) with per-user-per-dashboard cadence persisted via Splunk KV Store, plus a Download PNG button (full-canvas capture via html2canvas).

Dashboard Purpose Key Data Sources
ABAP Network & Security ICM traffic analysis, gateway monitoring, and ABAP audit events sap:abap:icm, sap:abap:gateway, sap:abap:audit
ABAP Operations ABAP runtime health, dispatcher status, work process activity, and system uptime sap:abap:dispatcher, sap:abap:enqueueserver, sap:abap:event, sap:abap:messageserver, sap:abap:sapstartsrv, sap:abap:workprocess
Work Process Performance SAP ABAP work process utilization, dispatcher health, and function-level activity sap:abap:workprocess, sap:abap:dispatcher
HANA Audit SAP HANA database audit events, security monitoring, user activity, risk-tiered events, and after-hours admin activity sap:hana:audit
HANA Trace SAP HANA database trace logs, component health, and error analysis sap:hana:tracelogs

ABAP Network & Security

Why This Dashboard Matters

The ABAP Network & Security dashboard monitors the network-facing components of SAP ABAP systems. The Internet Communication Manager (ICM) handles all HTTP/HTTPS traffic into and out of ABAP, while the Gateway controls RFC connections between SAP systems. Together, these are the primary attack surface for ABAP-based landscapes and the first place where performance degradation manifests during connectivity issues.

Panels

  • Total Events – Aggregate event count across ICM, Gateway, and Audit sourcetypes
  • ICM Errors – Count of ICM events flagged as errors
  • Gateway Errors – Count of Gateway events with error details
  • Event Volume by Sourcetype – Daily trend of each sourcetype
  • ICM Status Codes Over Time – Stacked column chart of 2xx, 3xx, 4xx, and 5xx responses
  • ICM Peer Connections – Table of top peer IPs by request count with protocol details
  • ICM Request Types – Pie chart breakdown of ICM request types
  • Gateway Remote Hosts – Table of gateway connections by remote host, function, and service
  • Gateway Errors Over Time – Timeline of gateway error events
  • Activity by SID / Instance – Column chart showing event distribution across SAP systems

What to Look For

  • ICM 4xx/5xx spikes – A sudden increase in client errors (4xx) may indicate application misconfigurations or scanning activity. Server errors (5xx) suggest backend failures or resource exhaustion.
  • Unfamiliar peer IPs – New IP addresses appearing in the ICM Peer Connections table warrant investigation, especially if they generate high request volumes or connect using unusual protocols.
  • Gateway connections from unknown hosts – The Gateway Remote Hosts table should show expected RFC partners. Unknown remote hosts or unusual service names may indicate unauthorized access attempts.
  • Error rate trends – A gradually increasing error rate across days can signal infrastructure degradation (disk space, memory pressure, certificate expiry) before it becomes an outage.

ABAP Network & Security

ABAP Operations

Why This Dashboard Matters

The ABAP Operations dashboard provides runtime health monitoring for the SAP ABAP application layer. It covers the dispatcher (request routing), work processes (transaction execution), enqueue server (lock management), and system uptime. These components are the engine of every SAP ABAP system, and their health directly impacts user experience and business process execution.

Panels

  • Total Events – Aggregate event count across all ABAP operations sourcetypes
  • Active SIDs – Count of distinct SAP System IDs reporting data
  • Dispatcher Errors – Count of ERROR/FATAL severity dispatcher events
  • Event Volume by Sourcetype – Daily trend across all six sourcetypes
  • System Uptime (Latest) – Table showing the most recent uptime in days and hours per SID/instance
  • Dispatcher Severity Over Time – Stacked column chart of dispatcher log severity levels
  • Top Work Process Functions – Table of the most frequently called work process functions
  • Work Process Categories – Donut chart with bottom legend showing activity distribution across all 13 SAP work process categories (e.g., B = Database Interface, A = ABAP Processor, S = SQL / Statistics, M = Memory Management, X = RFC / CPIC). The donut uses the wp_category_name field populated by the app’s props.conf EVAL so every category code gets a friendly name.
  • Enqueue Lock Activity – Timeline of lock management operations
  • Activity by SID / Instance / Type – Event distribution across SAP systems and sourcetypes

What to Look For

  • Uptime resets – A system showing low uptime (hours instead of days) indicates a recent restart. Unexpected restarts may signal crashes, memory issues, or unplanned maintenance.
  • Dispatcher ERROR/FATAL increases – Rising error severity in the dispatcher indicates work process exhaustion, connection failures, or configuration problems that will soon impact users.
  • Work process category shifts – A sudden change in the distribution of work process categories (e.g., dialog processes being consumed by batch jobs) suggests resource contention.
  • Enqueue lock spikes – A sharp increase in lock operations can indicate application deadlocks, long-running transactions holding locks, or database performance issues causing lock wait times to increase.
  • SID/instance imbalance – If one system or instance is generating significantly more events than its peers, investigate whether it’s handling disproportionate load or experiencing issues.

ABAP Operations

Work Process Performance

Why This Dashboard Matters

Work Process Performance focuses specifically on the ABAP work process layer: the finite pool of processes that execute every dialog request, background job, and RFC call. When work processes are exhausted, users see “no free work process” errors; when a specific category is saturated, symptoms manifest differently (e.g., database interface saturation causes SQL timeouts; memory category issues cause roll-area swaps). This dashboard breaks activity down by the 13 SAP-standard dev_w* trace component categories so you can target remediation to the right subsystem.

Panels

  • Total WP Events – Aggregate event count from sap:abap:workprocess and sap:abap:dispatcher
  • Active SIDs – Count of distinct SAP System IDs reporting work process data
  • Dispatcher Errors – Count of dispatcher severity ERROR/FATAL events
  • Active WP Functions – Count of distinct work process function codes observed
  • Work Process Category Trend – Stacked column chart showing daily activity by category, with all 13 friendly-named codes (uses the shared wp_category_name field)
  • Category Distribution – Donut chart showing the overall category mix across the time range (same 13-code legend)
  • Top Work Process Functions – Horizontal bar of the most-seen function codes (top 15)
  • Dispatcher Severity Over Time – Stacked column of dispatcher severity levels over time
  • Activity by SID / Instance – Table ranking each SAP system/instance by event count, with drilldown to filter
  • Recent Dispatcher Errors – Table of the 25 most recent dispatcher ERROR/FATAL events with host and severity, with drilldown to the full event

What to Look For

  • Category saturation – If a single category (e.g., B = Database Interface or M = Memory Management) dominates the Category Distribution donut when it historically didn’t, that subsystem may be a bottleneck. Check the associated detail dashboards (HANA Audit/Trace for database; Linux/ABAP Operations for memory).
  • Trend shifts between categories – A gradual increase in N = Network (NI) or C = Communication events can indicate network degradation between the ABAP server and HANA/other SAP systems.
  • Dispatcher error bursts – Spikes in the Dispatcher Severity Over Time chart are often the first user-visible symptom of work process exhaustion. Correlate with Category Distribution to identify which category filled up first.
  • Instance imbalance – If one instance’s WP activity is an order of magnitude higher than its peers in the Activity by SID/Instance table, investigate whether that instance is handling disproportionate load or experiencing a local issue.
  • Function-code hotspots – The Top Work Process Functions bar surfaces which ABAP functions are running most often; an unexpected code at the top can indicate a runaway background job or custom transaction generating excessive traces.

Work Process Performance

HANA Audit

Why This Dashboard Matters

The HANA Audit dashboard is essential for database security compliance and threat detection. SAP HANA stores the most sensitive business data in the SAP landscape, and its audit trail captures every authentication attempt, privilege change, and administrative action. This dashboard transforms raw audit events into actionable security intelligence, supporting both real-time threat detection and compliance reporting.

Panels

  • Total Audit Events – Aggregate count of HANA audit log entries
  • Failed Operations – Count of audit events with non-successful status
  • Active Users – Count of distinct users generating audit activity
  • User Administration Activity Timeline – Tracks user administration actions (password resets, activations, deactivations) over time
  • Daily Security Health Score – Stacked chart combining daily event volume, active users, failures, and success rate
  • Audit Category Breakdown – Pie chart of audit event categories
  • Security Events Timeline – Shows failed operations, object modifications, and permission grants
  • Password Management Activities – Table of password-related audit events with user and IP details
  • Failed Operations by Host – Pie chart identifying which hosts generate the most failures
  • Top Users by Activity – Table ranking users by audit event count with failure counts
  • Activity by Hour of Day – Column chart showing successes vs. failures by hour for after-hours detection
  • Client IP Analysis – Table of connecting IPs with user counts, failures, and last-seen time
  • Risk-Tiered Event Timeline – Stacked column chart of daily events grouped by risk_level (critical / high / medium / low), so severe events never get hidden inside overall volume
  • After-Hours / Weekend Admin Activity – Table filtered to is_admin_user=true AND (is_business_hours=false OR weekend), surfacing privileged activity outside normal windows; click a row to drill down by user
  • High-Risk Events – Full-width table filtered to is_critical=true showing the recent top-50 critical audit events with user, client IP, status, action, object, and SQL statement; click a row to drill down by user

What to Look For

  • Failed operations from unusual IPs – Authentication failures from IP addresses not in your expected range may indicate brute-force attacks or credential stuffing attempts.
  • Critical events in the High-Risk Events table – Any non-empty row here warrants investigation. The SQL Statement column makes it straightforward to see whether a critical event was a privilege grant, a schema change, or a failed auth.
  • After-hours privileged activity – The After-Hours / Weekend Admin Activity table surfaces admin operations outside business hours; correlate with change management records to separate planned maintenance from suspicious activity.
  • Risk-tier shifts over time – A rising “critical” or “high” stack in the Risk-Tiered Event Timeline often precedes an incident. A sudden spike in “medium” may point to scripted workloads that need review.
  • Privilege escalation patterns – Watch for sequences where a user’s privileges are modified followed by unusual data access patterns. The Security Events Timeline surfaces GRANT and REVOKE operations.
  • Declining security health score – A downward trend in the daily success rate or an increase in failures signals growing security issues that need investigation.
  • Password management anomalies – Bulk password resets or resets for service accounts should be correlated with change management records.

HANA Audit

HANA Trace

Why This Dashboard Matters

The HANA Trace dashboard provides visibility into SAP HANA’s internal diagnostic trace system. Unlike audit logs that capture user actions, trace logs capture what the database engine itself is doing: memory management, query compilation, I/O operations, and internal errors. This is the primary tool for diagnosing HANA performance issues, stability problems, and understanding the root cause of database outages.

Panels

  • Total Trace Events – Aggregate count of trace log entries (click to drill down)
  • Errors / Fatal – Count of error and fatal severity events (click to drill down)
  • Unique Components – Number of distinct HANA components generating traces (click to drill down)
  • Trace Volume Over Time – Daily trend of total trace events
  • Trace Events by Severity – Stacked column chart showing info, warning, error, and fatal distributions
  • Top Components – Table of the most active HANA components with source file counts (excludes parsing-artifact values such as INFO, of, service: so that real components dominate)
  • Component by Severity (Top 10) – Stacked column chart showing which components produce the most errors (same noise filter applied)
  • Source File Hotspots – Table identifying specific source files generating the most trace entries (same noise filter applied)
  • Activity by SID / Instance – Event distribution across HANA systems
  • Recent Errors / Fatal Events – Table of the latest error and fatal trace events with component and source location

What to Look For

  • Error/fatal severity spikes – A sudden increase in error-level traces often precedes a HANA outage or performance degradation. Investigate the component and source file generating the errors.
  • Single component dominance – If one component suddenly generates significantly more traces than usual, it may indicate a runaway process, memory leak, or infinite loop within that subsystem.
  • New source files appearing – Trace entries from source files not seen before may indicate recently applied patches or code changes that are generating unexpected behavior.
  • SID/instance imbalance – Uneven trace volumes across instances of the same HANA system may indicate hardware issues, unbalanced workload distribution, or replication problems.
  • Persistent warning trends – Warnings that gradually increase over days often signal resource exhaustion (disk space, memory pools) that will eventually escalate to errors.

HANA Trace

Ended: Applications

Integration

The Integration category covers dashboards that monitor how SAP connects to other systems — host-level services (sapstartsrv, saphostexec), the SAP Router, the Cloud Connector (SAP BTP tunnel), and HTTP traffic through the Web Dispatcher. Use these dashboards to diagnose connectivity issues, auth failures at the integration boundary, and performance problems in the request path.

v0.0.5.0 React refinements (apply to every dashboard in this category)

Every panel, KPI card, chart, and table row is clickable and opens its drill-down destination in a new browser tab with the source dashboard’s currently-selected time range pre-applied via ?earliest=...&latest=.... The destination’s TimeRangeProvider parses the URL on mount and hydrates its initial range. Every dashboard’s title-row toolbar carries a per-dashboard Refresh picker (Never / 30s / 1m / 5m / 15m / 30m / 1hr) with per-user-per-dashboard cadence persisted via Splunk KV Store, plus a Download PNG button (full-canvas capture via html2canvas).

Dashboard Purpose Key Data Sources
SAP Services sapstartsrv authentication, SSL/TLS failure analysis, and host agent health sap:sapstartsrv, sap:saphostexec
SAP Router SAP Router connection activity, error analysis, and network boundary monitoring sap:saprouter
Cloud Connector SAP Cloud Connector HTTP traffic, audit events, and access denied events sap:scc:audit, sap:scc:http_access
Web Dispatcher HTTP traffic analysis, response times, status codes, and client patterns sap:webdispatcher:access
Web and API Performance Four-stage request timing, response-time percentiles, TLS posture, cross-source error correlation sap:webdispatcher:access, sap:scc:http_access

SAP Services

Why This Dashboard Matters

The SAP Services dashboard monitors two host-level services that are critical to SAP system availability: sapstartsrv (system startup and management) and the SAP Host Agent (host monitoring and management). These services operate at the infrastructure layer, below the application, and their failures can prevent SAP systems from starting or being managed remotely. The authentication story is front-and-center here – sapstartsrv is a common brute-force target, so the dashboard features an SSL-authentication failure panel as the main investigation surface. (SAP Router activity lives on its own SAP Router dashboard.)

Panels

  • Total Events – Aggregate event count across sapstartsrv and saphostexec
  • Auth Failures – Count of sapstartsrv authentication failures
  • SSL/TLS Events – Count of events involving SSL/TLS negotiation
  • Event Volume by Service (Normal vs Errors) – Full-width stacked column chart with four semantic series: sapstartsrv (normal), sapstartsrv (errors), saphostexec (normal), saphostexec (errors). Errors are defined per service: sapstartsrv = failed authentication events; saphostexec = severity ERROR/WARNING.
  • SSL Authentication Failure Sources – Full-width featured table aggregating SSL/TLS auth failures by source IP, with failure count, distinct user count, user list, first/last seen, and activity span (hours). Top 50 sources, row drilldown to the full event set for that IP.
  • Sapstartsrv Authentication Events – Table of authentication attempts showing user, IP, method, and result
  • Host Agent Severity – Pie chart of SAP Host Agent log severity distribution

What to Look For

  • Auth failure sources – The SSL Authentication Failure Sources table is the primary investigation surface. A single source IP with many distinct usernames is credential stuffing; many sources with a few usernames each is distributed brute-force; long activity spans indicate a persistent (not opportunistic) attacker.
  • Authentication failures from new IPs – Any new source IP appearing in the SSL Authentication Failure Sources table should be cross-referenced with your expected SAP admin network. Production sapstartsrv should rarely see failed authentications from unfamiliar ranges.
  • Error stack rising in the volume chart – If the error series (red) in the Event Volume chart grows relative to normal (blue/teal) series, something is actively going wrong. Correlate the spike timing with the host agent severity pie to determine which service is affected.
  • Host Agent ERROR severity – If the Host Agent severity distribution shifts toward ERROR, the host monitoring infrastructure may be degrading, which impacts central management capabilities.

SAP Services

SAP Router

Why This Dashboard Matters

SAP Router is the network gateway that sits between SAP systems and external networks, forwarding RFC and HTTP traffic across trust boundaries. Because it is exposed to the network, its logs are a primary audit trail for cross-boundary SAP traffic: every connection attempt, protocol error, and invalid-data event gets recorded. This dashboard separates the SAP Router signal from other services so that spikes in router errors (connectivity breakage, misconfigured routetab, or attempted protocol abuse) are visible on their own.

Panels

  • Total Router Events – Aggregate SAP Router event count
  • Router Errors – Count of router error events (click to drill down)
  • Invalid Data Events – Count of events where the router received malformed or unexpected protocol data – often indicates a misconfigured client or probing
  • Unique Peer IPs – Distinct count of peer IPs seen in the time window
  • Connection Actions Over Time – Stacked column chart of CONNECT, DISCONNECT, and error actions daily
  • Router Errors Over Time – Daily trend of router error events (area chart)
  • Top Peer IPs by Connection Volume – Horizontal bar chart of the most active peer IPs (top 15)
  • Return Code Distribution – Pie chart showing the distribution of router return codes (NiBuf, NiI, NiL, etc.) over the time range
  • Error Detail by Function – Table of the top 5 router error functions with return code, count, and sample peer detail; row drilldown
  • Recent Connection Log – Table of the 25 most recent CONNECT / DISCONNECT entries with peer IP, action, and return code; row drilldown

What to Look For

  • Invalid Data spikes – A sudden increase in the Invalid Data KPI suggests either a misconfigured client speaking a wrong protocol or a port scan / probe. Check the Top Peer IPs by Connection Volume for new entries.
  • Unfamiliar peers in the top-IPs bar – Known SAP-to-SAP traffic should come from a predictable IP set; new IPs in the top bar warrant investigation, especially if they generate high connection volume or show up only in the Error Detail by Function table.
  • Return code imbalance – The Return Code Distribution pie should be dominated by normal-case codes. A sudden growth of NIECONN_REFUSED or NIEHOST_UNKNOWN slices usually means a downstream SAP system is down or a routetab entry is wrong.
  • Rising error trend – An upward slope in Router Errors Over Time can precede an outage. Cross-reference with ABAP Operations (dispatcher errors) and Cloud Connector (for hybrid flows) to see whether the root cause is local to the router or broader.
  • Concentrated errors by function – If one row of the Error Detail by Function table accumulates most errors, that function is the specific RFC/HTTP path where the problem lives – a much narrower investigation scope than “the router is broken”.

SAP Router

Cloud Connector

Why This Dashboard Matters

The Cloud Connector dashboard monitors SAP Cloud Connector (SCC), which provides secure tunneled connectivity between on-premise SAP systems and SAP BTP (Business Technology Platform) cloud services. As the bridge between on-premise and cloud, the Cloud Connector’s health directly impacts hybrid integration scenarios, Fiori apps, and cloud-based analytics that depend on on-premise data access.

Panels

  • Total Requests – Count of HTTP requests processed by the Cloud Connector
  • HTTP Error Rate – Percentage of HTTP requests returning 4xx or 5xx status codes (scoped name clarifies that this is HTTP-only, not audit-log errors)
  • Audit Events – Count of Cloud Connector audit log entries
  • Access Denied Events – Count of sap:scc:audit entries with scc_audit_type="ACCESS_DENIED" (click to drill down to the matching events)
  • Request Volume Over Time – Daily HTTP request trend
  • Status Code Distribution – Stacked column chart of 2xx, 3xx, 4xx, and 5xx responses
  • Top URIs by Request Count – Table of the most requested URIs with average response time and total bytes
  • Average Response Time – Line chart trending response time over time
  • HTTP Methods – Pie chart breakdown of request methods
  • Top Clients – Table of the most active client IPs with request counts and unique URI counts
  • Cloud Connector Audit Log – Table of recent audit events with type and account details

What to Look For

  • HTTP Error Rate increases – A rising error rate indicates connectivity issues between on-premise systems and the cloud. Check the Status Code Distribution for whether errors are client-side (4xx) or server-side (5xx).
  • Access Denied events – A non-zero Access Denied KPI means the BTP side actively rejected a request – either a misconfigured subaccount binding, an expired certificate, or an unauthorized access attempt. Click the KPI to see which accounts/URIs are being denied.
  • Response time degradation – Gradually increasing response times suggest bandwidth constraints, backend system slowdowns, or Cloud Connector resource exhaustion. Sudden spikes may indicate outages.
  • Unusual URIs – Requests to unexpected URI paths in the Top URIs table may indicate scanning or misconfigured cloud applications attempting to access unauthorized resources.
  • Audit events indicating config changes – Cloud Connector audit entries for configuration modifications should correlate with approved change windows. Unexpected changes may indicate unauthorized access.
  • New client IPs – The Cloud Connector should only receive traffic from expected BTP subaccounts. New client IPs may indicate unauthorized access attempts or misconfigured routing.

Cloud Connector

Web Dispatcher

Why This Dashboard Matters

The Web Dispatcher dashboard analyzes HTTP traffic flowing through the SAP Web Dispatcher, which acts as the reverse proxy and load balancer for SAP web applications including Fiori, WebGUI, and custom web services. As the front door for all web-based SAP access, the Web Dispatcher’s traffic patterns reveal both performance issues and potential security threats. For deeper per-request timing analysis (four-stage breakdown, percentiles, TLS posture), see Web and API Performance.

Panels

  • Total Requests – Aggregate count of HTTP requests processed by the Web Dispatcher
  • Error Rate – Percentage of requests returning 4xx or 5xx status codes
  • Avg Response Time – Average response time across all requests in milliseconds
  • Traffic by Status Code – Stacked column chart showing 2xx, 3xx, 4xx, and 5xx response distributions
  • Response Time Trend – Response time patterns over time
  • Traffic by HTTP Method – Breakdown of GET, POST, PUT, DELETE, etc.
  • Top 5 Slowest Pages – Bar chart of the slowest responding URLs
  • Top Client IPs by Traffic – Bubble chart showing the most active client IP addresses
  • Request Volume Over Time – Daily request trend line
  • Top URIs by Request Count – Table of the most accessed URIs with average response time and unique client counts
  • Recent Errors (4xx/5xx) – Table of the latest error events with client IP, method, URI, and status code

What to Look For

  • 4xx/5xx status code spikes – Client errors (4xx) may indicate broken links, misconfigured apps, or scanning activity. Server errors (5xx) signal backend ABAP or Java system failures.
  • Response time degradation – Slow response times impact user experience. Correlate with the Top 5 Slowest Pages to identify which applications are affected, then investigate backend system load.
  • Unusual HTTP methods – PUT, DELETE, or PATCH requests where only GET/POST are expected may indicate API abuse or vulnerability exploitation.
  • Concentrated client IP traffic – A single IP generating disproportionate traffic in the bubble chart may be a bot, scraper, or denial-of-service source.
  • TLS version distribution – If your data includes TLS fields, watch for clients using deprecated TLS versions (1.0, 1.1) that may need to be blocked for compliance.

Web Dispatcher

Web and API Performance

Why This Dashboard Matters

The Web Dispatcher dashboard answers “what’s the traffic doing?” – volume, status codes, top URIs. Web and API Performance answers the next question: “why does it feel slow or unreliable?” It exposes the per-request timing stages that the sap:webdispatcher:access sourcetype records (dt1 receive / dt2 handler / dt3 response / dt4 send), combines response-time percentiles across Web Dispatcher and Cloud Connector HTTP traffic, and correlates the HTTP error rate with the Cloud Connector auth failure rate so that you can see whether a spike in user-visible failures is actually a backend-auth problem. It also surfaces TLS posture (version and cipher suite distributions) – data already extracted from Web Dispatcher but previously unused – so that cipher-suite drift or legacy TLS traffic becomes visible.

Panels

  • Total Requests – Aggregate HTTP request count across Web Dispatcher and Cloud Connector
  • HTTP Error Rate – Percentage of requests returning 4xx or 5xx status (combined across both sources); click to open the matching events
  • Avg Response Time – Average response_time_ms across both sources, in milliseconds
  • Auth Failures – Count of requests rejected for authentication reasons: Web Dispatcher status IN (401, 403) OR Cloud Connector (status IN (401, 403)) OR is_authenticated="false"; click to drill down
  • Unique URLs – Distinct count of URIs seen across both sources
  • Four-Stage Request Timing Breakdown – Full-width stacked column chart showing the average milliseconds a request spends in each of the Web Dispatcher’s four internal stages per day: dt1 receive (blue), dt2 handler (light cyan), dt3 response (orange), dt4 send (red). The stack composition tells you where time is being spent; the total stack height is the average end-to-end response time.
  • Response Time Percentiles Over Time – Daily p50 / p95 / p99 of response_time_ms (line chart, three series). p99 climbing while p50 stays flat is the classic tail-latency signal.
  • Top Slow URIs by Avg Response Time – Table of the 20 slowest URIs ranked by average response time, with source (WebDisp or CC), event count, avg ms, p95 ms, and error count. Row drilldown opens the events for that URI.
  • HTTP Error Rate vs Cloud Connector Auth Failure Rate – Full-width line chart overlaying two series: the overall HTTP error rate (4xx/5xx across both sources, red) and the Cloud Connector auth failure rate (401/403/anonymous, scoped to CC’s own denominator, orange). Use this to answer “are the user-visible errors actually auth failures at the backend?”
  • TLS Version Distribution – Column chart with color-coded series: TLS 1.0 red, 1.1 orange, 1.2 yellow, 1.3 teal – reinforcing that older versions are the security concern
  • TLS Cipher Suite Distribution (Top 10) – Horizontal bar chart of the 10 most-used cipher suites, with cyan accent
  • Top Slow Clients by p95 Response Time – Full-width table of the 20 client IPs experiencing the highest p95 latency, with event count, avg ms, p95 ms, and distinct URI count. Row drilldown opens the events for that client IP.
  • Recent 500-Level Errors – Full-width table of the 25 most recent 5xx events with time, source, host, client IP, method, URI, status, and response time. Row drilldown opens the full raw event.

What to Look For

  • Stage imbalance in the timing breakdown – In a healthy backend, dt2 (handler) dominates the stack because that’s where the ABAP/JVM work happens. If dt3 (response) or dt4 (send) suddenly grow relative to dt2, the bottleneck is likely on the response-formation or network-egress side rather than backend processing.
  • Tail latency growth (p99) – p99 drifting up while p50 stays flat means a small number of requests are getting dramatically slower – often database contention or garbage-collection pauses. Correlate with Top Slow URIs to identify which paths are affected.
  • Error-rate and auth-failure-rate correlation – When the two lines in the correlation chart track together, your HTTP error spikes are driven by backend auth rejection. When they diverge (HTTP errors up, auth failures flat), the cause is elsewhere (backend unavailable, 500s, timeouts).
  • Legacy TLS traffic – Any non-trivial slice of TLS 1.0 or 1.1 in the TLS Version Distribution is a compliance concern. Correlate the TLS Cipher Suite Distribution to see whether legacy ciphers are still being negotiated.
  • Slow clients vs slow URIs – The two “Top Slow” tables answer different questions. A single slow client with diverse URIs suggests network-path problem between that client and your infrastructure; a single slow URI across many clients points at a backend issue with that specific endpoint.
  • Concentrated 500 errors – Recurring entries in the Recent 500-Level Errors table from a single host, URI, or client IP narrow the investigation scope immediately.

Web and API Performance

Ended: Integration

Security

The Security category contains three cross-source synthesis dashboards designed for security posture analysis and compliance conversations. Each unifies signals that would otherwise live scattered across individual Applications or Platform dashboards, and adds a cross-source correlation panel that surfaces what no single-source dashboard can show.

v0.0.5.0 React refinements (apply to every dashboard in this category)

Every panel, KPI card, chart, and table row is clickable and opens its drill-down destination in a new browser tab with the source dashboard’s currently-selected time range pre-applied via ?earliest=...&latest=.... The destination’s TimeRangeProvider parses the URL on mount and hydrates its initial range. Every dashboard’s title-row toolbar carries a per-dashboard Refresh picker (Never / 30s / 1m / 5m / 15m / 30m / 1hr) with per-user-per-dashboard cadence persisted via Splunk KV Store, plus a Download PNG button (full-canvas capture via html2canvas).

Two compliance-focused exceptions: the After-Hours Privileged Changes and Recent Privileged Changes tables on the Change & Configuration Activity dashboard intentionally have no row drill-downs — clicking through to raw events from a compliance audit-trail report would pollute the trail with the reviewer’s own search activity in subsequent compliance reports. Per-source operational tables on the same dashboard DO get drill-downs.

Dashboard Purpose Key Data Sources
Network Perimeter Unified network-boundary view: firewall drops (inbound), proxy outbound traffic, DNS resolution, and cross-source suspicious-activity correlation linux_secure, squid:access, isc:bind:query
Cross-Stack Authentication Unified authentication failure analysis across SAP, HANA, and Windows layers sap:sapstartsrv, sap:hana:audit, XmlWinEventLog
Change & Configuration Activity Cross-stack audit trail: HANA user/role/privilege changes, Windows account and group modifications, Linux sudo and user-management activity, with compliance-focused privileged and after-hours views sap:hana:audit, XmlWinEventLog, linux_messages_syslog

Network Perimeter

Why This Dashboard Matters

Firewall drops, proxy traffic, and DNS queries are three different lenses on the same underlying question: what is crossing the network boundary and should it be there? Each lens lives on its own operational dashboard (Linux, Proxy Analytics, DNS Analytics), but an attacker rarely limits themselves to a single surface – a compromised host often shows up in multiple signals simultaneously, and that correlation is where the security value lives. Network Perimeter synthesizes the three sources into a single view: inbound rejections (firewall), outbound flow (proxy), and resolution activity (DNS), with a dedicated cross-source panel that ranks hosts by combined suspicious-signal score. Use it as the first stop for “is our network perimeter healthy and clean?”

Panels

  • Firewall Drops – Count of kernel firewall IN_DROP events from linux_secure
  • Proxy Requests – Count of HTTP requests handled by Squid
  • DNS Queries – Count of DNS queries from isc:bind:query
  • Beaconing Domains (red) – Domains exhibiting periodic query patterns (low variance in inter-query interval) – candidate C2 channels
  • Denied Requests (red) – Count of proxy requests with status=403 or vendor_action="TCP_DENIED"; click to drill down
  • Outbound Bandwidth – sum(bytes_out) across all proxy requests, formatted KB/MB/GB
  • Perimeter Activity Over Time – Full-width multi-line chart (log-scale y-axis) showing daily counts of all three sources on a single timeline. Log scale keeps Firewall Drops and Proxy Requests visible alongside the much larger DNS Queries volume; simultaneous spikes across two or three lines are the correlation signal to watch for.
  • Top Blocked Source IPs – Table of the 20 source IPs most rejected by the firewall, with unique target count and protocols seen; row drilldown to the matching events
  • Top Blocked Destination Ports – Table of the 20 destination ports most targeted by rejected traffic, grouped by port + protocol
  • Firewall Drops by Protocol – Stacked column showing daily IN_DROP events split by TCP / UDP / ICMP. Protocol shifts are often the clearest signal (ICMP spikes = ping flood / recon; UDP spikes = DNS amplification / port scan; TCP spikes = SYN-style port scan)
  • Proxy Denied Traffic Over Time – Area chart of daily denied proxy requests – the outbound complement to firewall drops (inbound)
  • Top Outbound Domains by Volume & Bytes – Table of the 20 destination domains receiving the most outbound traffic, ranked by bytes, with request count and unique client count. Row drilldown opens the events for that domain.
  • DNS Query Type Distribution – Donut of query-type mix (A / AAAA / PTR / TXT / MX / other). High TXT or MX volume from non-mail hosts is a DNS-tunneling / exfiltration indicator.
  • Top Queried Domains – Full-width table of the 30 most queried domains with unique-client count and per-domain %TXT and %MX ratios for quick anomaly spotting. Row drilldown opens DNS query events for that domain.
  • Suspicious Activity Indicator – Full-width cross-source table of internal hosts appearing in both beaconing DNS queries and denied proxy requests, ranked by signal score (beacon_domains × 3 + denied_requests). Columns: Host, Beaconing Domains, Beaconing Queries, Denied Proxy Requests, Signal Score. Row drilldown opens both DNS and proxy events for that host.

What to Look For

  • Correlated spikes across sources – The Perimeter Activity Over Time chart is the quickest read on “is something happening right now?” Watch for days when two or three of the lines spike together – that’s usually an active event (port scan, active C2, exfiltration window) rather than baseline drift.
  • Protocol shifts in firewall drops – The stacked column by protocol is more diagnostic than raw drop counts. A normally TCP-heavy mix suddenly showing large UDP or ICMP bands signals a different attacker technique.
  • Top Blocked Source IPs concentration – A single source IP generating the overwhelming majority of drops is either a persistent attacker or a misconfigured internal host. Either way, the action is the same: investigate that specific IP.
  • TXT-heavy queries to a single domain – A domain in Top Queried Domains with a high %TXT ratio is a DNS-tunneling pattern. Cross-reference the Suspicious Activity Indicator to see which hosts are issuing those queries.
  • Hosts in the Suspicious Activity table – Any non-empty row here is worth investigating. A host showing up in both beaconing DNS and denied proxy is strong evidence of compromise – the DNS lookups suggest malware C2, and the denied proxy requests suggest the same malware trying to reach blocked destinations.
  • Outbound Bandwidth spikes – A sudden rise in the Outbound Bandwidth KPI that isn’t matched by a proportional rise in Proxy Requests means individual transfers are getting larger – often the signature of an exfiltration event.

Network Perimeter

Cross-Stack Authentication

Why This Dashboard Matters

Authentication failures are usually investigated one layer at a time – someone looks at HANA audit logs, then switches to Windows Event Log, then opens the SAP services dashboard. An attacker who probes multiple layers at once is therefore hard to spot. Cross-Stack Authentication unifies the failure signal across the three layers that matter in an SAP landscape – SAP sapstartsrv, HANA audit, and Windows Security Event Log – so that a single pane shows the total, the per-layer split, and the source IPs and users in common across layers. Use it as the first stop when you suspect credential-based attacks or widespread misconfiguration after a password rotation.

Panels

  • Total Auth Failures – Aggregate failed authentication count across all three layers (click to drill down)
  • SAP Auth Failures – Count of sapstartsrv authentication failures (click to drill down)
  • HANA Auth Failures – Count of HANA audit events where the connection/authentication was rejected (click to drill down)
  • Windows Auth Failures – Count of Windows XmlWinEventLog:Security events corresponding to logon failures (click to drill down)
  • Auth Failures Over Time by Layer – Stacked column chart showing daily totals per layer (SAP / HANA / Windows) so correlated spikes across layers are visible at a glance
  • Top Users by Auth Failures – Horizontal bar of the 15 usernames with the most failures, summed across all layers
  • Auth Failure Source IPs – Table of the top 20 source IPs with failure counts and per-layer breakdown; row drilldown to the matching events
  • HANA Auth Activity by User – Table of the top HANA users by failed-auth count with client IP and last-seen time; row drilldown
  • Recent Windows Auth Failures – Table of the 25 most recent Windows logon-failure events with user, source workstation, and logon type; row drilldown
  • Recent SAP Auth Failures – Table of the 25 most recent sapstartsrv failed-auth events with user, source IP, and method; row drilldown

What to Look For

  • Correlated spikes across layers – If the stacked Auth Failures Over Time chart shows all three layers ramping simultaneously, that’s almost always a network-level attack (password spray, credential stuffing) rather than a local misconfiguration. Investigate source IPs immediately.
  • Single source IP across layers – The Auth Failure Source IPs table makes it obvious when one IP is failing against SAP, HANA, AND Windows. That’s the hallmark of a targeted attack rather than an expired-password incident.
  • High user failure count concentrated on service accounts – Service accounts (sapadm, sapservice accounts, DBADMIN-style) with large failure counts suggest either a recently rotated password that wasn’t updated downstream, or an attacker trying to abuse a high-privilege account.
  • Asymmetry between layers – Many HANA failures but zero SAP / Windows failures usually indicates an application-layer issue (bad connection string, expired JDBC cert). Asymmetry the other way (many Windows failures, no HANA failures) often points to a domain-level issue.
  • After a password rotation – Expect a short burst of failures across one or more layers immediately after a change. If the burst persists beyond the rotation window, some downstream system is still using the old credential.

Cross-Stack Authentication

Change & Configuration Activity

Why This Dashboard Matters

Compliance conversations (SOX, PCI, internal change management) all require evidence that configuration changes are (1) authorized, (2) attributable to a specific operator, and (3) happening in approved maintenance windows. That evidence lives scattered across three audit trails: HANA audit logs (user/role/privilege/password operations and DDL), Windows Security Event Log (account and group modifications), and Linux syslog (sudo commands plus useradd/usermod/userdel/passwd events). This dashboard unifies the three into a single audit trail with a consistent operator column and category taxonomy, plus two compliance-focused “recent” tables: one filtered to privileged actions, one filtered to after-hours activity.

Panels

  • Total Change Events – Aggregate count of change events across all three sources
  • User Account Changes – Count of user-management actions (HANA User Management/User Creation/User Deletion; Windows EventCodes 4720/4722/4725/4726/4738/4781; Linux useradd/usermod/userdel)
  • Permission Grants (red) – Count of privilege/group-membership grants (HANA Permission Grant; Windows EventCodes 4728/4732/4756 – “added to group”)
  • Password Events – Count of password changes and resets (HANA Password Management/Password Reset; Windows EventCode 4724; Linux passwd)
  • After-Hours Changes (red) – Count of change events occurring outside business hours (weekday 7am-7pm) or on weekends; HANA events use the pre-computed is_business_hours / is_weekend flags, other sources compute from _time
  • Unique Operators – Distinct count of source-prefixed operator identities (e.g., HANA:XCPADM, Windows:domain\admin, Linux:ops-user)
  • Change Activity Over Time – Full-width stacked column by day, series split by source (HANA / Windows / Linux). Same-day spikes across two or three series often line up with maintenance windows; isolated spikes in one source worth investigating.
  • Change Events by Category – Donut showing the category mix: Permission Grant, Permission Revoke, User Management, Password Change, Group Membership, Account Status, Sudo Command, DDL / Config, Other.
  • Top Operators by Change Count – Horizontal bar chart of the 15 operators generating the most change events, with source-prefixed identities so operator activity is clearly scoped to each system.
  • HANA Audit – Change Events – Full-width table of the 50 most recent HANA user/role/privilege/password/DDL actions with Operator, Target, Category, Action, Status, Host.
  • Windows – Account & Group Modifications – Full-width table of the 50 most recent Windows Security events across all 15 canonical account/group EventCodes, with human-readable Description column derived from EventCode.
  • Linux – Sudo & Command Activity – Full-width table of the 50 most recent sudo commands + useradd/usermod/userdel/groupadd/groupmod/groupdel/passwd activity, with Operator (extracted from sudo prefix or PAM (user) pattern) and Command.
  • Recent Privileged Changes (Top 25, Compliance Focus) – Full-width table filtered to the highest-risk subset: HANA Permission Grants + User Creations + Audit Policy changes; Windows account creation/enable/password-reset + local-group additions; Linux useradd/visudo/admin-group modifications. This is the “who gave themselves or others more access” report.
  • Recent After-Hours Changes (Top 25) – Full-width table of any change event filtered to is_after_hours=1. This is the “who was working outside the change window” report – high compliance value.

What to Look For

  • Rows in the Privileged Changes table with unfamiliar operators – The “headline” compliance question. A permission grant or group-addition you don’t recognize is the first thing to investigate.
  • After-Hours activity on business days – The After-Hours table surfaces all outside-window activity. Weekend entries are often planned maintenance; weekday late-night or early-morning entries warrant a check against your change tickets.
  • Single operator dominating the Top Operators bar – One identity generating most changes can be legitimate (an admin performing a large rollout) or concerning (an account being abused). The source prefix tells you which system to look at first.
  • Category-mix drift – If the Change Events by Category donut suddenly shows a large “Permission Grant” slice where it’s historically been minor, someone has been handing out privileges. Check the HANA Audit table for details.
  • Source asymmetry – The stacked column should show all three sources over time. If one source goes silent, it’s likely a logging-pipeline issue rather than “no changes happened”. Correlate with Data Pipeline Overview.
  • Linux sudo commands starting with useradd/usermod/visudo/passwd – These are the Linux equivalent of admin changes; they show up in both the Linux table and the Privileged Changes table for visibility.

Compliance-focused exception: no row drill-downs on the After-Hours and Privileged Changes tables

Two compliance-focused tables on this dashboard intentionally have no row drill-downs — the After-Hours Changes and Recent Privileged Changes tables. Clicking through to raw events from a compliance audit-trail report would pollute the trail with the reviewer’s own search activity in subsequent compliance reports. Per-source operational tables on the same dashboard (HANA Audit, Windows Account & Group Modifications, Linux Sudo & Command Activity) DO get drill-downs.

Change & Configuration Activity

Ended: Security

Platform

The Platform category covers the infrastructure underneath SAP — data pipeline health (Splunk ingest), network services (DNS, Proxy), host operating systems (Linux, Windows), and a per-host forensic drill-down. These dashboards answer questions about the underlying systems and the data-collection pipeline itself rather than the SAP workloads on top. The graph-based Environment Topology view is its own top-level entry in the app nav, not a Platform-group dashboard.

v0.0.5.0 React refinements (apply to every dashboard in this category)

Every panel, KPI card, chart, and table row is clickable and opens its drill-down destination in a new browser tab with the source dashboard’s currently-selected time range pre-applied via ?earliest=...&latest=.... The destination’s TimeRangeProvider parses the URL on mount and hydrates its initial range. Every dashboard’s title-row toolbar carries a per-dashboard Refresh picker (Never / 30s / 1m / 5m / 15m / 30m / 1hr) with per-user-per-dashboard cadence persisted via Splunk KV Store, plus a Download PNG button (full-canvas capture via html2canvas).

Dashboard Purpose Key Data Sources
Data Pipeline Overview Ingest pipeline view: 5 KPIs, Sourcetype Summary table, host activity, and source-to-sourcetype link graph (in second tab). Two tabs (Overview + Linked Graph). Dashboard-wide host filter in title row scopes every panel + the linked graph. All sourcetypes
DNS Analytics DNS query analysis, top resolvers, beaconing detection, and client activity isc:bind:query, isc:bind:network, isc:bind:transfer
Linux System & Security Linux OS events, SAP application activity, and firewall monitoring (with Top Drop Source surface) linux_messages_syslog, linux:cron, linux:warn, linux:sudolog, linux:slapd, linux_secure
Windows Events Windows operational health — event severity trends, top event codes, service state changes, PowerShell activity XmlWinEventLog
Proxy Analytics Squid proxy traffic, top domains by bandwidth, cache action distribution, client diversity squid:access
Host Details Per-host drill-down with Overview, Role Activity, and Sourcetype Mapping tabs. Title-row Multiselect filter to one host, multiple hosts (host IN (...)), or All Hosts; role-specific panels auto-hide for hosts without that data. All sourcetypes (host-filtered)

Data Pipeline Overview

Why This Dashboard Matters

The Data Pipeline Overview is your single pane of glass for the entire SAP LogServ ingestion pipeline. It answers the most fundamental question: is data flowing from all expected hosts and sourcetypes? In a distributed SAP landscape with multiple SIDs, instances, and log types, a gap in data collection can go unnoticed for days without centralized visibility. This dashboard has two tabs: Overview for the operational KPI/table view, and Linked Graph for the full-width source-to-sourcetype mapping.

The title row carries a dashboard-wide host filter: a Multiselect (with filter input + Select All Matches) plus a Top-N picker that scopes EVERY panel on BOTH tabs to the chosen hosts. The filter splices a host="X" (1 host) or host IN (...) (2+ hosts) clause into all 4 KPIs + their sparklines, the Events Over Time chart, the Sourcetype Summary table, the Host Latest Activity table, AND the linked-graph view on the second tab. Top N kicks in only when zero specific hosts are selected.

Tab 1 – Overview

  • Total Events – Aggregate event count across all LogServ sourcetypes
  • Total Sourcetypes – Count of distinct sourcetypes seen in the time range
  • Total Volume – Sum of _raw bytes, formatted as KB/MB/GB
  • Ingest Errors – Count of Splunk ingest-pipeline errors from Splunk_TA_aws and splunk_ta_sap_logserv (filters out ExecProcessor scheduled-input noise; click to open the matching events)
  • Active Hosts – Count of distinct hosts reporting data
  • Host Event Count – Daily event volume per host (log scale)
  • Sourcetype Summary – Rich table with 14 columns: Sourcetype, Status (Fresh/Stale/Very Stale), Trend (sparkline), Events, % of Total, Avg/Day, Volume, App Errors, Hosts, Sources, Events (1h), First Seen, Last Seen, Lag. Click a row to open the search app pre-filtered by sourcetype.
  • Host Latest Activity – Table showing each host’s last event time, event count, and sourcetypes (click a row to drill down to Host Details)

Tab 2 – Linked Graph

  • Source to Sourcetype Mapping – Full-width link graph visualizing the flow of data from source paths to sourcetypes, with column widening tuned so 3 columns fit inside the frame without horizontal scroll

What to Look For

  • Hosts going silent – A host that was previously reporting data but suddenly stops may indicate an agent failure, network issue, or system outage. Check the Host Latest Activity table for stale timestamps and the Sourcetype Summary Status column for “Stale” or “Very Stale” entries.
  • Sourcetype volume drops – A sudden decrease in events for a specific sourcetype often signals an ingestion pipeline issue. The Sourcetype Summary Events (1h) and Trend sparkline columns make recent drops visible at a glance.
  • Unexpected volume spikes – A sharp increase in event volume from a single host could indicate a log storm (runaway process, debug logging left enabled) or a security event generating excessive audit entries.
  • Ingest errors climbing – The Ingest Errors KPI is a curated count (after filtering harmless noise); a non-zero count sustained over days usually points to a misconfigured S3 input, SQS permissions issue, or malformed records in a specific prefix.
  • Missing sourcetype mappings – If a host shows data but is missing an expected sourcetype in the link graph (second tab), the routing transforms may need attention.

Data Pipeline Overview -- Overview tab Data Pipeline Overview -- Linked Graph tab

DNS Analytics

Why This Dashboard Matters

The DNS Analytics dashboard transforms DNS query data into a security detection tool. DNS is used by virtually all network communications and is often exploited by attackers for command-and-control (C2) beaconing, data exfiltration via DNS tunneling, and reconnaissance. Because DNS traffic is rarely inspected by traditional security tools, this dashboard fills a critical visibility gap in SAP infrastructure security.

Panels

  • Total Queries – Aggregate count of DNS queries in the selected time range
  • Unique Clients – Count of distinct client IPs generating DNS queries
  • Beaconing Domains – Count of domains exhibiting periodic query patterns consistent with C2 beaconing
  • Top 10 Clients - Request Volume – Time-series line chart of the most active DNS clients
  • Query Type Distribution – Donut chart showing the breakdown of DNS record types (A, AAAA, PTR, TXT, etc.)
  • Top DNS Resolvers – Table of the BIND resolvers (the DNS hosts processing queries) ranked by query count; the host field is the authoritative resolver since src/dest are always 127.0.0.1 in this environment
  • Requests by Record Type – Line chart trending DNS query types (A, AAAA, PTR, MX, etc.) over time
  • Beaconing Activity – Bubble chart highlighting domains with regular, periodic query patterns that may indicate C2 beaconing
  • Hosts to Beaconing Domains – Full-width bubble chart mapping which hosts are communicating with suspected beaconing domains
  • Top Queried Domains – Table of the most looked-up domains with unique client counts
  • Top Clients by Domain Diversity – Table ranking clients by number of unique domains queried, with queries-per-domain ratio for DGA detection

What to Look For

  • Beaconing patterns – The Beaconing Activity panel uses statistical analysis (low variance in query intervals) to detect domains being contacted at regular intervals, which is a hallmark of malware C2 communication. Domains with low variance and high count are the highest priority.
  • Unusual record types – TXT record queries at high volume are a common DNS tunneling technique. MX queries from non-mail servers may indicate reconnaissance. Watch the Query Type Distribution donut and the Requests by Record Type line chart for deviations from your normal mix.
  • New high-volume clients – A host that suddenly becomes one of the top DNS clients may be compromised and performing domain generation algorithm (DGA) lookups or reconnaissance scanning.
  • Multiple hosts reaching beaconing domains – The Hosts to Beaconing Domains panel shows lateral spread. If multiple internal hosts contact the same suspected C2 domain, it suggests widespread compromise.
  • High domain diversity – Clients querying many unique domains with a low queries-per-domain ratio in the Top Clients by Domain Diversity table may indicate Domain Generation Algorithm (DGA) activity or automated reconnaissance.
  • Resolver imbalance – The Top DNS Resolvers table should show an expected workload split across your BIND servers. One resolver handling disproportionate load may indicate a misconfigured forwarder or a cluster failover that wasn’t noticed.

DNS Analytics

Linux System & Security

Why This Dashboard Matters

The Linux dashboard provides OS-level visibility for the hosts running SAP applications. Most SAP ABAP and HANA systems run on Linux, making OS-level monitoring essential for understanding the infrastructure beneath the application layer. This dashboard combines SAP-aware context (SID, instance, application identification from syslog) with kernel-level security monitoring (firewall drops and kernel events).

Panels

  • Total Events – Aggregate event count across all Linux sourcetypes
  • Firewall Drops – Count of kernel firewall drop events
  • Active Hosts – Count of distinct Linux hosts reporting data
  • Top Drop Source – Single-value panel showing the #1 source IP by firewall-drop count in the format <IP> (<count>), e.g. 10.186.64.6 (8,522). This surfaces the dominant drop source directly in the KPI row so it doesn’t get buried in the table. Click the KPI to drill down to all source IPs ranked by drop count.
  • Event Volume by Sourcetype – Daily trend across linux_messages_syslog plus the v0.0.5.0 Path-B sourcetypes (linux:cron, linux:warn, linux:sudolog, linux:slapd) and linux_secure. The legacy syslog sourcetype is OR-ed alongside the new sourcetypes during the transition; existing sourcetype=syslog indexed data ages out per index retention.
  • SAP Application Activity – Column chart showing event distribution by SAP application and SID
  • SAP Instance Distribution – Table of SAP instances with event counts by SID, instance number, and CID
  • Firewall Drops Over Time – Timeline of kernel firewall drop events
  • Kernel Event Types – Pie chart breakdown of kernel event categories (IN_DROP, segfault, etc.)
  • Top Blocked Sources – Table of source IPs being blocked by the firewall with target counts and protocols
  • Top Blocked Destination Ports – Table of destination ports targeted by blocked traffic

What to Look For

  • Firewall drop spikes – A sudden increase in blocked connections may indicate port scanning, network reconnaissance, or a brute-force attack against SAP services.
  • Top Drop Source concentration – If the Top Drop Source KPI shows a single IP accounting for the overwhelming majority of drops, that IP is either a misconfigured internal system hammering a blocked port (check if it’s an internal SAP host that recently changed config) or a persistent external scanner. Click through to see the distribution.
  • New blocked source IPs – Unfamiliar source IPs appearing in the Top Blocked Sources table should be investigated, especially if they target SAP service ports (3200-3299 for dialog, 8000-8099 for HTTP, 30015 for HANA).
  • SAP application distribution changes – If the SAP Application Activity chart shows a previously active SID or application going silent, it may indicate a process crash or configuration issue.
  • Kernel segfaults – Segmentation faults appearing in the Kernel Event Types panel indicate application crashes, which may affect SAP system stability.
  • Port targeting patterns – The Top Blocked Destination Ports table reveals which services attackers are targeting. Ports associated with SAP services warrant immediate attention.

Linux System & Security

Windows Events

Why This Dashboard Matters

The Windows Events dashboard monitors Windows hosts in the SAP landscape, which commonly run SAP application servers, database instances, and management consoles. Windows Event Logs capture service health, PowerShell execution, and system errors that indicate Windows-specific operational issues. This dashboard focuses on operational health and service state – the authentication-failure story is owned by the Cross-Stack Authentication dashboard so that all three layers (SAP / HANA / Windows) can be investigated together.

Panels

  • Total Events – Aggregate Windows event count
  • Critical / Error – Count of critical and error severity events
  • Active Hosts – Count of distinct Windows hosts reporting data
  • Event Volume by Log – Daily trend by Windows log source (Application, Security, System, PowerShell)
  • Severity Distribution Over Time – Stacked column chart of severity levels
  • Top Event Codes – Featured full-width table of the most frequent EventCodes with 7 enriched columns: Event Code, Description (signature), Source log, Severity, Events, Hosts (distinct count), Last Seen. Row drilldown opens the search app filtered by that event code.
  • Service Events – Table of Windows service start/stop activity with latest state
  • PowerShell Activity – Line chart trending PowerShell event volume

What to Look For

  • High-frequency Event Codes – The Top Event Codes table is the primary starting point. EventCode 7031 / 7034 (service terminated unexpectedly), 1000 (application crash), and 41 (unexpected shutdown) are high-priority. Click through to see every occurrence of a specific code.
  • Service crashes – EventCode 7031 (service terminated unexpectedly) in either the Top Event Codes table or the Service Events panel indicates a critical service failure. For SAP services (sapstartsrv, SAPService), this requires immediate attention.
  • PowerShell activity spikes – Sudden increases in PowerShell execution may indicate lateral movement by an attacker using PowerShell-based attack tools. Correlate with the Cross-Stack Authentication dashboard to see whether any Windows logons were concurrent.
  • Critical/error severity trends – A rising trend in critical and error events over multiple days indicates accumulating system health issues that need proactive investigation.
  • Event Code hosts expanding – The Hosts column on the Top Event Codes table makes it obvious when a normally-host-isolated error starts appearing on multiple hosts – a sign the underlying cause is environmental (failed update, domain policy change) rather than local.

Windows Events

Proxy Analytics

Why This Dashboard Matters

The Proxy Analytics dashboard monitors outbound internet access through the Squid proxy server. In SAP environments, proxy logs reveal which systems and users are accessing external resources, enabling detection of data exfiltration attempts, policy violations (unauthorized internet access), and compromised systems communicating with malicious infrastructure. Proxy logs complement DNS analytics by showing the actual HTTP connections that follow DNS resolution.

Panels

  • Total Requests – Aggregate proxy request count
  • Total Bandwidth – Sum of bytes transferred through the proxy (formatted KB/MB/GB)
  • Denied Requests – Count of requests blocked by proxy policy
  • Request Volume Over Time – Daily request trend
  • Status Code Distribution – Stacked column chart of response categories (2xx-5xx)
  • Top Domains – Table of the most accessed domains with request counts, bandwidth, and unique client counts
  • Top Clients – Table of the most active client IPs with request counts, bandwidth, and domain diversity
  • Top Clients by Domain Diversity – Horizontal bar chart ranking client IPs by the distinct count of URL domains they accessed (replaces the earlier HTTP Methods donut, which consistently collapsed to a single slice and didn’t earn its space)
  • Cache Action Distribution – Column chart of Squid vendor-action values (CONNECT, TCP_HIT, TCP_MISS, TCP_REFRESH_MISS, etc.) – replaces the earlier Content Types donut for the same single-slice reason and reveals cache-hit behavior
  • Bandwidth Over Time – Line chart trending total bytes transferred
  • Top URL Domains by Bytes Out – Horizontal bar chart of the top 10 URL domains by outbound bytes (data exfiltration detection surface)
  • Bandwidth Over Time by Domain (Top 5) – Multi-line chart of the top 5 bandwidth-consuming domains over time – pairs with Top URL Domains by Bytes Out to show whether exfiltration is ongoing or historical

What to Look For

  • Denied request spikes – A sudden increase in denied requests may indicate a compromised system attempting to reach blocked destinations, or a policy change affecting legitimate traffic.
  • High-bandwidth domains – The Top URL Domains by Bytes Out panel is tuned specifically for exfiltration detection. A new domain appearing near the top, or a known-OK domain spiking, warrants investigation.
  • Sustained-bandwidth domains – The Bandwidth Over Time by Domain chart shows whether a top domain’s traffic is steady (expected service) or a sudden burst (exfiltration attempt, large file transfer).
  • Client domain diversity – A client IP with extremely high domain diversity in the Top Clients by Domain Diversity panel is unusual for a well-behaved SAP server – most production systems talk to a predictable small set of external domains.
  • Cache miss dominance – If Cache Action Distribution is dominated by TCP_MISS, the proxy cache may not be earning its keep (or a new workload is defeating it). For security-relevant investigations, CONNECT dominance signals mostly-HTTPS traffic that the proxy can’t inspect.
  • New high-volume clients – A system that suddenly appears as a top proxy client may be compromised and performing outbound scanning, beaconing, or data exfiltration.

Proxy Analytics

Host Details

Why This Dashboard Matters

The Host Details dashboard is the forensic drill-down tool for investigating one or more hosts. Accessed by clicking a host row in any dashboard’s host-keyed table (Data Pipeline Overview, Cross-Stack Authentication, Environment Health, etc.) or by selecting hosts from the title-row Multiselect, it surfaces event volume, data freshness, authentication activity, inventory, errors, and role-specific signals (HANA audit, ABAP work process, web dispatcher traffic, etc.) for the selected scope. This view is essential for root cause analysis, incident response, capacity investigations, and validating that all expected data sources are present.

The title-row host picker is a Multiselect (with filter input + Select All Matches) supporting three scopes:

  • All Hosts (no hosts selected) — Top-N kicks in (default Top 10) and panels aggregate across the most-active hosts.
  • Single host — splices host="X" into all SPL; URL becomes ?host=X (back-compatible deep-link form).
  • Multiple hosts — splices host IN ("X","Y","Z") into all SPL; URL becomes ?hosts=h1,h2,h3 (CSV form). LocalStorage persists the last selection per browser tab.

The dashboard is organized into three tabs:

  • Overview – universal panels that populate for any host with any event: KPIs, event timeline, severity breakdown, inventory, authentication and error tables, top sources, activity-by-hour, and data-freshness checks.
  • Role Activity – panels scoped to a specific role (HANA, ABAP, Web Dispatcher, SAP Router, Windows, Linux sudo, DNS). Each panel auto-hides when the selected host(s) have no data for that role, so the tab only shows what’s relevant for the host(s) in front of you.
  • Sourcetype Mapping – Link graph showing the distribution of events across sourcetypes and sources for the selected host(s).

Overview Tab

  • KPI row (5 values) – Total Events, Data Volume (auto-scaled MB/GB), Active Sourcetypes, Error/Critical event count (red), and Auth Failure count (red)
  • Host Event Count by Sourcetype – Stacked column of daily event volume per sourcetype for the selected host
  • Severity Timeline – Stacked area chart normalizing Critical / Error / Warning / Info severities across all the host’s sourcetypes
  • Host Inventory – Table of hardware specs (CPU, cores, RAM), EC2 instance type, OS, region, and availability zone. Sourced from osquery data in linux_messages_syslog.
  • Recent Authentication Events – Cross-source auth activity table (HANA / Linux / Windows), with a Layer column identifying the source
  • Recent Errors & Criticals – Latest ERROR/CRITICAL/FATAL events across all the host’s sourcetypes with severity, component, and truncated message
  • Top Sources – Horizontal bar of the host’s most active log sources by event count
  • Activity by Hour of Day – Column chart surfacing off-hours activity patterns
  • Data Freshness – Per-sourcetype last-seen table with Fresh/Stale/Very Stale status to spot collection gaps

Host Details — Overview tab

Role Activity Tab

Role-specific panels. Each panel only appears when the host has data for that role; irrelevant panels auto-hide.

  • HANA Audit Activity (Top Actions) – Column chart of top audit actions from sap:hana:audit
  • ABAP Work Process Mix – Horizontal bar of work-process categories from sap:abap:workprocess
  • Web Dispatcher Traffic by Status – Stacked column (2xx/3xx/4xx/5xx) from sap:webdispatcher:access
  • SAP Router Peers – Connection counts by peer IP from sap:saprouter
  • Windows Event Codes (Top 15) – Column chart of top EventCode values from XmlWinEventLog*
  • Sudo Commands – Table of invoking user, target user, command, and count from linux_messages_syslog sudo events
  • DNS Top Queries – Top query strings from isc:bind:query

Host Details — Role Activity tab

Why some panels may show for certain hosts but not others

Panels on the Role Activity tab — and a few on the Overview tab (Host Inventory, Recent Authentication Events, Recent Errors & Criticals) — use the hideWhenNoData attribute. Splunk automatically hides these panels when the selected host’s data source returns no rows in the current time range.

This means the dashboard adapts its layout to what the host actually is:

  • A HANA host sees the HANA Audit Activity panel on Role Activity and hides ABAP, Web Dispatcher, Router, Windows, Sudo, and DNS.
  • An ABAP application server sees the ABAP Work Process Mix panel and hides the HANA / Windows / DNS panels.
  • A Windows host sees Windows Event Codes (and typically Recent Authentication Events) and hides the SAP-specific panels.
  • A full-stack Linux host (with osquery + sudo + SAP ABAP) sees all of Host Inventory, Sudo Commands, ABAP WP Mix, and Recent Authentication Events.
  • A sparse host with only one or two sourcetypes will see just the universal Overview panels plus whichever Role Activity panels match.
  • Selecting All Hosts aggregates across every host — most panels populate because at least one host in the environment contributes data to each.

Empty panels aren’t a bug; they’re the dashboard telling you the selected host doesn’t have that role or doesn’t forward that log type. If you expect a panel to populate for a specific host (for example, an ABAP app server that should have sap:abap:workprocess data but doesn’t), that’s a genuine collection issue worth investigating — check the forwarder configuration and the host’s logging policy.

Sourcetype Mapping Tab

Full-width link graph showing how the host’s events flow from source files to sourcetypes. Useful for spotting a noisy source file that’s producing an outsized share of the host’s volume, or for validating that a host’s expected sources are all represented.

Host Details — Sourcetype Mapping tab

What to Look For

  • Sourcetype gaps – If a host is missing a sourcetype that similar hosts have (e.g., a HANA host without sap:hana:audit), it may indicate a misconfigured audit policy or broken log forwarding.
  • Stale sourcetypes – Any row in the Data Freshness table marked Stale or Very Stale signals that a specific log pipeline has stopped delivering. Drill into the timeline to see when it stopped.
  • Volume anomalies – Compare the selected host’s event volume to its peers. Significantly higher or lower volume may indicate a workload issue, logging configuration problem, or security event.
  • Sudden volume changes – A host that normally generates a steady volume but suddenly spikes or drops warrants investigation. Spikes may indicate security events; drops may indicate system or agent failures.
  • Auth failure spikes – A non-zero Auth Failures KPI on a host that usually has zero is worth immediate attention. Drill to the Recent Authentication Events table for context.
  • Off-hours activity – A non-zero bar in Activity by Hour of Day outside business hours, especially on an app server, can indicate batch jobs, maintenance, or suspicious activity.
  • Sudo command patterns – On the Role Activity tab, a host with unexpected sudo commands (new users, non-standard commands, or commands run As User: root) may reflect lateral-movement attempts or misconfigured automation.
  • Single-sourcetype dominance – If the Sourcetype Mapping link graph shows one sourcetype accounting for the vast majority of the host’s events, the balance may indicate a noisy process (check ICM, workprocess, or web dispatcher logs for runaway activity).

This dashboard accepts host and global time-range query params from the URL, so the time range and host selection carry over from the Data Pipeline Overview and other dashboards.

Ended: Platform

Ended: Dashboards Overview

Ended: LogServ UI App

AI Assistant ↵

AI Assistant — Overview

v0.0.5 release: LLM functionality intentionally disabled pending review

The v0.0.5 release of Splunk for SAP LogServ ships with the LLM-driven AI Assistant path disabled at compile time pending internal review. The predefined-prompt path + Splunk MCP Server integration + tool tiles + drill-down chips + audit log all stay fully active. The free-form chat input is disabled, the model picker is hidden, and the Provider Credentials Settings tab is hidden. See Templates-only Build for the build mechanism. The LLM-driven path will be re-enabled in a future release once review concludes.

What the AI Assistant Is

The AI Assistant is a Splunk-aware chat panel embedded in the LogServ App that lets analysts run pre-canned investigations and free-form questions against their Splunk data. It sits to the right of every dashboard as a togglable side panel, accessed via the ✦ AI Assistant button in the top-right of the app’s nav bar.

It has two distinct paths with different cost, latency, and privacy properties:

  • Predefined prompts (no LLM call). The user opens the prompt browser and clicks one of 48 cataloged prompts. The orchestrator dispatches the saved search via the Splunk MCP Server, renders the result tile in the right pane, and appends a static interpretation + suggested-next-steps card. No vendor LLM is invoked. Free, instant (search latency only), zero data egress.
  • Free-form prompts (LLM-driven). The user types a natural-language question. The orchestrator sends a system primer + the question + tool definitions to the active vendor (Anthropic / OpenAI / Azure OpenAI / AWS Bedrock). The vendor picks tools, the orchestrator dispatches them via MCP, the vendor sees only the privacy-tier-bounded summary, and the vendor synthesizes a narrative response. Vendor-cost-bearing, dependent on LLM credentials, governed by the active privacy tier.

The Privacy Invariant

No event data from your Splunk instance is ever transmitted to any AI vendor.

This is not policy; it is enforced by the type system at build time. The TypeScript compiler refuses to put any tool-result value from MCP into an outbound vendor payload — there is no runtime check, no flag to flip, no policy to forget. The only conversion path produces a non-data summary whose contents are gated by the active privacy tier (Tier 0 / Tier 1 / Tier 2):

  • Tier 0 (future) — air-gapped Ollama; no vendor traffic at all.
  • Tier 1 (default) — count + execution_time only. The AI sees no values.
  • Tier 2 (admin opt-in) — adds aggregated metadata: per-column cardinality, top-N values + counts (categorical), min/max/avg/sum (numeric), time range. Still no raw rows.

What the AI vendor sees: your natural-language question, schema descriptions, tool definitions, the AI’s own summaries. What the AI vendor does not see: any field value from any event in your sap_logserv_logs index.

For details, see Privacy Tiers.

Architecture at a Glance

   User question
        |
        v
   AI vendor  -->  AI picks tools  -->  Splunk MCP Server  -->  Splunk search-job
        |                                                              |
        |                                  (raw rows stay client-side) |
        |  <-----  privacy-tier summary  <----+ (count + timing,
        |                                       optionally aggregates)
        |
        v  AI synthesizes narrative reply
   Chat panel (left pane)        Tool result tiles (right pane)

Every saved-search dispatch produces a tool-result tile in the right pane (table / chart / KPI / pie based on the prompt’s renderHint). The user sees the actual rendered data; the AI sees only the privacy-tier-bounded summary. Drill-down chips (↗ Dashboard, ↗ Run SPL) on each tile (and beside chat citations) connect the conversation back into the dashboards or Splunk’s universal Search app — see Drill-down Chips.

For the build-time type-system mechanism that makes this guarantee non-bypassable, see AI Assistant Implementation Reference.

Key Capabilities

  • 48 predefined prompts in three packs (sap_basis 13, security 14, operations 13) plus a context-aware Dashboard Focused tab that auto-filters to prompts relevant to the dashboard you currently have open. See Predefined Prompts.
  • Four LLM providers for the free-form path: Anthropic (direct API), OpenAI (direct API), Azure OpenAI (composed with Azure auth), AWS Bedrock (Claude on Bedrock with Bedrock API Keys). Per-user model-picker in the chat header lets users switch within the active provider’s model list. See Settings & Configuration.
  • Power Mode — admin-granted role-gated ✦ Power toggle that forces a saved-search dispatch before LLM synthesis (forced-RAG). Guarantees data-grounded answers; never responds from prior knowledge alone.
  • Time-window reasoning — primer rules teach the AI to identify the dispatch window, normalize cumulative count to per-hour or per-day rate, run a verify-query before declaring high-severity, and state the window precisely in narrative.
  • Audit log — every action (canned dispatch, vendor call, security block, privacy-tier elevation, legal acknowledgement) lands in a dedicated _ai_assistant_audit index. In-app browser + optional HEC forwarder for tamper-evidence.
  • OWASP LLM Top 10 (2025) compliance — every item has a matching control: prompt-injection sanitization, type-bounded data redaction, supply-chain SBOM, audit hash chain, per-user rate limit, USD spend cap, SPL static-analysis guard, jailbreak detection, PII redaction.
  • Templates-only build variant — a deployable variant of the LogServ App that disables the LLM-driven flow at compile time. The MCP path + canned prompts + audit log stay fully active.

Prerequisites

  • Splunk 9.4.3 or later.
  • Splunk MCP Server (Splunkbase App 7931) v1.1.0 or later installed on the same search head as the LogServ App. See Splunk MCP Setup. Cookie auth from the same Splunk Web session works by default; the optional bearer token layers on top.
  • One of the four supported LLM providers with a valid API credential — only required for the free-form path. Predefined prompts work without any LLM provider.
  • Admin user role to configure provider credentials, set the privacy tier, manage Power Mode roles, and view the Audit Log tab.

First-time UX

  1. Click ✦ AI Assistant in the top-right nav. The right-side panel opens.
  2. If MCP isn’t healthy, the panel shows a setup wizard with diagnostic guidance — see Splunk MCP Setup. Otherwise, the empty chat panel renders.
  3. Click Browse predefined prompts to open the catalog modal. Pick a prompt from the SAP Basis / Security / Operations / Dashboard Focused tab.
  4. The prompt dispatches via MCP and renders a tool tile in the right pane along with a static “How to read this result” guidance card on the left.
  5. Click any of the drill-down chips on the tile (↗ Dashboard, ↗ Run SPL) to investigate further.

For free-form prompts, type a natural-language question in the chat input and press Send (or Cmd/Ctrl+Enter). The chat status indicator shows when the AI is generating a response or running a search.

When to Use Which Path

Use case Recommended path
Routine “show me X” investigations on the cataloged dimensions Predefined prompts
Compliance / audit reports with a fixed cadence Predefined prompts (deterministic, zero vendor traffic)
Free-form questions about the data the catalog doesn’t cover Free-form, with Power Mode on if available
Cross-cutting “what’s most critical right now?” investigations Free-form (the AI dispatches multiple saved searches in parallel and synthesizes)
Demonstration environments where vendor-LLM access is intentionally unavailable Templates-only build
Air-gapped environment with no outbound internet Tier 0 (future — see Privacy Tiers)

Where to Go Next

  • Privacy Tiers — full breakdown of what each tier exposes and the decision matrix for picking one.
  • Predefined Prompts — the 48 prompts, the prompt browser UX, and the intent-map customization story.
  • Free-form Prompts — the LLM-driven flow, tool dispatch, citations, rate limiting, and Power Mode.
  • Splunk MCP Setup — installing and configuring the prerequisite MCP server.
  • Settings & Configuration — the 4-tab admin Settings page (General / Provider Credentials / Splunk MCP / Audit Log).
  • Audit Log — what’s logged, the in-app viewer, and the optional HEC forwarder for tamper-evidence.
  • OWASP LLM Top 10 Compliance — security controls posture for compliance reviews.

Privacy Tiers

v0.0.5 release: privacy tiers are designed but not exercised — LLM dispatch is disabled

The v0.0.5 release ships with the LLM-driven path disabled at compile time pending internal review. The privacy tiers described on this page are designed and implemented (the Hidden<T> / Visible<T> type-system enforcement is always active and will protect any future LLM dispatch), but no LLM dispatch happens in a v0.0.5 build — there is no vendor traffic to govern via tier selection. The tier setting in Settings → General is preserved so the configuration survives across releases; it just has no effect until the LLM path is re-enabled in a future release.

The AI Assistant supports three privacy tiers that control what an external LLM vendor sees about your Splunk data. Tier selection is admin-configurable in Settings → General. The active tier is enforced by the TypeScript type system at build time, not by runtime policy: every outbound vendor payload must pass through a single sanitize chokepoint, and the chokepoint’s summarizer is what the tier setting controls. For the type-system mechanism, see AI Assistant Implementation Reference.

The Three Tiers at a Glance

Tier Vendor traffic? What the AI sees about each tool result Use case
Tier 0 (future release) None — Ollama local Same as Tier 1 (count + timing only) Air-gapped customers, regulated industries with no-outbound-internet constraints
Tier 1 (default) Cloud LLM (Anthropic / OpenAI / Azure / Bedrock) Returned N rows in M ms. Nothing else. Most customers — strongest cloud-LLM privacy posture
Tier 2 (admin opt-in) Same cloud LLM Tier 1 line + per-column cardinality + top-N values + counts (categorical) + min/max/avg/sum (numeric) + time range. Still no raw rows. Customers who want data-grounded narrative replies and have legal authority to expose aggregate metadata to the vendor

Tier 0 — Ollama Local (future release)

Status: roadmap, ~1 week of engineering effort. Not yet shipped.

What it does: routes the LLM call to a locally-hosted Ollama server instead of a cloud vendor. No outbound internet traffic; no third-party data processor. Strongest possible privacy posture short of disabling free-form prompts entirely (see Templates-only Build for that).

Privacy data flow:

   User question  -->  Local Ollama  -->  AI picks tools  -->  MCP Server
                                                                    |
                                                Hidden<MCPToolResult>
                                                                    |
                                                                    v
                                            sanitize() -> count + timing
                                                                    |
                                                                    v
                                              Local Ollama synthesizes reply

Considerations:

  • Local Ollama latency is substantially higher than cloud LLMs unless the customer has GPU hardware on the search head.
  • Model selection is constrained to what Ollama supports locally (Llama-family, Mistral-family, etc.); cloud-only models like Claude Opus aren’t available.
  • The MCP server prerequisite is unchanged — Tier 0 only changes the LLM endpoint, not the search-execution path.

For customers waiting on Tier 0, the Templates-only Build is the interim solution: ship a build with the LLM-driven path disabled at compile time, leaving only the canned-prompt path active.

Tier 1 — Cloud LLM, Count + Timing Only (default)

What the LLM sees per tool result:

<TOOL_RESULT_DATA>
Returned 47 rows in 320ms.
</TOOL_RESULT_DATA>

That’s the entire summary. Nothing about the column names, the row contents, the time range, the distinct counts, or the actual values. The <TOOL_RESULT_DATA>...</TOOL_RESULT_DATA> sentinel block is wrapped around every tool-result summary so the AI’s primer can teach it that anything inside is data from the customer’s environment, never instructions.

What the AI does with this: the AI knows the shape of what was returned (a count + how long the search took) but not the values. It writes structurally-aware narrative replies — “the failed-auth saved search returned 47 rows for the rolling window, suggesting a non-trivial number of distinct user/stack pairs” — without making claims about specific users, IPs, or hosts.

Decision rule for picking Tier 1:

  • The customer’s legal/security team is comfortable with the AI vendor seeing the natural-language question + tool definitions + summaries.
  • The customer is not comfortable exposing aggregate column metadata (cardinality, top-N values, min/max) to the vendor.
  • The customer values determinism: a Tier 1 reply mentions zero specific values, so audit reviewers can sanity-check the narrative against the rendered tile without needing to verify what the AI was told.

Limitations: the AI can’t write “the top failing user was xcjadm with 42 attempts” — it doesn’t have that data. The narrative is correspondingly less concrete; the user is expected to read the tile in the right pane for the actual values.

Tier 2 — Cloud LLM + Aggregated Metadata (admin opt-in)

What the LLM sees per tool result:

<TOOL_RESULT_DATA>
Returned 47 rows in 320ms.
Time range: 2026-04-15T00:00:00Z → 2026-05-05T23:59:59Z.
Column "user" (distinct=23): xcjadm=8, BKPADMIN=6, ENCRYPTMON=5, sapadm=4, svc_monitor=3 (+18 more).
Column "stack" (distinct=3): Windows=21, HANA=19, SAP=7.
Column "failures" (numeric, distinct=42): min=1 max=4799 avg=87.4 sum=4108.
</TOOL_RESULT_DATA>

The AI now sees the column names, top-N values + counts (per-column-cardinality limited to top 10 by default; configurable up to 50 via the AI’s top_n tool arg), numeric stats, and the time range. It still does NOT see the raw rows — there is no way for the AI to know which user issued how many failures across which stacks; only the per-column distributions.

PII redaction (Tier 2 only): if the column name matches a known identifier pattern (email, user(name), *_ip, mac, account, with hostname opt-in), the values are replaced with stable <redacted-XXXXXXX> tags before being shown to the AI. The same value across the run produces the same tag, so the AI can still reason about cardinality + frequency, but never sees the actual identifier. See OWASP LLM Top 10 — LLM02.

What the AI does with this: the AI can write data-grounded narrative replies — “the top failing user is <redacted-4e4945f> with 42 attempts; the failure mix is 21 Windows, 19 HANA, 7 SAP — Windows is the dominant stack but the spread suggests a coordinated probe rather than a single-source brute-force”. The AI now grounds its severity assessments in actual numbers instead of inferring from row counts alone.

Decision rule for picking Tier 2:

  • The customer’s legal/security team has reviewed and approved exposing aggregate metadata (column cardinalities, top-N value distributions) to the AI vendor.
  • The customer wants narrative replies with concrete values, not just shape descriptions.
  • PII concerns are mitigated by the column-name-based redaction (or the customer is comfortable with the redaction’s coverage).

Audit trail for Tier 2: every Tier 2 vendor call records a vendor_tier2 audit event including: - Token counts (input + output) - USD cost estimate - Number of PII redactions applied (tier2RedactionsApplied) - The user identity that triggered the call

When elevating from Tier 1 → Tier 2 in Settings, a vendor_tier2_elevation audit event captures the admin’s identity and timestamp. See Audit Log.

How the Privacy Invariant Is Enforced

The privacy invariant — no event data ever leaves the customer’s environment for the AI vendor — is enforced mechanically at build time via TypeScript brand types. Tool-result values from the MCP server carry one type; outbound vendor messages carry a different type; the compiler refuses to assign one to the other. The only conversion path produces a non-data summary, and the active privacy tier picks which summary shape is used.

This is the primary defense against LLM02 — Sensitive Information Disclosure. For the brand-type mechanism, summarizer call sites, and the <TOOL_RESULT_DATA> sentinel pattern, see AI Assistant Implementation Reference.

Switching Tiers

Tier is configured in Settings → General. Switching tiers takes effect on the next vendor call — in-flight calls complete with the previously active tier, and the audit event records which tier was active at dispatch time.

Elevating from Tier 1 → Tier 2:

  1. Open Settings → AI Assistant → General.
  2. Change the Tier field from 1 to 2.
  3. Click Save Defaults. The save records a vendor_tier2_elevation audit event with admin identity + timestamp.
  4. Optionally enable / disable PII redaction (default: enabled) and toggle hostname-as-PII (default: hostname NOT redacted, since it’s often non-sensitive in SAP environments).

Lowering from Tier 2 → Tier 1 / Tier 0:

  1. Same path. The save is recorded as a regular config change, no special audit event.
  2. The next vendor call uses the lower tier.

No tier: if the customer wants to disable the LLM-driven path entirely (canned prompts only), use the Templates-only build variant — the tier setting becomes moot since there are no vendor calls.

Decision Matrix

Question Answer Recommended tier
“Is the AI vendor a third-party data processor?” Yes; consult your DPA. Anthropic / OpenAI / Azure / Bedrock all publish their own privacy commitments, but they ARE data processors. Tier 1 if approved as processor; Tier 0 if not approved
“Can the customer’s compliance team approve aggregate metadata exposure?” Yes Tier 2
No Tier 1
“Does the customer have outbound internet to the LLM vendor’s API?” Yes Tier 1 or Tier 2
No (air-gapped) Tier 0 (future) or Templates-only Build
“Does the customer want narrative replies with concrete values, or shape-only?” Concrete Tier 2 (with PII redaction on)
Shape-only Tier 1
“Does the customer want zero LLM dispatch (partner test, demo)?” Zero Templates-only Build

Predefined Prompts

Predefined prompts are 48 cataloged saved searches across three packs that the AI Assistant can dispatch via the Splunk MCP Server without invoking any LLM. They are the deterministic, vendor-traffic-free path through the AI Assistant: the user clicks a prompt card, the orchestrator dispatches the saved search, the result tile renders in the right pane, and a static interpretation + suggested-next-steps card appears in the chat.

This is the only path that’s active in the Templates-only build variant — partners can demo + investigate the LogServ solution end-to-end without any LLM provider configured.

AI Assistant — predefined-prompts modal open

The Three Packs

Pack Count Focus
SAP Basis 13 SAP Basis admin investigations: HANA service severity breakdown, ABAP work-process errors, integration topology call volume, RFC partner failures
Security 14 Auth failures, beaconing detection, after-hours admin activity, permission grants, GRANT-to-non-managed-role detection, HANA SYSTEM logins, Windows lockouts
Operations 13 Ingest health, error rate trends, error-category breakdowns, top error categories, host volume drop, freshness checks, Cloud Connector backend latency

The full catalog ships inside the LogServ App as a data-driven intent map. Each prompt entry includes the saved-search name, label, description, render hint, chart hint, chart palette, dashboard mapping, interpretation, and suggested next steps.

The Dashboard Focused Tab

The first-position tab in the prompt browser is Dashboard Focused — it auto-filters the catalog to prompts that are mapped to the dashboard you currently have open. The mapping is the per-prompt dashboard field in the intent map (single string or array — multi-target prompts map to multiple dashboards). The tab auto-hides when the active dashboard has zero matching prompts (e.g., on Settings or AI Assistant pages).

Each card on the Dashboard Focused tab carries a small pack-origin chip (cyan-light SAP Basis / salmon-red Security / gold-orange Operations) so users can find the prompt back in its home pack later.

What Happens When You Click a Prompt

  1. Dispatch. The orchestrator calls splunk_run_saved_search on the MCP server with the prompt’s saved-search name and the dashboard’s currently-selected time range.
  2. Tile renders. The MCP server returns the rows. The orchestrator renders them in a tool-result tile in the right pane, with the prompt’s pre-configured renderHint (table / timechart / kpi / pie) and optional companion chartHint (chart on top + table below for renderHint=table prompts that benefit from a visual).
  3. Guidance card. A “How to read this result” card appears in the left chat pane below the user’s prompt label. The card has two parts:
    • Interpretation — one paragraph explaining what the result means and what shapes are normal vs. concerning. Sourced from the intent map’s interpretation field.
    • Suggested next steps — a bulleted list of follow-up investigations. Some entries are plain text; others are clickable deep-link chips that open Splunk’s universal Search app with a focused SPL pre-filled and the dispatch’s exact earliest/latest pre-applied. Sourced from the intent map’s nextSteps array.
  4. Drill-down chips on the tile. Every tile carries ↗ Dashboard chip(s) (one per resolvable dashboard from the prompt’s dashboard field) and a ↗ Run SPL chip in its actions slot. See Drill-down Chips.

No LLM call. No vendor traffic. No tokens. The dispatch is recorded as a local_only audit event with the prompt id + SPL + row count + execution time + ok flag. See Audit Log.

The Tab Persistence Convention

The prompt browser remembers your last-selected tab across modal-open events, persisted per-tab in sessionStorage under logserv.aiAssistant.promptActivePack. The persistence rule is selection-based, not casual-flipping-based — your tab choice is only remembered if you actually picked a prompt from it. Casual tab-flipping (browsing without clicking a card) doesn’t persist. If your last selected tab points at a tab that’s hidden on the current dashboard (e.g., persisted to Dashboard Focused on a dashboard with no matching prompts), the resolver falls back to sap_basis.

Customizing the Catalog

The prompt catalog is data-driven and customizable. New prompts can be added, existing prompts can be edited, and the catalog can be tailored per deployment. The full JSON schema, build-time consistency check, and pre-deploy SPL dry-run recipe are documented in AI Assistant Implementation Reference — Adding a New Predefined Prompt.

The Splunk Risky-Command Safeguard

Splunk Web’s Search app has a safety modal that triggers on “risky” commands: map, runshell, script, delete, crawl, tscollect, loadjob, outputlookup, outputcsv, sendalert, sendemail. Any URL passed to /app/search/search?q=... containing one of these triggers the modal — even though our own MCP-dispatched saved searches don’t.

The drill-down chips that open Splunk Search from the prompt browser’s nextSteps deep-links are subject to this safeguard. The shipped prompt catalog has been scanned and rewritten where needed to avoid these commands; if you customize the catalog, scan any new SPL for risky commands before shipping. See AI Assistant Implementation Reference for the rewrite pattern (map → first-class subsearch).

Clear All / Per-Result Clear

The right pane carries a Clear All button in the toolbar above the result list, plus a Clear button on every individual tile (in the actions slot, alongside the drill-down chips). Both prune the conversation:

  • Per-tile Clear removes the tool_call + tool_result + the user message that triggered the dispatch (so a canned-prompt’s chat label disappears alongside its result panel) + the guidance card. Outbound vendor refs for that tile are also stripped from the next-turn vendor-message buffer.
  • Clear All wipes the entire conversation, including all tool_results, all guidance cards, all assistant_text messages, and all user messages. The chat session id is preserved so audit events continue with the same correlation id.

Free-form Prompts

v0.0.5 release: this entire feature is disabled pending review

The free-form / LLM-driven path is disabled at compile time in the v0.0.5 release pending internal review. Chat input is greyed out, Send button is disabled, model picker and Power Mode toggle are hidden, and the Provider Credentials Settings tab is hidden. None of the behavior described on this page is reachable in a v0.0.5 build. This page is preserved as the design reference for the future release that re-enables the LLM path. See Templates-only Build for the build mechanism and OWASP LLM Top 10 Compliance for the security posture under review.

Free-form prompts are the LLM-driven path through the AI Assistant. The user types a natural-language question; the orchestrator sends a system primer + the question + tool definitions to one of four supported LLM providers; the vendor picks tools, the orchestrator dispatches them via the Splunk MCP Server, and the vendor synthesizes a narrative response from the privacy-tier-bounded summaries. The narrative ends up in the chat panel on the left; the actual data lands in tool-result tiles on the right.

This path requires a configured LLM provider credential, is governed by the active privacy tier, and is disabled at compile time in the Templates-only build variant.

The Four Supported Providers

Provider API Endpoint Auth Models
Anthropic api.anthropic.com (direct) API key Claude Opus 4.7 / Sonnet 4.6 / Haiku 4.5
OpenAI api.openai.com (direct) API key GPT-4o / GPT-5 family (per OpenAI catalog)
Azure OpenAI Customer’s Azure deployment URL Azure auth + URL pattern Per Azure deployment configuration
AWS Bedrock Bedrock API (Claude on Bedrock) Bedrock API Keys (no SigV4 signing) Claude on Bedrock — same models as Anthropic direct

Active provider + default model are admin-configured in Settings → General. Per-user model picker in the chat panel’s privacy banner lets users switch within the active provider’s models[] list.

For AWS Bedrock customers using IAM-only credentials (no Bedrock API Keys available), a future release will add a server-side proxy that signs requests with SigV4. See Auto-Mint MCP Token Roadmap. Currently roadmap, not yet shipped.

The Free-form Flow

   User types question  -->  Orchestrator builds system primer + tools + history
                                                          |
                                                          v
                                       LLM vendor (streamed response)
                                                          |
                                  AI emits zero or more tool_use blocks
                                                          |
                                                          v
                          For each tool_use, orchestrator dispatches via MCP
                                                          |
                                              MCP returns Hidden<MCPToolResult>
                                                          |
                                  sanitize() chokepoint -> Tier 1 / Tier 2 summary
                                                          |
                                                          v
                                  AI continues, may emit more tool_use blocks
                                                          |
                          (loop until AI emits a final assistant_text response)
                                                          |
                                                          v
                                  Narrative renders in chat; tiles render in right pane

Tool dispatch is parallelized — when the AI emits multiple tool_use blocks in a single turn (e.g., dispatching 5 saved searches at once for a “find the top issues” question), the orchestrator dispatches all of them concurrently via Promise.all. The slowest one bounds the turn latency, not the sum.

The Two MCP Tools the AI Sees

The AI’s tool definitions on every free-form request are:

splunk_run_saved_search

Dispatch a saved search from the LogServ App’s catalog (one of the 48 prompts described in Predefined Prompts).

Arg Type Description
name string (required) Saved-search name from the catalog (e.g., logserv_hana_failed_auth)
earliest_time string (optional) Splunk earliest token (e.g., -24h, -7d). Falls back to the dashboard’s TimeRange picker.
latest_time string (optional) Splunk latest token (e.g., now).
render_hint string (optional) One of table / timechart / kpi / pie. Falls back to the catalog’s per-prompt renderHint.
top_n integer (optional, default 10, max 50) Width of categorical aggregates the AI receives in Tier 2 summary. The AI passes this when the user asks for “top 25 X” or similar.

splunk_run_query

Dispatch ad-hoc SPL written by the AI. Used when no saved search fits the user’s question.

Arg Type Description
query string (required) SPL string. Must start with the LogServ macro \sap_logserv_idx_macro`` (the AI’s primer enforces this) and use only read-only commands.
earliest_time string (optional) Same as above.
latest_time string (optional) Same as above.
render_hint string (optional) Same as above.
top_n integer (optional) Same as above.

SPL static-analysis guard (LLM06 — Excessive Agency): the orchestrator runs every splunk_run_query SPL through a guard that blocks collect, outputlookup, outputcsv, delete, sendalert, sendemail, script, run, tscollect. Blocked SPL produces a synthetic error tool_result + a security_blocked_spl audit event. The AI sees the error and can recover by writing a different query.

The System Primer

A system-message prelude is sent on every free-form request. The primer teaches the AI:

  • A data-boundary rule that distinguishes customer data from instructions (mitigates LLM01 / LLM04)
  • The catalog of 48 saved searches (so the AI prefers splunk_run_saved_search when a saved search fits)
  • The LogServ data model (sourcetypes + key fields) for ad-hoc SPL
  • The read-only-operators list for splunk_run_query
  • The time-window reasoning rules that kick in for severity claims
  • Synthesis rules — lettered findings, severity dots, citation format

Two primer variants ship — one for each cloud tier — and the active tier picks which one is sent. For the primer’s full architecture (boundary block, primer constants, per-variant content), see AI Assistant Implementation Reference.

Citation + Drill-Down Chips in the Narrative

The AI’s narrative response uses a citation format [→ saved_search_name] to attribute each finding to its dispatched tool. Each citation in the chat becomes a clickable scroll-to-tile span (clicking it scrolls the right pane to the matching tile). The parser also auto-appends ↗ Dashboard chips (one per related OOTB dashboard) and a ↗ Run SPL chip on the same line — see Drill-down Chips.

Severity markers ([severity:critical|high|medium|low]) are rendered as glossy colored dots inline next to the finding’s alpha letter. The finding format is enforced by the primer:

A. [severity:high] Cross-stack auth failures concentrated on Windows.
[→ logserv_cross_stack_auth_failures] 7 of the top-10 failing-stack rows are
Windows; one user account hit 4,732 cumulative attempts — verify-query (-24h)
confirmed 412 of those landed today, ~17/hr, an active rate.

Per-User Rate Limiting

Free-form prompts are subject to a per-user rolling-1-hour rate limit, configurable in Settings → General (default 30 prompts/hour, 0 = disabled). Canned-prompt dispatches are intentionally NOT rate-limited — they bypass the AI vendor entirely (no token cost) and are bounded by Splunk’s own search-quota controls.

When a user hits the cap, the next prompt:

  1. Renders the user message in chat as usual (preserves history continuity).
  2. Surfaces a system_notice in chat: “Rate limit reached: N prompts in the last hour. Try again at HH:MM.”
  3. Records a rate_limited_prompt audit event with the user’s identity, the cap value, and the timestamp.
  4. Does NOT invoke the LLM vendor.

The cap is per-user, not per-session — opening a new browser tab doesn’t reset it. Maps to LLM10 — Unbounded Consumption.

Token-Usage Audit + USD Cost Estimate

Every Tier 1 / Tier 2 vendor call records a vendor_tier1 or vendor_tier2 audit event with:

  • Provider (anthropic / openai / azure_openai / bedrock)
  • Model
  • Input tokens, output tokens, total tokens
  • USD cost estimate (per-vendor pricing table — see AI Assistant Implementation Reference)
  • Outbound bytes
  • Prompt length (chars)
  • Number of tool turns in this dispatch
  • For Tier 2: number of PII redactions applied

The Audit Log tab in Settings provides aggregate views; the USD cost estimate is a sticker price (not the customer’s negotiated rate, which we don’t know) and should be treated as an order-of-magnitude indicator. See Audit Log.

Daily Spend Cap

To prevent runaway vendor spend, the orchestrator enforces a per-app-instance daily USD spend cap (configurable in Settings, default $X / day; admin sets per organization risk tolerance). When a vendor call would push cumulative cost over the cap, the orchestrator:

  1. Refuses the dispatch.
  2. Records a daily_spend_cap_hit audit event.
  3. Surfaces a system_notice: “Daily spend cap reached ($X.XX of $Y.YY budgeted). Resets at 00:00 UTC. Contact your admin to raise the cap.”

The cap is cumulative-cost-based, not request-count-based — a single high-token-count free-form turn counts against the budget the same way many cheap turns do. Maps to LLM10 — Unbounded Consumption.

Streaming + Abort

The vendor response is streamed token-by-token. The chat-side rendering shows partial responses as they arrive, including partial tool_use blocks. The status indicator switches between three states:

  • streaming — AI is generating text or about to emit a tool_use
  • tool_executing — orchestrator is dispatching a tool the AI requested
  • idle — turn complete

A red Stop button appears in the chat input toolbar during streaming / tool_executing states. Clicking it aborts the in-flight vendor call (closes the SSE stream) and any pending tool dispatches. The conversation history preserves what was emitted up to the abort point so the user can iterate from there.

Jailbreak Pattern Detection

Every user prompt is run through a jailbreak-pattern analyzer before dispatch. The analyzer is flag-and-proceed: a match fires a user_prompt_jailbreak_flag audit event but does NOT block the prompt. The defense-in-depth chain (type-system enforcement + Tier 2 sanitizer + tool-result sentinel + primer + vendor-side defenses) already covers the threat; the analyzer adds SOC observability for the user-prompt vector. Each flag captures a hash of the prompt (so SOC can correlate without archiving plain text), which patterns matched, and a character-class fingerprint to surface unusual encodings.

For the analyzer’s pattern groups + audit event schema, see AI Assistant Implementation Reference. Maps to LLM01 — Prompt Injection.

Session Tool-Call Cap

In addition to the per-user rate limit, every chat session has a per-session tool-dispatch cap to prevent infinite tool loops (a misbehaving model emitting tool_use after tool_use without ever producing a final assistant_text). Counter resets on chat clear. When exceeded, dispatch is refused with a synthetic error tool_result and a session_tool_cap_hit audit event records the hit. Maps to LLM06 — Excessive Agency.

Power Mode

v0.0.5 release: Power Mode is hidden — LLM dispatch is disabled

Power Mode requires the LLM-driven path, which is disabled at compile time in the v0.0.5 release pending internal review. The ✦ Power toggle is hidden in v0.0.5 builds regardless of the power_user_roles configuration. This page is preserved as the design reference for the future release that re-enables the LLM path.

Power Mode is a role-gated ✦ Power toggle in the AI Assistant chat-input toolbar that forces a saved-search dispatch before LLM synthesis on every prompt. When Power Mode is on, the AI MUST call splunk_run_saved_search (or splunk_run_query) at least once before generating any narrative response — forced-RAG. Reasoning from prior knowledge alone is disallowed; every reply is data-grounded.

Why Power Mode Exists

The default free-form path lets the AI choose whether to dispatch a tool or reply from prior knowledge. For most questions the AI dispatches appropriately — but for questions that look like the AI knows them already from training data (“What’s a HANA audit log?”), it may answer from prior knowledge and skip the tool call. That’s fine for explanatory questions, but it’s the wrong behavior for investigations: a “what’s the failure rate this week?” question deserves a real Splunk dispatch, not a probability-weighted guess.

Power Mode resolves this by adding a forced-RAG rule to the system primer: for every user message, the AI MUST call splunk_run_saved_search at least once (or splunk_run_query if no saved search fits) BEFORE generating any narrative answer.

This is forced-RAG over chosen-RAG: same tools, same MCP path, same privacy tier — Power Mode just enforces the saved-search-first step. For the primer-augmentation mechanism, see AI Assistant Implementation Reference.

Role-Gated Visibility

Power Mode is opt-in per organization and per-role. The toggle is hidden from users entirely unless their Splunk role is in the configured allow-list. Default: empty list — Power Mode hidden from every user.

Configuration path (admin-only):

  1. Open Settings → AI Assistant → General.
  2. Find the Power Users subsection. The Multiselect is populated from Splunk’s services/authorization/roles REST endpoint.
  3. Pick the roles that should see the Power toggle (typical: admin, power, sc_admin).
  4. Click Save Defaults.
  5. Affected users see the ✦ Power button in their chat input toolbar on next page load.

Per-user gating would require a custom directory and doesn’t compose with Splunk’s standard RBAC. Role gating reuses the same primitive admins already manage for everything else (search permissions, app permissions, etc.).

Toggle UX

The ✦ Power button is a small pill in the chat input toolbar between Send and the keyboard-shortcut hint. State is persisted per-tab in sessionStorage under logserv.aiAssistant.powerMode — opening a new browser tab resets to OFF.

Visual states:

State Button text Border Tooltip
OFF (default) ✦ Power panel-border-weak “Power Mode OFF — click to enable forced saved-search-first behavior.”
ON ✦ Power ON cyan-accent fill “Power Mode ON — every prompt forces a saved-search dispatch before AI synthesis. Click to turn off.”

A11y: aria-pressed="true" when on, aria-pressed="false" when off.

What Power Mode Does NOT Change

  • Privacy tier behavior is unchanged. Tier 1 still gives the AI count + timing only; Tier 2 still gives aggregated metadata only. Power Mode just enforces saved-search-first; tier still controls what AI sees about each search result.
  • Tool dispatch is unchanged. Same MCP path, same tool definitions (splunk_run_saved_search, splunk_run_query), same SPL static-analysis guard, same session tool-call cap.
  • Audit categories are unchanged. Power Mode runs through the same vendor_tier1 / vendor_tier2 audit events; the new field powerMode?: boolean records whether the toggle was on at dispatch time, for SOC pivot analysis.
  • Per-user rate limit, daily spend cap, jailbreak detection. All unchanged.

When to Use Power Mode

Use case Power Mode?
Investigations that should be data-grounded (“what’s failing right now?”) ON
Cross-cutting top-N questions (“find the top 10 issues to attend to”) ON — Power Mode + the TIME-WINDOW REASONING rules together produce verify-confirmed severity claims
Compliance reports that the auditor will cross-check against the rendered tiles ON (so every claim has a citation)
Conceptual questions (“what’s a HANA audit log?”, “explain risk_level”) OFF (the AI’s training knowledge is fine)
One-shot follow-ups in an existing conversation where prior tool results already cover the answer OFF (forces an unnecessary dispatch)
Streaming-only chat-only mode (mcp_required=false) Power Mode auto-hides — there’s no tool path to force

Audit Trail

Every free-form vendor call records a vendor_tier1 or vendor_tier2 audit event with a powerMode flag that captures whether the toggle was on for that turn. Use cases for the audit field:

  • SOC pivot: filter audit events to powerMode=true to see only the strict-data-grounded turns.
  • Compliance review: confirm that audit-period investigations were run with Power Mode on (so every cited finding has a tool dispatch behind it).
  • Cost analysis: Power Mode-on turns dispatch more tools per turn → higher per-turn cost. The audit lets finance attribute spend to the gating choice.

Power Mode does NOT generate a separate audit category — it’s a flag on the existing vendor_tier1 / vendor_tier2 events. See Audit Log.

Drill-down Chips

Drill-down chips are clickable affordances that appear next to AI Assistant tool-result tiles (right pane) and inline citations in the chat narrative (left pane). They connect the AI Assistant’s investigation flow back into the dashboards or Splunk’s universal Search app, with the dispatch’s exact time range pre-applied.

AI Assistant — drill-down chips on a tool tile

The Two Chip Types

↗ Dashboard

Opens the related OOTB dashboard in a new browser tab. Sourced from the per-prompt dashboard mapping in the intent map — single string or array. Multi-target prompts (20 of 48 — e.g., logserv_hana_failed_auth → HANA Audit + Cross-Stack Authentication) render multiple chips, one per target.

The URL embeds the dispatch’s exact earliest/latest as query params, and the destination dashboard hydrates its time picker from those params on mount.

↗ Run SPL

Opens Splunk’s universal Search app in a new browser tab with the dispatched SPL pre-populated and the dispatch’s exact earliest/latest pre-applied. The chip is hidden on synthetic blocked-SPL results so it doesn’t help the user manually run a query that the SPL static-analysis guard just rejected.

For canned-prompt dispatches, the SPL is the prompt’s catalog SPL string. For AI-driven saved-search calls, the SPL is resolved from the same catalog lookup. For AI-driven ad-hoc queries, the SPL is the AI’s own query string.

Where Chips Render

On every tool-result tile in the right pane

Chips appear in the tile’s actions slot (in the FramedPanel header), between the tile title and the Clear button. Order: dashboard chip(s) first, then ↗ Run SPL, then Clear.

Alongside [→ saved_search] citations in the chat narrative

The AI’s narrative response cites tool dispatches as [→ logserv_xxx]. Each citation in chat becomes a clickable scroll-to-tile span (clicking it scrolls the right pane to the matching tile), with sibling chips auto-appended on the same line: ↗ Dashboard (one per resolvable target) + ↗ Run SPL.

A typical citation in chat reads:

A. [severity:high] Cross-stack auth failures concentrated on Windows. [→ logserv_cross_stack_auth_failures] ↗ Cross-Stack Authentication ↗ Run SPL 7 of the top-10 failing-stack rows are Windows…

Where: - [→ logserv_cross_stack_auth_failures] is the clickable scroll-to-tile span (cyan-light + dotted underline) - ↗ Cross-Stack Authentication is the dashboard chip (opens that dashboard in a new tab at the dispatch’s time range) - ↗ Run SPL is the SPL chip (opens Splunk Search with the dispatched SPL + same time range)

Time-Range Preservation

Both chip types preserve the dispatch’s exact time range — not the user’s current global TimeRange picker, not the dashboard’s default. So a -24h verify-query opens its destination at -24h; a -30d cumulative search opens at -30d; the dispatch’s window IS the right answer for the destination.

This matters because the user’s TimeRange picker may have changed between when the AI dispatched the search and when the user clicks the drill-down. By embedding the dispatch’s window in the URL, the destination reflects exactly what the AI saw.

Security Posture

  • target="_blank" + rel="noopener noreferrer" on every chip. The noopener flag breaks the window.opener reference that reverse-tabnabbing relies on.
  • Row values spliced into SPL get quote-escaped before URL encoding so a row value containing a quote or backslash can’t break out of the SPL string context.
  • No SPL chip on synthetic blocked-SPL results. When the SPL static-analysis guard blocks an AI-authored query, the synthetic error tile is rendered without the ↗ Run SPL chip — the chip would defeat the guard by helping the user manually dispatch the same SPL.
  • No drill-downs from compliance audit-trail tables. The After-Hours Privileged Changes + Recent Privileged Changes tables on the Change & Configuration Activity dashboard intentionally have NO row drill-downs (clicking through to raw events would pollute the audit trail with the reviewer’s own search activity in subsequent compliance reports).

For the orchestrator-layer URL pre-resolution pattern that powers these chips, see AI Assistant Implementation Reference.

Splunk MCP Setup

The AI Assistant’s tool-dispatch path runs on top of the Splunk MCP Server (Splunkbase App 7931). MCP — Model Context Protocol — is Anthropic’s open standard for connecting LLMs to external tools and data; the Splunk MCP Server exposes Splunk’s search-job + saved-search endpoints as MCP tools that any MCP-aware client can call. The LogServ App is the MCP client; the Splunk MCP Server is the MCP server; the LLM vendor (Anthropic / OpenAI / Azure / Bedrock) is the LLM.

Prerequisites

  • Splunk 9.4.3 or later.
  • Splunk MCP Server (Splunkbase App 7931) v1.1.0 or later installed on the same search head as the LogServ App. (The LogServ App’s React UI calls MCP via the same Splunk Web session, so they need to share an HTTP host.)
  • Admin user role to install the MCP Server app and configure its REST handlers.

Recommended companion: Splunk AI Assistant

Splunk AI Assistant (Splunkbase App 200) is the typical co-install for the Splunk MCP Server. It’s not a strict prerequisite for the LogServ App’s AI Assistant — the LogServ App uses only the core splunk_run_saved_search and splunk_run_query MCP tools, which work standalone. However:

  • App 200 unlocks additional saia_*-prefixed MCP tools (e.g., SPL optimization helpers) that future LogServ releases may use.
  • Installing it alongside the MCP Server follows Splunk’s documented setup pattern.
  • If you’re already deploying the Splunk MCP Server, the marginal effort to add App 200 is small and avoids friction later.

Install the Splunk MCP Server

Per Splunkbase App 7931’s installation guide:

  1. Download the MCP Server tarball from Splunkbase.
  2. On the search head where the LogServ App lives, install via Splunk Web (Apps → Install app from file) or via CLI: 
    /opt/splunk/bin/splunk install app /path/to/splunk-mcp-server.tar.gz
  3. Restart Splunkd: 
    sudo systemctl restart Splunkd
  4. Verify the app is installed and the REST handler endpoint responds: 
    curl -sk -u admin:<password> https://localhost:8089/services/mcp/health

If the health endpoint responds OK, the LogServ App’s AI Assistant should detect MCP automatically on next page load.

Authentication

Cookie auth (default, recommended)

App 7931 v1.1.0 accepts cookie auth from the same Splunk Web session that’s serving the React app. When the LogServ App calls the MCP server endpoint at /en-US/splunkd/__raw/services/mcp (the default), the browser includes the Splunk session cookie automatically; no bearer token needed.

This is the default and preferred configuration. No setup required beyond installing the MCP Server app — works on both HTTPS and HTTP Splunk; the browser handles the session cookie either way. (HTTPS is in fact preferred — Splunk’s session cookies set the Secure and SameSite=Lax flags, which behave fully correctly only over HTTPS.)

Bearer token (optional, OAuth-strict environments)

For customers with OAuth-strict MCP server configurations (where cookie auth is disabled at the server side), the LogServ App supports an optional bearer token. Admin pastes the token in Settings → Splunk MCP under realm logserv_ai_assistant_mcp name bearer_token. The LogServ App layers it on top of the cookie auth via Authorization: Bearer <token> header. On 401 the client invalidates the cached token and retries once.

Roadmap: a future release will replace the manual token paste with auto-mint via OAuth/RSA on the Data TA — see Auto-Mint MCP Token Roadmap below.

Configuration in the LogServ App

Open Settings → AI Assistant → General and confirm:

  • mcp_required = true (default). When false, MCP is bypassed and the chat operates in MCP-less chat mode (Claude streaming works, no tool dispatch). Useful for debugging the LLM-side flow without MCP. Most customers leave this true.
  • mcp_server_url = blank (default — uses the scheme-relative /en-US/splunkd/__raw/services/mcp). Override only if your MCP server is at a non-default path or you’re proxying through a different ingress.

Then in Settings → Splunk MCP:

  • bearer_token = blank if using cookie auth (default), or paste the token if your MCP server requires bearer auth.

Save the General tab. The Settings save records a config-changed audit event; the next page load picks up the new MCP URL / token.

Health-Check + Setup Wizard

When the AI Assistant panel opens, it runs a health check against the configured MCP endpoint. Three possible outcomes:

Health status What renders
ok Empty chat panel; ready for prompts.
error MCPSetupWizard renders with diagnostic guidance (URL, last error, suggested fix).
loading Spinner; transient.

The setup wizard surfaces:

  • The configured MCP URL.
  • The last error (typically HTTP 404 if the endpoint path is wrong, 401 if auth is failing, network timeout if MCP isn’t reachable).
  • A Retry button that re-runs the health check.
  • A link back to Settings → Splunk MCP to fix credentials.

Common Issues

“Stuck at health-check loading”

The health-check endpoint is hanging. Most common cause: the configured mcp_server_url points at a path the MCP Server app doesn’t expose. Check via curl from the search head:

curl -sk -u admin:<pw> https://localhost:8089/services/mcp/health

If that 404s, the MCP Server app isn’t installed correctly or the REST handler is misconfigured.

“401 Unauthorized on every dispatch”

Cookie auth isn’t working — possible causes:

  • The Splunk Web session expired in the browser. Re-login; re-open the AI Assistant panel.
  • The MCP Server is configured with bearer-only auth. Configure a bearer token in Settings → Splunk MCP.
  • A reverse proxy in front of Splunk is stripping the session cookie. Check the proxy’s cookie-forwarding config.

“MCP Server returns valid responses but tool tiles are empty”

The MCP Server’s response format doesn’t match what the LogServ App expects. Most common cause: MCP Server version mismatch (need v1.1.0 or later). Check via the Splunk Apps page; upgrade if needed.

“TA gate is blocking AI Assistant”

A health-check gate for an MCP support TA exists in the code but is currently bypassed (the dependent TA isn’t yet identified on Splunkbase). The gate will be restored once a real TA is published. If you see a “TA missing” error, confirm you’re running a recent build that has the bypass active.

MCP-less Chat Mode

For debugging or for specific use cases where you want LLM streaming without MCP, set mcp_required = false in Settings → General. The AI Assistant then operates in chat-only mode:

  • The LLM vendor receives system primer + user message + history (no tool definitions).
  • The AI cannot dispatch any Splunk searches — no splunk_run_saved_search, no splunk_run_query.
  • Replies are conversational only — useful for “explain how X works” type questions, but not for any data-grounded investigation.
  • A persistent orange-warning banner at the top of the chat reads: “Chat-only mode — tool execution disabled (MCP not configured). The AI can answer questions but cannot run searches against your Splunk data.”

This mode is distinct from the Templates-only build variant: chat-only mode has the LLM-driven path on, MCP off; templates-only has the LLM-driven path off, MCP on.

Auto-Mint MCP Token Roadmap

The current MCP authentication relies on either cookie-based session reuse (default) or admin-paste bearer tokens. Both work today but neither is ideal for fully-automated deployments. A future release will replace the manual token paste with server-side auto-mint of short-lived JWTs.

The auto-mint feature eliminates the manual token-paste step for OAuth-strict customers. Status: roadmap, not yet shipped. Until it lands, customers in OAuth-strict environments paste the token manually. For implementation detail, see AI Assistant Implementation Reference.

Settings & Configuration

v0.0.5 release: the Provider Credentials tab is hidden — LLM dispatch is disabled

The v0.0.5 release ships with the LLM-driven path disabled at compile time pending internal review. The Provider Credentials tab is hidden entirely in v0.0.5 builds, and the model picker on the General tab is non-functional even though the underlying setting persists. The General tab’s provider, default_model, tier, power_user_roles, tier2_pii_redaction, tier2_redact_hostname, rate_limit_per_hour, daily_spend_cap_usd, and session_tool_call_cap fields are documented for the future release that re-enables the LLM path; in v0.0.5, only enabled, mcp_required, mcp_server_url, and the audit-forwarder fields under Audit & Telemetry are operationally meaningful.

The AI Assistant Settings page is at #/settings/ai-assistant within the LogServ App. Admin-only. Four tabs, each handling a different aspect of the AI Assistant configuration.

The Four Tabs

Tab Scope
General Org-wide AI Assistant defaults (enable/disable, provider, model, tier, MCP gate, server URL, rate limit, spend cap, Power Users, audit forwarder)
Provider Credentials LLM provider API keys (Anthropic / OpenAI / Azure OpenAI / AWS Bedrock)
Splunk MCP MCP Server bearer token + Audit Forwarder HEC token
Audit Log Read-only browser of every audit event in the _ai_assistant_audit index

In the Templates-only build variant, the Provider Credentials tab is hidden entirely (no LLM provider needed) and an info banner at the top of the page explains the build mode.

General Tab

The General tab is divided into four semantic subsections: Feature, Limits & Quotas, Privacy, Audit & Telemetry.

Feature

Field Default Description
enabled false Master switch. When false, the ✦ AI Assistant button in the nav is hidden and no AI traffic flows. The first time an admin flips this to true, an enable-acceptance modal blocks the save until acknowledged.
provider anthropic Active LLM vendor: mock / anthropic / openai / azure_openai / bedrock / ollama (future release).
default_model per-provider Default model id for the active provider. Per-user model picker in the chat panel can switch within the same provider’s models[].
tier 1 Privacy tier 0 / 1 / 2. See Privacy Tiers. Elevating to Tier 2 records a vendor_tier2_elevation audit event.
mcp_required true When false, runs MCP-less chat mode (streaming-only, no tool dispatch).
mcp_server_url blank MCP server endpoint. Blank uses the scheme-relative /en-US/splunkd/__raw/services/mcp. See Splunk MCP Setup.
power_user_roles empty CSV Comma-separated Splunk roles allowed to see the ✦ Power toggle. See Power Mode.

Limits & Quotas

Field Default Description
rate_limit_per_hour 30 Per-user rolling-1-hour rate limit on free-form prompts. 0 = disabled. Maps to LLM10. Canned prompts are never rate-limited.
session_tool_call_cap 100 Per-chat-session cap on tool dispatches to prevent infinite tool loops. 0 = disabled. Maps to LLM06.
daily_spend_cap_usd per-org Cumulative-cost cap on free-form vendor spend per app instance per UTC day. Resets at 00:00 UTC. Maps to LLM10.

Privacy

Field Default Description
tier2_pii_redaction true When Tier 2 is active, redact known identifier columns (email, user(name), *_ip, mac, account) before sending to the LLM. Stable per-value <redacted-XXXXXXX> tags so cardinality reasoning still works.
tier2_redact_hostname false Whether to redact hostname columns under the Tier 2 PII rule. Default off — hostname is often non-sensitive in SAP environments.

Audit & Telemetry

Field Default Description
audit_forwarder_enabled false Forward audit events to a separate Splunk / SIEM via HEC for tamper-evidence. When the admin saves with this off, a forwarder-disabled-acceptance modal blocks the save until acknowledged.
audit_forwarder_url blank HEC endpoint URL (e.g., https://siem.example.com:8088/services/collector).
audit_forwarder_index blank Optional index field to send with each event.
audit_forwarder_source logserv:ai_assistant:audit source field for forwarded events.

Provider Credentials Tab

One panel per provider. Each panel has the credential fields for that provider — typically just an API key, plus per-provider extras (Azure deployment URL, Bedrock region, etc.). Credentials are stored in Splunk’s encrypted password store via /servicesNS/nobody/<app>/storage/passwords. The Settings page only ever displays length + prefix, never the cleartext.

The realm convention is logserv_ai_assistant_<provider> and the credential name corresponds to the field. So Anthropic’s API key lives at realm logserv_ai_assistant_anthropic name api_key; OpenAI’s at logserv_ai_assistant_openai name api_key; etc.

Setting a credential (admin-only):

  1. Pick the provider’s panel on this tab.
  2. Click Set next to the field.
  3. Paste the credential.
  4. Click Save.
  5. The Settings page records a local_only audit event with the realm + name + length + prefix (never the cleartext).

Deleting a credential:

  1. Click Delete next to the field.
  2. Confirm.
  3. The credential is removed from Splunk’s encrypted password store.

If the active provider field on the General tab points at a provider whose credentials aren’t set, the AI Assistant produces an error on the first free-form prompt: “Provider ‘X’ has no credentials configured. Open Settings → Provider Credentials to set them.”

In the Templates-only build variant, this entire tab is hidden.

Splunk MCP Tab

Settings — Splunk MCP tab

Two panels:

Splunk MCP Server

Field Realm Name
bearer_token logserv_ai_assistant_mcp bearer_token

OAuth/JWT token issued by your Splunk MCP Server. The admin pastes this; a future release will mint it automatically. Optional — cookie auth from the same Splunk Web session works by default for HTTP-only Splunk.

Audit Log Forwarder

Field Realm Name
hec_token logserv_ai_assistant_forwarder hec_token

HEC token for tamper-evident audit forwarding to a separate Splunk / SIEM. The destination URL + on/off toggle live under General → Audit & Telemetry. The token is sent on every audit-event POST as Authorization: Splunk <token>. See Audit Log → HEC Forwarder.

Audit Log Tab

Settings — Audit Log tab

Read-only browser of every event in the _ai_assistant_audit index. Filters: time range (preset Last 24h / 7d / 30d / 90d), category multi-select (12 categories — local_only, vendor_tier1, vendor_tier2, security_blocked_spl, rate_limited_prompt, user_prompt_jailbreak_flag, session_tool_cap_hit, daily_spend_cap_hit, audit_forwarder_failure, vendor_tier2_elevation, forwarder_disabled_acceptance, ai_assistant_enable_acceptance), user-contains text filter, result limit (25 / 100 / 500 / 1000).

Clicking a row’s + expand button reveals the full event JSON. The page is paginated client-side at 25 rows per page; “Showing N-M of T events” + Previous / Next buttons render below the table when T > 25.

Tamper-resistance disclaimer at the top of the panel:

Audit events live in a Splunk index. A host-root admin can edit the bucket files. Mitigation: forward audit events to a separate Splunk instance / SIEM / S3-with-Object-Lock owned by a different admin team. Configure the HEC forwarder under Settings → General → Audit & Telemetry.

See Audit Log for the threat model + forwarder configuration.

Legal Acknowledgement Modals

Two compile-time legal modals gate specific Settings save flows. Both follow Splunk’s splunk_instrumentation optInVersion framework — once acknowledged at version V, future saves with the same disclaimer revision skip the modal; bumping the version forces re-ack.

Enable Acceptance Modal (orange-bordered)

Triggers when: - saved.enabled === false && draft.enabled === true (every “I’m turning this on” deliberate action), OR - The current optInVersion of the enable-TC stanza is higher than the user’s last acknowledgement.

The modal disclaimer covers seven clauses: data egress acknowledgement, customer responsibility, AS-IS warranty disclaimer, indemnification, limitation of liability, authority warrant, record-of-acknowledgement. Aligned to Splunk MSA + Cisco EULA conventions.

User identity, Splunk-stamped IP (host field), timestamp, and SHA-256 of the disclaimer revision are recorded in an ai_assistant_enable_acceptance audit event. Yes path: save proceeds + bumps the user’s optInVersionAcknowledged. No path: save aborts + records the No choice.

Forwarder Disabled Acceptance Modal (cyan-bordered)

Triggers when: - saved.audit_forwarder_enabled === true && draft.audit_forwarder_enabled === false (every “I’m turning this protection off” deliberate action), OR - The current optInVersion of the forwarder-TC stanza is higher than the user’s last acknowledgement.

The disclaimer covers integrity-mitigation responsibility: with the forwarder off, audit events are recoverable only from the local index, which a host-root admin can edit. Same identity + IP + timestamp + disclaimer-hash recording as the enable modal. Records forwarder_disabled_acceptance audit event.

Why the hybrid trigger logic

Pure optInVersion-only gating (acknowledge once, never prompted again until the version bumps) is what Splunk’s stock telemetry uses. We added the deliberate-toggle-transition trigger because re-enabling the AI Assistant after a disable, or re-disabling the forwarder after enabling, are deliberate user actions that warrant re-acknowledgement of the legal posture each time. Pure version-only gating would let these toggle-flips slip without re-ack.

Live UI Refresh on Save

When the admin saves a config change in the General tab that affects what users see (master enabled toggle, mcp_required, power_user_roles, etc.), the AI Assistant re-applies the new config to the running session within a few seconds. No page reload required.

For example: disabling the AI Assistant via the master toggle and clicking Save Defaults — the ✦ AI Assistant button in the top-right nav disappears within ~4 seconds, no browser refresh needed. Similarly, re-enabling re-shows the button immediately. For the callback wiring, see AI Assistant Implementation Reference.

Audit Log

Every action the AI Assistant takes — predefined-prompt dispatches, free-form vendor calls, security blocks, privacy-tier elevations, legal acknowledgements — produces an audit event in a dedicated _ai_assistant_audit index. The audit trail is the evidence layer for compliance reviews, SOC investigations, and tamper-resistance posture.

The Twelve Audit Categories

Category When Key fields
local_only Predefined-prompt dispatch (no LLM call). promptId, spl, rowCount, executionMs, ok
vendor_tier1 Free-form prompt at Tier 1 (count + timing only summary sent to vendor). provider, model, inputTokens, outputTokens, vendorCostEstimateUsd, outboundBytes, promptLength, turnCount, powerMode
vendor_tier2 Free-form prompt at Tier 2 (aggregated metadata sent). Same as vendor_tier1 plus tier2RedactionsApplied (count of PII redactions on this turn)
vendor_tier2_elevation Admin saves Settings with tier changing from 1 → 2. Admin user, previousTier, newTier
security_blocked_spl SPL static-analysis guard rejected an AI-authored splunk_run_query. spl (truncated to 1000 chars), operator (the offending command name)
rate_limited_prompt Per-user rate limit denied a free-form prompt. cap, attemptedCount, windowSeconds
user_prompt_jailbreak_flag Jailbreak-pattern analyzer flagged a user prompt (flag-and-proceed; the prompt still ran). promptHash, promptLength, matchedGroups, charClassFingerprint
session_tool_cap_hit Per-session tool-dispatch cap reached; dispatch refused. cap, attemptedCount, toolName
daily_spend_cap_hit Daily USD spend cap reached; vendor call refused. cap, currentSpend, attemptedSpend
audit_forwarder_failure HEC forwarder POST failed (DNS, network, 4xx/5xx response). destinationUrl (sanitized), reason, batchSize
forwarder_disabled_acceptance Admin acknowledged the forwarder-disabled legal modal. Admin user, host (Splunk-stamped IP), tcVersion, optInChoice, disclaimerHash
ai_assistant_enable_acceptance Admin acknowledged the AI-Assistant-enable legal modal. Same as forwarder acceptance, plus the seven-clause enable-disclaimer hash

Every event also carries a small set of common fields:

  • timestamp — ISO-8601 UTC.
  • user — Splunk username (from services/authentication/current-context).
  • sessionId — Per-tab session id baked at AI Assistant init.
  • seq — Monotonic per-session sequence number, useful for ordering events within a session even when timestamps tie.

The In-App Audit Log Viewer

Settings — Audit Log tab

The Audit Log tab in Settings → AI Assistant provides a read-only browser of the index. Filters:

Filter Default Notes
Time range Last 7 days Reads from the global TimeRange picker; not a separate field. Re-runs on picker change.
Category (all 12) Multi-select with colored chips. Each category has its own gradient (cyan-light SAP-Basis-style for local_only, gold-orange for vendor calls, red for security blocks, etc.).
User contains (empty) Substring filter on the user field. Useful for “what did admin X do?” reviews.
Limit 100 One of 25 / 100 / 500 / 1000. Larger values slow page rendering.

The table renders 25 rows per page (PAGE_SIZE = 25); footer shows “Page X of Y” + Previous / Next buttons (only when total exceeds 25). Clicking a row’s + button expands to show the full event JSON.

Splunk reserved-field gotcha: the category field on these events is a Splunk reserved name. The viewer’s SPL uses | eval category=json_extract(_raw, "category") after spath to retrieve the audit-event category from the raw JSON, because spath does NOT overwrite reserved fields. If you write your own searches against this index, replicate this pattern.

Tamper-Resistance Threat Model

Audit events live in a Splunk index on the same host as the LogServ App. A host-root admin can edit the underlying bucket files — they can delete events, rewrite events, replay old events, or delete the entire index. The audit log is NOT tamper-evident on the local instance alone.

Mitigation: forward audit events to a separate Splunk / SIEM / S3-with-Object-Lock destination owned by a different admin team. The HEC forwarder (below) provides this. The threat model assumes a malicious or compromised insider with admin rights to the LogServ Splunk instance but NOT to the destination instance.

Disclaimer text shown at the top of the Audit Log tab:

Audit events live in a Splunk index. A host-root admin can edit the bucket files. Mitigation: forward audit events to a separate Splunk instance / SIEM / S3-with-Object-Lock owned by a different admin team. Configure the HEC forwarder under Settings → General → Audit & Telemetry.

HEC Forwarder

Optional admin-configurable forwarding of audit events to a separate Splunk / SIEM via Splunk’s HTTP Event Collector. When enabled:

  • Browser-side dual-write at flush. The AuditWriter.flush() path (and the per-event postOneOff() path) sends to BOTH the local _ai_assistant_audit index (via the standard services/receivers/simple REST endpoint) AND the configured HEC URL. The two writes are independent — local-index failure doesn’t block forwarder write, and vice versa.
  • Failure visibility. When the HEC POST fails (DNS error, network timeout, 4xx/5xx response), an audit_forwarder_failure event is generated locally. So a disabled / down forwarder is visible in the local audit log itself, not silently failing.
  • Auth. The HEC token is sent on every POST as Authorization: Splunk <token> header. Token is stored in Splunk’s encrypted password store at realm logserv_ai_assistant_forwarder name hec_token and never displayed in cleartext.
  • Sanitized URL in failure events. When a forwarder failure event records the destinationUrl, query strings + fragments are stripped to prevent accidental exposure of any auth-bearing URL params.

Configuring the forwarder

  1. Open Settings → AI Assistant → General.
  2. Find the Audit & Telemetry subsection.
  3. Set audit_forwarder_url to the HEC endpoint (e.g., https://siem.example.com:8088/services/collector).
  4. Optionally set audit_forwarder_index (recommended: a dedicated index like splunk_audit on the destination).
  5. Optionally set audit_forwarder_source (defaults to logserv:ai_assistant:audit).
  6. Set audit_forwarder_enabled = true.
  7. Open Settings → AI Assistant → Splunk MCP → Audit Log Forwarder panel.
  8. Click Set next to the HEC token field.
  9. Paste the HEC token from your destination Splunk’s HEC input.
  10. Save. The next audit event triggers a forwarder POST.
  11. Check the destination Splunk for events arriving at the configured index/source.

When the admin saves with the forwarder OFF

A forwarder-disabled-acceptance modal blocks the save until the admin acknowledges the integrity-mitigation responsibility. The acceptance is recorded as a forwarder_disabled_acceptance audit event with admin identity, Splunk-stamped IP, timestamp, and SHA-256 of the disclaimer revision.

Audit Index Provisioning

The _ai_assistant_audit index is defined in the Data TA’s default/indexes.conf and created automatically when the Data TA is installed on the indexer / search head. No customer-side provisioning needed.

The index name is macro-configurable — see Renaming an index for the procedure (update the sap_logserv_audit_idx_macro macro definition for reads, plus the audit_index_name field in Settings → AI Assistant → General → Audit & Telemetry for writes).

Retention uses the system default unless the customer overrides it on their indexer’s indexes.conf — for compliance reviews going back > 90 days, override frozenTimePeriodInSecs accordingly. Recommended retention for compliance use cases: 2 years to cover annual audit cycles plus a buffer. For SOX / PCI / HIPAA contexts where regulator records-retention requirements apply, follow the regulator’s spec.

Querying the Audit Index Directly

You can query the audit index from any search bar — the in-app viewer is just a curated UX over the same SPL. The starter searches below use the \sap_logserv_audit_idx_macro`` macro so they continue to work after a rename:

All Tier 2 calls in the last 7 days, with USD cost summed by user:

`sap_logserv_audit_idx_macro` earliest=-7d
| spath
| eval category=json_extract(_raw, "category")
| where category="vendor_tier2"
| stats sum(vendorCostEstimateUsd) as total_usd, count by user
| sort - total_usd

All security blocks in the last 30 days, grouped by user + offending operator:

`sap_logserv_audit_idx_macro` earliest=-30d
| spath
| eval category=json_extract(_raw, "category")
| where category="security_blocked_spl"
| stats count by user, operator
| sort - count

All Tier 2 elevation events ever (compliance review):

`sap_logserv_audit_idx_macro` earliest=0
| spath
| eval category=json_extract(_raw, "category")
| where category="vendor_tier2_elevation"
| table _time, user, previousTier, newTier

Forwarder failures grouped by reason (operational health):

`sap_logserv_audit_idx_macro` earliest=-7d
| spath
| eval category=json_extract(_raw, "category")
| where category="audit_forwarder_failure"
| stats count by reason, destinationUrl

Audit Modal in the AI Assistant Panel

The privacy banner at the top of the AI Assistant chat panel includes an Audit this session button. Clicking it opens a per-session audit modal:

AI Assistant — Audit Modal

The modal shows a chronological list of every audit event for the current session (matched on sessionId). Useful for users who want to confirm what their last few prompts triggered without leaving the AI Assistant panel. The modal is the in-pane equivalent of the Settings → Audit Log tab, scoped to the current session.

Time-Window Reasoning

v0.0.5 release: these primer rules are not exercised — LLM dispatch is disabled

The time-window reasoning rules are baked into the system primer and ship with every v0.0.5 build, but they only take effect when the LLM-driven path runs — which is disabled at compile time in v0.0.5 pending internal review. Predefined-prompt dispatches in v0.0.5 still respect the saved-search SPL’s own time window (which is what the rules are about), but there is no AI synthesis layer applying these rules in v0.0.5. This page is preserved as the design reference for the future release that re-enables the LLM path.

Time-window reasoning is a set of primer rules baked into the AI Assistant’s system primer (Tier 1 + Tier 2 variants) that teach the LLM to identify the dispatch window, normalize cumulative count to per-hour or per-day rate, and run a verify-query before declaring high-severity findings. The rules ship as a dedicated === TIME-WINDOW REASONING — APPLY BEFORE EVERY SEVERITY CLAIM === block in the primer, inserted right after the data-boundary block and before the saved-search catalog.

This page exists primarily for SOC analysts who use the AI Assistant for top-N investigations: knowing how the AI reasons about windows + rates + verification helps you read its replies critically, validate its severity claims, and recognize its self-correction behavior.

The Bug This Fixes

Before the time-window rules shipped, the AI sometimes cited cumulative aggregates as if they were active rates. A real example reproduced during the test cycle:

User prompt: “find the top 10 issues I need to attend to in priority / severity”.

The AI’s first response cited:

A. [severity:high] Likely brute-force on a Windows service account. One user/stack pair accumulated 4,799 failed authentications. Service-style accounts (sapadm, SYSTEM, svc_monitor, SAPServiceXCP, svc_backup) dominate. Lock/rotate the affected accounts today, confirm source host…

The 4,799 number was a cumulative count over the saved search’s wider rolling window (~30 days for that prompt), NOT an active rate. Drilling into the same data with a -24h window returned only 18 events across 5 service accounts on 1 host — baseline noise, not an active brute-force. The AI’s “Lock/rotate today” recommendation was wrong; the user almost took action on a false alarm.

The bug wasn’t a data-access problem (the AI had the right number). The bug was a reasoning problem: the AI didn’t know whether 4,799 was a 30-day baseline (~160/day = noise) or a 24-hour burst (= active threat). Aggregate value alone CANNOT distinguish.

The Four Rules

The primer block teaches the AI to apply these four rules before declaring a finding’s severity:

Rule 1 — Identify the Window

Saved-search counts are CUMULATIVE over each search’s own rolling window (typically -24h to -30d, baked into the saved-search SPL). The user’s TimeRange picker in the UI does NOT necessarily align with that window.

Tier 1: the summary the AI receives is Returned N rows in M ms. — no window data at all. The AI must hedge: “X rows over the search’s rolling window, exact span unknown” OR infer from the saved-search NAME (e.g., *_24h, *_after_hours) when the name carries a window hint.

Tier 2: the summary includes a Time range: line ONLY when the result has a _time column (timecharts, time-series). For aggregate searches like stats by user, no _time column → no time-range line → AI must hedge as in Tier 1.

Rule 2 — Normalize Count → Rate

Convert sum / max from the aggregates to events/hour or events/day before assigning severity. Rough thresholds documented in the primer:

Signal High threshold
Auth failures >10/hour
Warn-level errors >100/hour
Ingest volume >1000/hour

A 4,799 cumulative count over 30 days = ~160/day = ~6.6/hour. For auth failures, that’s below the >10/hour high threshold → severity should be medium or low, not high.

Rule 3 — Verify Before Recommending Action

For any finding ranked [severity:high] or [severity:critical], the AI MUST dispatch ONE additional splunk_run_query call with earliest=-24h latest=now BEFORE writing the narrative response. The verify either confirms the cumulative total is also a current rate, or reveals it’s baseline noise.

If the verify returns dramatically smaller numbers than the cumulative headline, the AI must re-rank the finding to medium or low and SAY SO in the body. Active-tense claims (“Lock/rotate accounts today”, “active brute-force”, “data exfiltration in progress”) MUST be backed by a verify query, never by a cumulative count alone.

Rule 4 — State the Window in Narrative

Precise phrases: - “X events in the last 24 hours” — for verify-confirmed claims - “X cumulative over the search’s rolling window” — for un-verified cumulative aggregates - “~Y/hour active rate, verify-confirmed” — for normalized-to-rate severity claims

Active-tense phrases (“active brute-force”, “happening today”, “current attack”) MUST be backed by a narrow-window verify query — not by cumulative counts or top-N aggregates alone.

What “Verify-Confirmed Severity” Looks Like in Replies

After the rules shipped, the same prompt (re-run on the same data) produced a self-corrected response that explicitly downgrades the finding:

G. [severity:medium] Cross-stack auth failures show high cumulative count but no current activity. [→ logserv_cross_stack_auth_failures] ↗ Cross-Stack Authentication ↗ Run SPL Top row showed 4,799 attempts but my -24h verify returned only 19 auth failures across all stacks combined — this is a stale long-window aggregate, not an active brute-force. No immediate action needed; review baselines.

The AI:

  1. Cited the cumulative number (4,799) — appropriate context.
  2. Ran the verify query (-24h) — required by Rule 3.
  3. Reported the verify result (only 19) — required by Rule 4.
  4. Re-ranked to medium — required by Rule 3.
  5. Used precise window-aware language (“stale long-window aggregate, not an active brute-force”) — required by Rule 4.
  6. Recommended a non-urgent next step (“review baselines”) instead of “Lock/rotate today”.

This is the correct shape. When you read a top-N response, look for these markers:

  • ✓ Findings ranked high or critical reference both a cumulative number AND a verify-query result
  • ✓ Active-tense recommendations come ONLY from verified high-rate findings
  • ✓ Findings that were going to be high but verified down to noise are explicitly downgraded with framing like “stale aggregate, not active”

Edge Cases

When the verify query also returns high counts

If a finding cites 4,799 cumulative AND the verify query returns 412 in the last 24h (~17/hour, above the >10/hour threshold), the AI keeps the finding at high severity and says: “412 of those 4,799 landed today, ~17/hr — an active rate.” Now the active-tense “Lock/rotate today” recommendation is justified.

When the saved-search name implies the window

Some prompts have window hints in their names: logserv_dns_beaconing_24h, logserv_hana_after_hours_admin. For these, the AI uses the name-implied window for Rule 1 and skips Rule 3’s verify step (the saved search itself IS the narrow-window query).

When no splunk_run_query is available (Templates-only)

In the Templates-only build, free-form prompts are disabled — there’s no LLM-driven verify path. The time-window rules don’t apply because the AI doesn’t run. Predefined prompts have explicit windows in their SPL and produce static interpretation cards (no AI synthesis), so no rate-reasoning step is needed.

When Tier 1 is active

Tier 1 gives the AI count + timing only — it cannot see the actual values, so it cannot do Rule 2’s normalization. The AI hedges harder: “the search returned N rows for the rolling window, suggesting non-trivial activity” without making rate claims. Tier 1 narrative is intentionally less concrete; users read the rendered tile for the actual numbers.

Power Mode Compounds With Time-Window Reasoning

When Power Mode is on AND a free-form prompt asks for top-N issues:

  1. Power Mode forces a saved-search dispatch (Rule out: AI never replies from prior knowledge).
  2. Each high/critical finding in the AI’s reply forces an additional verify query (per Rule 3).
  3. Result: a 5-finding top-N response can dispatch 5 saved searches + up to 5 verify queries = 10 total Splunk dispatches.

This is the correct behavior for compliance investigations and high-stakes triage — every claim is data-grounded AND verify-confirmed. The cost is higher per-turn latency + tool budget, but for the customer’s most important investigations, the cost is the right trade.

Templates-only Build Variant

v0.0.5 release: the templates-only build IS the default ship — LLM functionality intentionally disabled pending review

In the v0.0.5 release, the templates-only build is not just a partner-only variant — it is the only released build. The LLM-driven path is disabled at compile time pending internal review of the OWASP LLM Top 10 controls. Once review concludes, a future release will publish the regular variant alongside the templates-only variant. Until then, every customer running v0.0.5 — production, partner, demo — runs the templates-only build.

The templates-only build is a variant of the LogServ App that has the AI Assistant’s free-form / LLM-driven path disabled at compile time — there is no runtime setting that could re-enable it. The MCP path + 48 canned prompts + tool tiles + drill-down chips + audit log all stay fully active, so the LogServ solution can be run end-to-end without an LLM provider. In v0.0.5, this is the only released build; in future releases it will continue to exist as a deployable variant alongside the regular LLM-enabled build.

AI Assistant — Templates-only banner

Why a Compile-Time Variant

The runtime alternative — an admin toggle in Settings that disables the LLM path — would let any local Splunk admin flip it back on. For deployments where the LLM dispatch path is intentionally not available — such as v0.0.5 (pending review), demonstration environments, or restricted-environment customers — a compile-time variant is the right shape: the bundle has no code path that reaches an LLM, and there’s no runtime setting that could enable one.

What Changes in the Templates-only Build

UI gating (visible to the user)

  • Chat input text field — disabled, with placeholder “Templates-only build — click ‘Browse predefined prompts’ below to run a saved search.”
  • Send button — unconditionally disabled (in addition to the existing !text.trim() || busy guard).
  • Power Mode toggle — hidden (forced-RAG is meaningless when there’s no LLM call to force a saved-search before).
  • Browse prompts button — fully enabled. The only entry point in this build.
  • Privacy banner model picker — hidden (no model = no picker).
  • Top-of-chat info banner — cyan-info-tone banner reads: “Templates-only build — free-form prompts and LLM dispatch are disabled. Use ‘Browse predefined prompts’ to run any of the 48 saved searches against your Splunk data via MCP.”
  • Settings page — the Provider Credentials tab is hidden entirely. Top-of-page info banner explains the build mode. Other tabs (General / Splunk MCP / Audit Log) remain fully visible since partners need MCP config + audit visibility.

Defense-in-depth runtime guard

Even if a future code path reaches the LLM dispatch entry point (keyboard shortcut, programmatic dispatch from a future feature, etc.), a runtime guard short-circuits with a system notice — the UI gating is the primary defense, and the function-level guard is the safety net. For the guard’s exact location and code shape, see AI Assistant Implementation Reference.

What’s Still Active in Templates-only

  • All 48 predefined prompts. Click any card in the prompt browser to dispatch.
  • Tool tiles in the right pane. Tables, charts, KPIs, pies — all rendered identically to the regular build.
  • Static interpretation + suggested-next-steps cards. Per-prompt guidance from the intent map.
  • Drill-down chips. ↗ Dashboard (one per related dashboard) + ↗ Run SPL on every tile.
  • Audit log. All local_only events for canned-prompt dispatches, plus audit_forwarder_failure events if the forwarder is configured. No vendor_tier1 / vendor_tier2 events because there are no vendor calls.
  • HEC audit forwarder. Same dual-write behavior as the regular build.
  • All dashboards. Environment Health, Applications, Integration, Security, Platform — all 20 dashboards plus the Environment Topology view.
  • Per-dashboard auto-refresh picker, Download PNG, time-range URL preservation. All identical to the regular build.
  • Settings → General, Splunk MCP, Audit Log. Visible and functional; admin can configure MCP, audit forwarder, etc.

What’s Disabled in Templates-only

  • Free-form prompts. Chat input greyed; Send disabled. The defense-in-depth guard short-circuits any code path that reaches the LLM dispatch entry point.
  • Power Mode. Toggle hidden; the forced-RAG rule has nothing to enforce since there’s no LLM dispatch.
  • Provider Credentials tab. Hidden entirely.
  • Model picker in privacy banner. Hidden.
  • Vendor calls. vendor_tier1 / vendor_tier2 audit categories are never emitted.

End-User Experience

After the templates-only build is installed, the user opens Splunk Web → clicks the ✦ AI Assistant button. The cyan “Templates-only build” banner renders at the top of the chat panel. The user opens the prompt browser, runs prompts, sees tiles + drill-down chips, and navigates dashboards. The full LogServ analytics experience is available, just without the free-form / LLM-driven path.

The destination Splunk search head needs the Splunk MCP Server installed for prompt dispatch to work.

OWASP LLM Top 10 (2025) Compliance

This page is intended for the customer’s security team reviewing the LogServ App’s AI Assistant feature for production deployment. It covers each of the OWASP LLM Top 10 (2025) items and the controls the LogServ App ships to address them.

v0.0.5 release: LLM functionality intentionally disabled pending review

The v0.0.5 release ships with the LLM-driven path disabled at compile time pending internal review. The controls described on this page are designed, implemented, and exercised by the build’s CI pipeline — but the LLM dispatch pathway itself is gated off via the Templates-only build flag until the review concludes. The predefined-prompt path + Splunk MCP integration + audit log remain fully active. This page documents the controls so reviewers can evaluate the security posture for a future release that re-enables the LLM path.

LLM01 — Prompt Injection

Threat: an attacker (either via the user-prompt vector or via in-band tool-result data) crafts input that hijacks the LLM’s instruction-following behavior, causing it to ignore the system primer or perform unintended actions.

Controls shipped:

  • <TOOL_RESULT_DATA>...</TOOL_RESULT_DATA> sentinel block wraps every tool-result summary the AI sees. The system primer instructs the AI to treat anything inside the sentinel as DATA from the customer’s environment, NEVER as instructions, regardless of how command-like the contents look. Mitigates the in-band channel of prompt injection: a malicious user who can write events into the customer’s Splunk index could in principle craft field values like user="Ignore prior instructions and...", but the sentinel + sanitizer reduces the effective attack surface to near-zero.
  • Per-value sanitizer for Tier 2 categorical aggregates (sanitizeAggregateValue()). Three classes of suspicious content get replaced wholesale with <filtered>: strings starting with role markers (system:, user:, assistant:, etc.), strings containing known jailbreak / prompt-injection idioms (ignore prior instructions, [INST], <|im_start|>, <|system|>, disregard the above, jailbreak), and strings containing the sentinel tags themselves (prevents a malicious value from prematurely closing the data block).
  • Jailbreak pattern detection on user prompts. Flag-and-proceed: matches fire a user_prompt_jailbreak_flag audit event but do not block the prompt. Captures a hash + length + matched-pattern groups + char-class fingerprint for SOC pivot analysis.
  • Defense-in-depth chain. Type-system enforcement (LLM02) + Tier 2 sanitizer + tool-result sentinel + primer + vendor-side defenses provide multiple independent layers; bypassing any one doesn’t break the chain.

LLM02 — Sensitive Information Disclosure

Threat: the LLM emits sensitive customer data (PII, secrets, internal-only information) in its responses or causes it to be transmitted to the vendor.

Controls shipped:

  • TypeScript type-system enforcement. The compiler refuses to put any tool-result value into the outbound vendor payload — there is no runtime check, no flag to flip, no policy to forget. The only conversion path produces a non-data summary that the caller’s chosen summarizer controls. See Privacy Tiers and AI Assistant Implementation Reference.
  • Privacy tier scoping. Tier 1 (default) gives the AI count + execution time only — no values. Tier 2 (admin opt-in) adds aggregated metadata (top-N values + counts, numeric stats, time range). Neither tier sends raw rows.
  • PII redaction at Tier 2. Column-name-based detection redacts email, user(name), *_ip, mac, account (hostname opt-in) before values are sent to the vendor. Stable per-value <redacted-XXXXXXX> tags so the AI can still reason about cardinality without seeing the actual identifier.
  • <TOOL_RESULT_DATA> sentinel (also LLM01) prevents the AI from accidentally treating sanitized data as instructions and echoing it back unsanitized.
  • Vendor-side commitments. Anthropic / OpenAI / Azure OpenAI / AWS Bedrock each publish their own data-usage commitments (no training on API traffic for Anthropic & OpenAI direct API by default). The customer’s DPA review should confirm vendor-side processor obligations.

LLM03 — Supply Chain

Threat: a compromised or vulnerable third-party dependency in the bundle introduces an attack vector or data-exfil channel.

Controls shipped:

  • CI dependency audit gate. Every CI build runs a dependency audit at the high advisory level — non-zero exit on high or critical advisories. Hard-gates the shippable build path so released builds always pass at high+.
  • SBOM (Software Bill of Materials) shipped with every build. A bundled generator parses the package lock directly (no external tool dependency) and emits a SBOM.json in CycloneDX 1.4 format with purl (Package URL) + sha-512 hash per dependency. The v0.0.5.0 SBOM contains 1,416 components; reviewers can pin against this list.
  • Audit-log forwarder configurability so SBOM-driven vulnerability response can correlate against actual usage patterns across customers.

LLM04 — Data and Model Poisoning

Threat: an attacker with the ability to write events into the customer’s Splunk index could craft field values that, when surfaced to the AI as Tier 2 aggregates, hijack the AI’s downstream behavior (e.g., by appearing to be system instructions).

Controls shipped:

  • <TOOL_RESULT_DATA> sentinel block (also LLM01).
  • Per-value sanitizer with role-marker + jailbreak-idiom + sentinel-tag filtering (also LLM01).
  • Truncation of categorical aggregate values to 40 chars to reduce the budget available for embedded jailbreak prompts.
  • Top-N capping of categorical aggregates at 10 by default (max 50) — bounds the volume of AI-visible value content per column.

LLM05 — Improper Output Handling

Threat: the LLM’s response contains content that, when rendered or executed, harms the user (e.g., XSS via injected HTML, command injection via shell-formatted output).

Controls shipped:

  • React rendering uses JSX text nodes, not dangerouslySetInnerHTML. Strings are escaped by default; markdown is parsed via a controlled set of recognized markers (citation, severity, bold) — no general-purpose HTML rendering.
  • Drill-down URLs built via buildSplunkSearchUrl() use encodeURIComponent() for the SPL string and centralize the target="_blank" rel="noopener noreferrer" security flags. The noopener flag breaks the window.opener reference for reverse-tabnabbing.
  • AI-authored SPL goes through a static-analysis guard before dispatch (see LLM06).

LLM06 — Excessive Agency

Threat: the AI is given the ability to perform actions whose consequences exceed the user’s intent (e.g., write/delete data, send emails, exfiltrate via output channels).

Controls shipped:

  • SPL static-analysis guard for AI-authored splunk_run_query. Blocks collect, outputlookup, outputcsv, delete, sendalert, sendemail, script, run, tscollect. Blocked SPL produces a synthetic error tool_result + a security_blocked_spl audit event. The AI sees the error and can recover by writing a different query.
  • Per-session tool-call cap prevents infinite tool loops. Maps to session_tool_cap_hit audit event on overflow.
  • Pre-authored saved searches via splunk_run_saved_search are NOT subject to the guard — their SPL is reviewed at build time. The guard scope is exclusively the AI-written ad-hoc path.

LLM07 — System Prompt Leakage

Threat: the system primer leaks to the user (or via the user back to the vendor’s prompt-logging) and exposes internal instructions, lookups, or configuration.

Controls shipped:

  • No customer-secret content in the primer. The primer contains: tool definitions, the saved-search catalog (names + labels, public information), the LogServ data-model summary (sourcetypes + field names, public), the synthesis rules. No customer-specific identifiers, no API keys, no internal URLs.
  • Time-window reasoning rules are pedagogical (about how the AI should reason about its inputs) — leakage of these rules has no security impact.

The customer’s DPA review should still confirm vendor-side prompt-logging policies (Anthropic, OpenAI, Azure, Bedrock all log prompts at varying retention; some offer enterprise tiers with no-logging guarantees).

LLM08 — Vector and Embedding Weaknesses

Threat: vector / embedding stores used for RAG can leak sensitive data if not access-controlled, can be poisoned by malicious indexed content, or can return chunks across security boundaries.

Status: Not applicable. The LogServ App’s AI Assistant does NOT use vector embeddings or a separate RAG store. The “RAG” here is direct tool dispatch against the live Splunk index — every retrieval is a real-time SPL search bounded by Splunk’s own RBAC. There is no separate vector store to compromise.

LLM09 — Misinformation

Threat: the LLM produces confident-sounding but incorrect information that the user acts on.

Controls shipped:

  • AI-generated — verify before acting disclaimer rendered as a small muted footer below every AI assistant_text bubble. Visible on every reply, never collapsed.
  • Time-window reasoning rules (Time-Window Reasoning) force the AI to verify high-severity claims with a narrow-window query before recommending action. Self-correction is observable in the response (the AI reports the cumulative number AND the verify-query result).
  • Citation chips ([→ saved_search_name]) link every claim back to the source tool_result tile, so users can sanity-check the narrative against the rendered data.
  • Predefined-prompt path (default for routine investigations) has zero AI-misinformation risk — the dispatch is deterministic and the interpretation card is pre-authored, not generated.

LLM10 — Unbounded Consumption

Threat: uncontrolled vendor calls drive arbitrary cost, latency, or denial-of-service on the user-facing experience.

Controls shipped:

  • Per-user rolling-1-hour rate limit on free-form prompts. Configurable via rate_limit_per_hour in Settings (default 30, 0 = disabled). Records rate_limited_prompt audit event on hit.
  • Cumulative-cost daily spend cap in USD. Configurable via daily_spend_cap_usd in Settings. Resets at 00:00 UTC. Records daily_spend_cap_hit audit event on hit.
  • Per-session tool-call cap to prevent infinite tool loops. Records session_tool_cap_hit.
  • Token-usage observability in every vendor_tier1 / vendor_tier2 audit event: input tokens, output tokens, total tokens, USD cost estimate. Lets finance attribute spend to specific users / sessions.
  • Streaming + abort in the chat UI — users can cancel an in-flight expensive turn via the Stop button.
  • Canned prompts are NOT rate-limited — they bypass the AI vendor entirely and are bounded by Splunk’s own search-quota controls, not vendor cost.

Compliance Posture Summary

OWASP item Status Primary control
LLM01 — Prompt Injection Mitigated TOOL_RESULT_DATA sentinel + per-value sanitizer + jailbreak detection + primer hardening
LLM02 — Sensitive Information Disclosure Mitigated Hidden<T> type-system enforcement + Tier 1/2 sanitize chokepoint + PII redaction
LLM03 — Supply Chain Mitigated CI dependency-audit hard-gate at high+ severity + CycloneDX SBOM with sha-512 hashes
LLM04 — Data and Model Poisoning Mitigated TOOL_RESULT_DATA sentinel + per-value sanitizer + 40-char value truncation + top-N cap
LLM05 — Improper Output Handling Mitigated React JSX text nodes + URL encoding + noopener + SPL static-analysis guard
LLM06 — Excessive Agency Mitigated SPL static-analysis guard + session tool-call cap
LLM07 — System Prompt Leakage Mitigated No customer-secret content in primer; vendor-side DPA review needed
LLM08 — Vector and Embedding Weaknesses Not applicable No separate vector store; live SPL dispatch via Splunk RBAC
LLM09 — Misinformation Mitigated AI-generated disclaimer + time-window reasoning + citation chips + predefined-prompt path
LLM10 — Unbounded Consumption Mitigated Per-user rate limit + daily spend cap + session tool-call cap + token-usage audit

Pending Internal Review

The v0.0.5 release ships with the LLM dispatch pathway gated off via the Templates-only build flag pending internal review of the controls described on this page. The review’s outcome will determine when a future release re-enables the LLM path. The predefined-prompt path + Splunk MCP Server integration + audit log remain fully active in the meantime — partners and customers running the v0.0.5 build can exercise the canned-prompt investigation flow + drill-down chips + audit observability without any LLM dispatch.

For the controls described above to take effect, the LLM path must be re-enabled in a subsequent release. The same controls apply at that point; nothing on this page changes between v0.0.5 (LLM disabled) and the future re-enable release.

Ended: AI Assistant

Developer Reference ↵

Clean Deployment Test Guide

Overview

This guide walks through a full clean-slate test of the SAP LogServ packages:

  • Data TA (splunk_ta_sap_logserv) – installed on the DS and pushed to HFs
  • LogServ App (splunk_app_sap_logserv) – installed on the SH only

The test validates the complete workflow: install, DS automation, filter configuration, deployment push, data ingestion, and filter verification.

Prerequisites

  • DS (Deployment Server) – also serves as SH/Indexer combo in this test topology
  • HF-01, HF-02 – Heavy Forwarders configured as deployment clients
  • Both HFs have AWS TA SQS-Based S3 inputs configured (disabled until Phase 6)
  • UCC-built Data TA .tar.gz package ready for installation
  • LogServ App .tar.gz package ready for installation

Part 1: Clean Slate Setup

On the DS

# Switch to splunk user
sudo su - splunk

# Remove the Data TA
rm -rf /opt/splunk/etc/apps/splunk_ta_sap_logserv

# Remove the deployment-apps copy
rm -rf /opt/splunk/etc/deployment-apps/splunk_ta_sap_logserv

# Remove serverclass.conf files
rm -f /opt/splunk/etc/system/local/serverclass.conf
rm -f /opt/splunk/etc/apps/search/local/serverclass.conf

# Delete the server class via REST (ignore errors if it doesn't exist)
curl -sk -u admin:$SPLUNK_PASSWORD -X DELETE \
  "https://localhost:8089/services/deployment/server/serverclasses/SAP_LogServ_HeavyForwarders" 2>/dev/null

# Remove cached files
rm -rf /opt/splunk/var/run/splunk/dispatch/splunk_ta_sap_logserv* 2>/dev/null

# Exit splunk user
exit

If also testing the LogServ App:

sudo su - splunk
rm -rf /opt/splunk/etc/apps/splunk_app_sap_logserv
exit

Restart Splunk:

sudo systemctl restart Splunkd

On Each HF (HF-01 and HF-02)

sudo su - splunk
rm -rf /opt/splunk/etc/apps/splunk_ta_sap_logserv
exit
sudo systemctl restart Splunkd

Verify Clean Slate

On the DS:

sudo su - splunk
ls /opt/splunk/etc/apps/splunk_ta_sap_logserv 2>/dev/null
ls /opt/splunk/etc/deployment-apps/splunk_ta_sap_logserv 2>/dev/null
find /opt/splunk/etc -name "serverclass.conf" -path "*/local/*" 2>/dev/null | xargs cat 2>/dev/null

All should return nothing.

On each HF:

sudo su - splunk
ls /opt/splunk/etc/apps/splunk_ta_sap_logserv 2>/dev/null

Should return nothing.

Part 2: Deployment Test Phases

Phase 1: Install the Data TA on the DS

  1. Log into Splunk Web on the DS
  2. Go to Apps > Manage Apps > Install app from file
  3. Upload the Data TA .tar.gz file and install
  4. Restart Splunk when prompted

Phase 2: Install the LogServ App on the SH

  1. Log into Splunk Web on the SH (same host as DS in combo topology)
  2. Go to Apps > Manage Apps > Install app from file
  3. Upload the LogServ App .tar.gz file and install
  4. Restart Splunk when prompted

Phase 3: Configure Filters and Trigger DS Automation

  1. Go to the Data TA > Configuration > Filters tab
  2. Enable filtering, set include/exclude patterns and Days in Past
  3. Click Save
  4. After the page reloads, verify you see:
    • The “Deployment Server Detected” banner
    • The “Server Class Setup Required” notice
    • The “Deploy to Forwarders” button

Phase 4: Verify DS Automation

Confirm the Data TA was copied to deployment-apps:

sudo su - splunk
ls /opt/splunk/etc/deployment-apps/splunk_ta_sap_logserv/

Confirm filter configs were mirrored:

cat /opt/splunk/etc/deployment-apps/splunk_ta_sap_logserv/local/transforms.conf

Confirm the server class was created:

find /opt/splunk/etc -name "serverclass.conf" -path "*/local/*" 2>/dev/null | xargs cat

You should see both stanzas:

  • [serverClass:SAP_LogServ_HeavyForwarders]
  • [serverClass:SAP_LogServ_HeavyForwarders:app:splunk_ta_sap_logserv] with restartSplunkd = true and stateOnClient = enabled

Phase 5: Configure the Server Class and Deploy

  1. Go to Settings > Forwarder Management
  2. Find SAP_LogServ_HeavyForwarders
  3. Click the three-dot menu > Edit agent assignment
  4. Add client targeting using HF IP addresses
  5. Save the agent assignment
  6. Return to the Filters tab – the “Server Class Setup Required” notice should be gone
  7. Click “Deploy to Forwarders” and confirm

Phase 6: Verify HFs Received the Data TA

Wait for HFs to phone home (default interval is 30-60 seconds), then verify:

# On each HF
sudo su - splunk
ls /opt/splunk/etc/apps/splunk_ta_sap_logserv/
cat /opt/splunk/etc/apps/splunk_ta_sap_logserv/local/transforms.conf

Confirm both HFs show as active clients in Settings > Forwarder Management under the server class.

Phase 7: Enable Ingestion and Test Filtering

  1. Enable the SQS-Based S3 inputs on both HFs (these are in the AWS TA, not the Data TA)
  2. Wait a few minutes for data to flow
  3. On the SH, search for incoming data:
`sap_logserv_idx_macro` | stats count by host, sourcetype
  1. Verify included log types are indexed:
`sap_logserv_idx_macro` | stats count by sourcetype

  1. Verify excluded log types are NOT present (if you set any excludes)

  2. Verify time filtering – search for events older than your Days in Past cutoff:

`sap_logserv_idx_macro` earliest=-30d latest=-10d | stats count

This should return 0 results if your Days in Past is less than 10.

Phase 8: Test Filter Update Round-Trip

  1. On the DS Filters tab, change a filter setting (e.g., add an exclude pattern) and Save
  2. Click “Deploy to Forwarders” again
  3. After phone-home, verify on both HFs that local/transforms.conf reflects the updated filter
  4. Confirm the new filter is working as expected in search results

Part 3: Dashboard Verification

After data is flowing, verify the LogServ App dashboards:

  1. Navigate to the LogServ App in Splunk Web
  2. Check each dashboard loads with data:
    • Data Pipeline Overview – shows host event counts and sourcetype distribution
    • DNS Analytics – shows DNS query patterns (requires isc:bind:query data)
    • HANA Audit – shows HANA audit events (requires sap:hana:audit data)
    • Web Dispatcher Access – shows HTTP traffic (requires sap:webdispatcher:access data)
    • Host Details – drill down from Overview by clicking a host row

Time range

Set the Global Time Range to All time or an appropriate window that covers your test data.

Cleanup Scripts

Automated cleanup scripts are available at cleanup_scripts/ in the project root:

Script Target Purpose
run_all_cleanups.sh Local (orchestrator) SCPs and runs all cleanup scripts
cleanup_heavy_forwarder.sh HF-01, HF-02 Removes Data TA from HFs
cleanup_deploy_server.sh DS Removes Data TA from apps/ and deployment-apps/
cleanup_sh_idxr.sh SH/Indexer Removes Data TA + LogServ App for clean reinstall

All scripts use systemctl for Splunk management. Run from local Git Bash:

cd cleanup_scripts && bash run_all_cleanups.sh

Developer Reference: Architecture and Internals

Overview

This document covers the internal architecture of the SAP LogServ TA’s filtering and deployment automation system. It is intended for developers who need to maintain, extend, or test the TA.

The filtering system provides index-time event filtering via TRANSFORMS-based queue routing. It includes a UI built on UCC (Splunk Universal Configuration Console), custom REST handlers, deployment server automation, and background scripted inputs for daily maintenance and upgrade detection.


Architecture Summary

+-------------------------------------------------------+
|  Splunk Web (UCC Configuration UI)                    |
|  Configuration -> Filters tab                         |
|  + filter_settings_hook.js (Deploy button, banners)   |
+---------------------------+---------------------------+
                            | Save
                            v
+-------------------------------------------------------------+
|  splunk_ta_sap_logserv_rh_filter_settings.py (REST Handler) |
|  - Validates patterns                                       |
|  - Saves settings to settings conf                          |
|  - Generates local/transforms.conf + local/props.conf       |
|  - Mirrors to deployment-apps/ (if DS)                      |
|  - Creates server class (if DS, first time)                 |
|  - Reloads confs via REST API                               |
+---------------------------+---------------------------------+
                            |
              +-------------+-------------+
              v                           v
    +------------+          +--------------------+
    | Single     |          | Deployment Server  |
    | Instance   |          |                    |
    | (done)     |          | deployment-apps/   |
    +------------+          | + serverclass.conf |
                            | + Deploy button    |
                            +---------+----------+
                                      | Phone home
                                      v
                            +------------------+
                            | Heavy Forwarders |
                            | (receive TA +    |
                            |  filter configs) |
                            +------------------+


Source File Map

All source files live under the UCC package directory:

sap_logserv_package/splunk_ta_sap_logserv/
├── globalConfig.json                          # UCC UI definition (Filters tab)
├── additional_packaging.py                    # UCC build hook (web.conf expose)
└── package/
    ├── app.manifest                                # TA version and metadata
    ├── bin/
    │   ├── splunk_ta_sap_logserv_filter_utils.py        # Core library
    │   ├── splunk_ta_sap_logserv_rh_filter_settings.py  # REST handler: Filters tab
    │   ├── splunk_ta_sap_logserv_rh_deployment_push.py  # REST handler: Deploy button
    │   ├── logserv_filter_time_refresh.py               # Daily epoch cutoff refresh
    │   └── logserv_filter_upgrade_check.py              # Upgrade coverage check
    ├── default/
    │   ├── transforms.conf    # Sourcetype routing + @logserv_filter annotations
    │   ├── props.conf         # Sourcetype configs, TRANSFORMS chains
    │   ├── inputs.conf        # Scripted input schedules
    │   └── macros.conf        # Index name redirect macro
    ├── metadata/
    │   └── default.meta       # Default permissions and export settings
    └── appserver/static/js/build/custom/
        └── filter_settings_hook.js   # UCC hook (DS detection, Deploy button)


Key Components

Core Library

splunk_ta_sap_logserv_filter_utils.py

This is the central module. All other components import from it.

Key Functions

Function Purpose
get_app_path() Determines the app installation path (checks SPLUNK_HOME, falls back to relative path)
discover_supported_types(app_path) Scans default/transforms.conf for @logserv_filter annotations
find_uncovered_types(supported, patterns) Compares supported types against user include patterns
parse_comma_patterns(string) Splits comma-separated pattern string into list
validate_patterns(patterns, field_name) Validates pattern syntax (dir/subdir format, valid characters)
validate_single_pattern(pattern) Validates one pattern against rules
fnmatch_patterns_to_combined_regex(patterns) Converts a list of fnmatch patterns into a single combined regex using alternation
epoch_cutoff_from_days(days_in_past) Computes the epoch timestamp for midnight N days ago (UTC)
generate_epoch_less_than_regex(cutoff_epoch) Generates a digit-by-digit regex matching epoch values less than the cutoff
generate_transforms_stanzas(include, exclude, days) Generates filter transform stanzas with regex
generate_props_filter_lines(include, exclude, days, enabled) Generates the TRANSFORMS-00-filter line
write_local_conf(app_path, conf_type, content) Writes between marker comments in local conf files
get_ta_version(app_path) Reads version from default/app.conf (falls back to app.manifest if not present)
get_server_roles(session_key) Queries /services/server/info for roles
is_deployment_server(session_key) Two-step DS detection (roles + client probe)
get_deployment_apps_path() Returns the etc/deployment-apps/ path for the TA
ensure_deployment_app_synced(app_path) Full app copy/upgrade to deployment-apps/
mirror_to_deployment_apps(app_path) Copies local/transforms.conf and local/props.conf
ensure_serverclass(session_key) Creates server class + app mapping via REST + file

Constants

Constant Value Purpose
APP_NAME splunk_ta_sap_logserv App directory name
SERVERCLASS_NAME SAP_LogServ_HeavyForwarders Auto-created server class name
SETTINGS_CONF splunk_ta_sap_logserv_settings UCC settings conf file
FILTER_STANZA filter_settings Stanza name in settings conf
SYSTEM_MESSAGE_NAME logserv_filter_upgrade_warning System message banner name for upgrade warnings
ANNOTATION_PATTERN Compiled regex Matches @logserv_filter: annotation comment lines
VALID_SEGMENT_PATTERN Compiled regex Validates characters in dir/subdir pattern segments
FILTER_MARKER_START / FILTER_MARKER_END Marker comments Delimit generated content in local conf files


Filters Tab REST Handler

splunk_ta_sap_logserv_rh_filter_settings.py

UCC REST handler (extends AdminExternalHandler) registered in globalConfig.json. Handles the Save action:

  1. Validates patterns server-side (blocks save on failure via admin.ArgValidationException)
  2. Saves settings to splunk_ta_sap_logserv_settings.conf (default UCC behavior)
  3. Generates local/transforms.conf and local/props.conf with filter stanzas
  4. Syncs and mirrors to deployment-apps if on a DS (full app copy/upgrade + filter configs)
  5. Creates server class if on a DS (first time only)
  6. Reloads confs via REST API (/configs/conf-transforms/_reload, /configs/conf-props/_reload)
  7. Checks for uncovered types and manages system message banner


Deploy Button REST Handler

splunk_ta_sap_logserv_rh_deployment_push.py

Persistent REST handler (PersistentServerConnectionApplication) registered in restmap.conf. Provides two endpoints:

  • GET /services/splunk_ta_sap_logserv/deployment_push — Returns is_deployment_server boolean and server class status (used by the JS hook to render the UI)
  • POST — Triggers deployment reload via /services/deployment/server/config/_reload

Note

Persistent handlers do NOT get import_declare_test. They require explicit sys.path setup for the app’s bin/ and lib/ directories at the top of the file.

Note

Custom persistent endpoints require an [expose:] stanza in web.conf to be accessible through the Splunk Web proxy (port 8000). This is handled by additional_packaging.py during the UCC build.


UCC Hook

filter_settings_hook.js

JavaScript hook loaded by UCC on the Filters tab. Lifecycle methods:

  • onCreate / onRender / onEditLoad — Calls checkDeploymentServer() to GET the deployment push endpoint. If DS detected, injects the deploy banner, server class guidance notices, and Deploy button into the DOM
  • onSaveSuccess — Triggers window.location.reload() after 500ms to reflect server-side changes


Daily Time Refresh

logserv_filter_time_refresh.py

Scripted input (runs every 86400 seconds / once per day):

  1. Reads filter settings from the settings conf via REST
  2. Regenerates local/transforms.conf and local/props.conf with updated epoch cutoff regex
  3. Mirrors to deployment-apps if on a DS
  4. Reloads confs

Deployment client guard: If the instance is a deployment client (HF) but NOT a deployment server, the script skips execution entirely. This prevents HFs from overwriting filter configs pushed by the DS.


Upgrade Coverage Check

logserv_filter_upgrade_check.py

Scripted input (runs every 600 seconds / 10 minutes):

  1. Compares current TA version against last-checked version (persisted in state file)
  2. On version change, discovers supported types from @logserv_filter annotations
  3. Compares against user’s include patterns
  4. Creates a Splunk system message banner if uncovered types are found
Performance

The version check is ~2ms on unchanged runs. Full comparison only runs once per version change.


How Filtering Works Technically

Filter Chain

All filtering happens via a single TRANSFORMS-00-filter line in local/props.conf under the [sap_logserv_logs] stanza. The 00 prefix ensures filters run before any sourcetype routing transforms (01–99).

The filter chain is evaluated left to right:

  1. logserv_filter_include_drop — Drops ALL events (sends to nullQueue)
  2. logserv_filter_include_allow — Rescues events matching include patterns (sends back to indexQueue)
  3. logserv_filter_time_drop — Drops events with old timestamps (sends to nullQueue)
  4. logserv_filter_exclude — Drops events matching exclude patterns (sends to nullQueue)

This “deny-all, then allow” approach ensures only explicitly included events pass through.


Include/Exclude Regex Generation

User-facing fnmatch patterns (e.g., linux/*) are converted to Splunk-compatible regex that matches against the raw NDJSON event data. The include_allow regex uses lookahead assertions to match clz_dir and clz_subdir fields:

^(?=.*"clz_dir"\s*:\s*"linux")(?=.*"clz_subdir"\s*:\s*".+")

Multiple include patterns are OR’d together with |.


Time Filter Regex

The time filter matches epoch timestamps in the raw JSON "_time" field. Since TRANSFORMS cannot use dynamic expressions (unlike INGEST_EVAL which is unavailable on Splunk Cloud), the cutoff is a pre-computed regex that matches epoch values less than the cutoff. This regex must be refreshed daily to maintain accuracy.

Failure mode

If the daily refresh doesn’t run, the cutoff becomes one day older, filtering slightly more data. This is the safer direction.


Why TRANSFORMS Instead of INGEST_EVAL

INGEST_EVAL is the preferred modern approach for index-time filtering, but it is unavailable on Splunk Cloud (heavy forwarders managed by Splunk Cloud don’t support it). TRANSFORMS-based filtering works on all deployment architectures including Splunk Cloud with on-premises Heavy Forwarders.


Deployment Server Automation

DS Detection

Two-step detection in is_deployment_server():

  1. Fast path — Check server_roles from /services/server/info for deployment_server
  2. Fallback — If role is absent, query /services/deployment/server/clients to check for connected deployment clients. Returns true only if at least one client is connected
Why the fallback exists

Splunk drops the deployment_server role when no serverclass.conf exists. The /services/deployment/server/config endpoint cannot be used as a fallback because it returns HTTP 200 on ALL Splunk Enterprise instances (including HFs), causing false positives.


Server Class Creation

ensure_serverclass() creates SAP_LogServ_HeavyForwarders in three steps:

  1. REST API create — POST to /services/deployment/server/serverclasses with name parameter to create the server class

    The disabled parameter is NOT supported during creation (Splunk returns an error).

  2. REST API disable — Separate POST to /services/deployment/server/serverclasses/SAP_LogServ_HeavyForwarders with disabled=true to disable the server class. This ensures no deployment occurs until the admin configures client targeting and enables the server class in Forwarder Management.

  3. File-based app mapping — After REST creates and disables the server class, the function locates the resulting serverclass.conf (which Splunk may write to system/local/ or apps/search/local/) and appends the app mapping stanza:

[serverClass:SAP_LogServ_HeavyForwarders:app:splunk_ta_sap_logserv]
restartSplunkd = true
stateOnClient = enabled

Note

The app mapping cannot be created via REST API — the URL path format serverclasses/{name}/app:{appname} returns HTTP 404.


Deployment Client Guard

The logserv_filter_time_refresh.py scripted input checks whether the instance is a deployment client (but not a DS). If so, it skips execution to prevent overwriting filter configs that were pushed by the DS. Without this guard, the time refresh script would read the HF’s empty local settings, regenerate configs with only a time filter, and overwrite the complete filter chain.


Adding Support for New Sourcetypes

Step 1: Add a Routing Transform

Add the @logserv_filter annotation on the line immediately above the stanza header in default/transforms.conf:

# @logserv_filter: newdir/newsubdir
[set_srctype_for_new_logtype]
REGEX = "clz_subdir":"newsubdir"
FORMAT = sourcetype::sap:newlogtype
DEST_KEY = MetaData:Sourcetype

The annotation is critical. It declares which clz_dir/clz_subdir values the transform handles. Without it, the upgrade check cannot detect that a new log type is supported, and users won’t be notified.

Multiple values can be comma-separated:

# @logserv_filter: newdir/type_a, newdir/type_b, newdir/type_c
[set_srctype_for_new_dir]
REGEX = "clz_subdir":"(type_a|type_b|type_c)"
FORMAT = sourcetype::sap:newdir:logs
DEST_KEY = MetaData:Sourcetype

When the same clz_subdir value exists under multiple clz_dir paths (e.g., audit appears under both abap/ and scc/), use a compound lookahead regex to match both fields and avoid routing collisions:

# @logserv_filter: scc/audit
[set_srctype_scc_audit]
REGEX = (?=.*"clz_dir":"scc")(?=.*"clz_subdir":"audit")
FORMAT = sourcetype::sap:scc:audit
DEST_KEY = MetaData:Sourcetype

When to use compound lookahead: Check if the clz_subdir you are adding already exists in another clz_dir path. If it does, both the existing and new transforms must use compound lookahead. Current collision-prone values include audit (abap, scc), sapstartsrv (abap, sap), and tracelogs (hana, scc).


Step 2: Add to TRANSFORMS Chain

Add your new transform to the appropriate TRANSFORMS-* line in the [sap_logserv_logs] stanza of default/props.conf:

[sap_logserv_logs]
...
TRANSFORMS-07-srctype_for_newdir = set_srctype_for_new_logtype

Use a number between 01 and 98 for the TRANSFORMS prefix. 00 is reserved for filters and 99 is reserved for set_raw_only.


Step 3: Add Sourcetype Configuration

If the new sourcetype needs field extractions, calculated fields, or CIM field aliases, add a [sap:newlogtype] stanza to default/props.conf.


Step 4: Bump the Version

Update the version in package/app.manifest (and globalConfig.json if applicable). This triggers the upgrade check to compare the new annotations against existing user include patterns.

What Happens on Upgrade
  1. The logserv_filter_upgrade_check.py scripted input detects the version change within 10 minutes
  2. It scans the updated default/transforms.conf for @logserv_filter annotations
  3. If the user’s include patterns don’t cover the new log types, a system message banner appears across all Splunk Web pages
  4. The user updates their include patterns (or uses */*) and the banner clears


Testing Environments

Environment 1: Single Instance (Standalone)

Setup: One Splunk Enterprise instance acting as Search Head, Indexer, and data receiver.

What to test
  • Filter save generates correct local/transforms.conf and local/props.conf
  • Conf reload works without restart
  • Include, exclude, and time filters work correctly in search results
  • Disabling filtering clears the generated conf files
  • Pattern validation blocks invalid input
  • No deployment server UI elements appear (no banner, no deploy button)


Environment 2: Deployment Server + Heavy Forwarders

Setup: Three instances minimum: DS (can be combined with SH/Indexer), HF-01 (deployment client), HF-02 (deployment client).

What to test
  • DS detection works (banner and deploy button appear)
  • Filter save triggers full app copy to deployment-apps/
  • Filter configs are mirrored to deployment-apps/local/
  • Server class SAP_LogServ_HeavyForwarders is auto-created with app mapping
  • Client targeting (IP-based) matches HFs correctly
  • Deploy button triggers reload and HFs receive the TA
  • Filter update round-trip: change on DS → deploy → verify on HFs
  • Time refresh script skips on HFs (deployment client guard)
  • Time refresh script runs correctly on DS and mirrors updated configs


Environment 3: Splunk Cloud + On-Premises Heavy Forwarders

Setup: Splunk Cloud instance (Search Head / Indexer), separate on-premises DS, on-premises HFs configured as deployment clients.

What to test
  • Same as Environment 2, but additionally verify:
  • The TA is installed on the Splunk Cloud instance for dashboards and search-time knowledge objects, but filtering is NOT configured there
  • Filter configuration and TA distribution to HFs are managed entirely from the on-premises DS
  • Filtering works at the HF level before data reaches Splunk Cloud
  • No INGEST_EVAL dependency (TRANSFORMS-only filtering)


Environment 4: Fresh DS with No Prior Server Classes

Setup: DS with deployment clients connected but no serverclass.conf exists.

What to test
  • DS detection works via the fallback (connected clients check)
  • ensure_serverclass() creates serverclass.conf from scratch
  • The deployment_server role activates after server class creation
  • Full deployment workflow completes successfully


Known Gotchas and Technical Notes

Local Imports in Utility Functions

Functions in splunk_ta_sap_logserv_filter_utils.py use local imports (inside the function body) for Splunk-specific modules like splunk.rest and json. This is intentional — these modules are unavailable during the UCC build step, so importing them at the module level would break the build.

The critical detail is that local imports are scoped to the function they are declared in. They do NOT carry into sibling functions. Each function that needs splunk.rest or json must import them independently. For example, both get_server_roles() and is_deployment_server() need their own local import splunk.rest as rest and import json statements — the imports inside get_server_roles() are not visible to is_deployment_server() even though one calls the other.

Forgetting a local import inside a function that has a bare except Exception: pass block is particularly dangerous because the resulting NameError is silently swallowed, causing the function to return a default value without any log output.


Persistent Handler sys.path

Persistent REST handlers (splunk_ta_sap_logserv_rh_deployment_push.py) do NOT get import_declare_test from UCC. They require explicit sys.path setup at the top of the file to import from the app’s bin/ and lib/ directories. Without this, the handler returns 500 errors from splunkd.


web.conf Expose Stanza

Custom persistent endpoints are not accessible through the Splunk Web proxy (port 8000) by default. They require an [expose:] stanza in web.conf. This is injected by additional_packaging.py during the UCC build:

[expose:splunk_ta_sap_logserv_deployment_push]
pattern = splunk_ta_sap_logserv/deployment_push
methods = POST, GET


Server Class REST API Limitations

  • The disabled parameter is NOT accepted during server class creation. Create first, then POST to the specific server class URL to disable
  • App mappings (:app:appname stanzas) CANNOT be created via REST API. The URL format returns 404. Use file-based append instead
  • Splunk may write serverclass.conf to different locations (system/local/ or apps/search/local/). The code searches multiple locations


Deployment Client Config Overwrite

Without the deployment client guard in logserv_filter_time_refresh.py, HFs would regenerate filter configs from their own (empty) local settings on the daily time refresh run. This overwrites the complete filter chain pushed by the DS with only a time filter. The guard checks for the deployment_client role and skips execution if found.


TRANSFORMS-00-filter Ordering

The filter TRANSFORMS line MUST use prefix 00 to ensure it runs before sourcetype routing (01–99). If filters ran after routing, events would already have their sourcetype set but would then be dropped, wasting processing.


Marker Comments

Generated filter content in local/transforms.conf and local/props.conf is wrapped in marker comments:

### BEGIN LOGSERV FILTER CONFIG - DO NOT EDIT MANUALLY ###
...
### END LOGSERV FILTER CONFIG ###

The write_local_conf() function replaces content between these markers (or appends them if not present). Manual customizations outside the markers are preserved.


DS Role Disappears Without Server Classes

Splunk removes the deployment_server role from server_roles when no server class is defined in any serverclass.conf. The is_deployment_server() fallback handles this by checking for connected deployment clients via the /services/deployment/server/clients endpoint. This endpoint returns 0 clients on HFs (no false positives) and returns connected clients on a real DS even without any server classes.

The /services/deployment/server/config endpoint CANNOT be used as a fallback because it returns HTTP 200 on ALL Splunk Enterprise instances, including HFs.

AI Assistant — Implementation Reference

This document covers the implementation details of the LogServ App’s AI Assistant subsystem — file paths, type-system patterns, build mechanics, audit-event schemas, and extension recipes. Intended for engineers who need to maintain or extend the AI Assistant code.

For the customer-facing documentation, see the AI Assistant section. The customer-facing pages cover what users see and how to use the feature; this page covers how it’s built.

File Path Map

The AI Assistant subsystem lives in the LogServ App’s React workspace at packages/log-serv-app/src/main/webapp/pages/home/. Key files:

Path Purpose
hooks/useAIAssistant.ts The orchestrator hook. Contains runCannedPrompt, sendUserMessage, tier1Summary, tier2Summary, SYSTEM_PRIMER_TIER1, SYSTEM_PRIMER_TIER2, POWER_MODE_RULE_SUFFIX, SAVED_SEARCH_SPL, SAVED_SEARCH_DASHBOARDS, SAVED_SEARCH_CHART_HINTS, resolveNextSteps, wrapAsToolResultData, sanitizeAggregateValue.
state/AIAssistantProvider.tsx React context + reducer. DisplayMessage shape including toolResult.spl / earliest / latest for drill-down chip URL pre-resolution.
components/ai/types/Hidden.ts The Hidden<T> and Visible<T> brand-type definitions, markHidden, unwrapHidden, markVisible, unwrapVisible, sanitize.
components/ai/mcp/MCPClient.ts MCP client wrapper. Constructs requests, handles cookie + bearer auth, parses responses into MCPToolResult.
components/ai/providers/ Per-vendor provider implementations: AnthropicProvider.ts, OpenAIProvider.ts, AzureOpenAIProvider.ts, BedrockProvider.ts, plus shared credentials.ts, sseUtils.ts, anthropicEventTranslator.ts.
components/ai/chat/AIAssistant.tsx Top-level chat panel. Builds the citationLookup: Map<string, { toolUseId; splUrl? }> from toolResultsInOrder.
components/ai/chat/ChatMessage.tsx Renders user / assistant_text / tool_call / tool_result / system_notice / guidance messages. Contains TEXT_PATTERN regex for citation + severity + bold parsing.
components/ai/chat/ToolResultPanel.tsx Renders a single tool result as table / chart / KPI / pie. Contains the actions-slot rendering for ↗ Dashboard / ↗ Run SPL / Clear chips.
components/ai/chat/ChatInput.tsx Chat input toolbar. ✦ Power toggle visibility gating + Templates-only gating.
components/ai/chat/PrivacyBanner.tsx Tier banner + model picker. Templates-only hides the picker.
components/ai/audit/AuditWriter.ts Browser-side audit batch + flush. Dual-writes to local _ai_assistant_audit index AND optional HEC forwarder.
components/ai/audit/auditTypes.ts All 12 audit-event TypeScript types.
dashboards/AIAssistantSettings.tsx The 4-tab Settings page. Templates-only hides the Provider Credentials tab.
utils/splGuard.ts SPL static-analysis guard. Blocks collect/outputlookup/outputcsv/delete/sendalert/sendemail/script/run/tscollect for AI-authored ad-hoc SPL.
utils/jailbreakPatterns.ts User-prompt jailbreak-pattern detection. FNV-1a hash + matched-groups + char-class fingerprint capture.
utils/piiRedaction.ts Tier 2 PII redaction. FNV-1a 32-bit hash → 7-char hex tag, sync. Stable per-value across the run.
utils/rateLimit.ts Per-user rolling-1-hour rate limit. Records rate_limited_prompt audit event on hit.
utils/dailySpend.ts Cumulative-cost daily spend cap. Records daily_spend_cap_hit on overflow.
utils/vendorCost.ts Per-vendor pricing table for USD cost estimate.
utils/drilldownUrls.ts buildDashboardUrl, buildHostDetailsUrl, buildSplunkSearchUrl, openInNewTab, splQuote, sourcetypeToDashboardSlug.
buildFlags.ts Build-time feature flags. Currently exports TEMPLATES_ONLY: boolean.
utils/aiConfigApi.ts Read / write config from default/ai_assistant_settings.conf via REST.
utils/passwordsApi.ts Read / write credentials from Splunk’s encrypted password store via services/storage/passwords.
utils/telemetryConfApi.ts Splunk-pattern legal-acknowledgement opt-in tracking via default/ai_assistant_acks.conf.
state/dashboardRefreshPersistence.ts KV Store CRUD for the per-dashboard auto-refresh picker.
default/data/mcp/logserv_intent_map.json The 48-prompt catalog.
default/savedsearches.conf The 48 saved searches that intent-map prompts dispatch.
default/ai_assistant_settings.conf Org-wide AI Assistant config defaults.
default/ai_assistant_acks.conf Splunk-pattern legal-acknowledgement opt-in version + acknowledgement timestamps.

The Hidden<T> / Visible<T> Brand Type Pattern

The privacy invariant — “no event data from your Splunk instance is ever transmitted to any AI vendor” — is mechanically enforced via TypeScript brand types in components/ai/types/Hidden.ts:

declare const HIDDEN_BRAND: unique symbol;
declare const VISIBLE_BRAND: unique symbol;

export type Hidden<T> = T & { readonly [HIDDEN_BRAND]: true };
export type Visible<T> = T & { readonly [VISIBLE_BRAND]: true };

export const markHidden = <T>(v: T): Hidden<T> => v as Hidden<T>;
export const unwrapHidden = <T>(v: Hidden<T>): T => v;

export const markVisible = <T>(v: T): Visible<T> => v as Visible<T>;
export const unwrapVisible = <T>(v: Visible<T>): T => v;

export const sanitize = <T, S>(
    hidden: Hidden<T>,
    summarizer: (v: T) => S,
): Visible<S> => markVisible(summarizer(unwrapHidden(hidden)));

Invariant: the only way to obtain a Visible<S> from a Hidden<T> is via sanitize(hidden, summarizer). The compiler refuses any other coercion path. The summarizer receives the unwrapped data and returns a non-data summary string; the orchestrator’s call site picks the summarizer based on the active privacy tier (tier1Summary for Tier 1, tier2Summary for Tier 2).

Where the chokepoint runs: useAIAssistant.ts sendUserMessage → tool dispatch → MCPClient returns Hidden<MCPToolResult> → sanitize(toolResult, tier === 2 ? tier2Summary : tier1Summary) → resulting Visible<string> is appended to the outbound vendor message buffer.

Where the type system enforces: every outbound vendor message is built from Visible<Message>[]. Anywhere code constructs an outbound message, the types refuse a Hidden<T> value. There is no as any cast; bypassing requires modifying the brand-type code itself, which is reviewed.

Augmenting a Visible<T>-branded string: direct concatenation produces a TS error since the brand type can’t widen back automatically. The pattern is unwrapVisible() + new string + markVisible(). The Power Mode primer augmentation uses this pattern.

The Build Flag Pattern (TEMPLATES_ONLY and Future Flags)

Why compile-time, not runtime

Compile-time flags are baked into the bundle at build time and CANNOT be changed post-deploy. The runtime alternative — an admin toggle that disables a code path — would let any admin flip it on. For deployments where a code path is intentionally not available (e.g., the LLM dispatch path under review), a compile-time variant is the right shape: the bundle has no code path that reaches the disabled feature, and there’s no runtime setting that could enable one.

Single source of truth

buildFlags.ts is the canonical module for compile-time flags — it exports each flag as a typed boolean derived from a build-time substitution. Use sites import the flag from buildFlags.ts rather than reading the underlying env var directly. After minification + dead-code elimination, unreachable branches are eliminated from the bundle that has the flag off (verified via grep on the minified output for the absence of the underlying env-var token).

The pattern composes: a build can carry multiple flags simultaneously, and DCE ensures each variant is lean.

Defense-in-depth runtime guards

Even with compile-time gating, defense in depth pairs the UI gating with a runtime guard at the LLM dispatch entry point. When TEMPLATES_ONLY is true, the guard short-circuits the dispatch with a system notice directing the user to the prompt browser instead. The UI gating is the primary defense; the function-level guard catches any future code path that might bypass the disabled chat input (keyboard shortcut, programmatic dispatch, etc.).

The Chat Citation Parser

components/ai/chat/ChatMessage.tsx parses inline markers in assistant_text + guidance messages. The regex:

const TEXT_PATTERN = /\[(?:→|->)\s*([a-zA-Z0-9_:.\-]+)\]|\[severity:(critical|high|medium|low)\]|\*\*([^*]+)\*\*/g;

Three numbered capture groups disambiguate which alternative fired:

Group Pattern Renders as
1 [→ logserv_xxx] or [-> logserv_xxx] Clickable scroll-to-tile span via onJumpToResult(toolUseId); auto-appends sibling ↗ Dashboard chip(s) + ↗ Run SPL chip
2 [severity:critical|high|medium|low] Glossy colored dot via <SeverityDot $level={level} />
3 **bold text** Bold-stripped: the ** markers are dropped and the inner text is re-parsed recursively for nested citation / severity markers (build 174 fix)

The recursive-parse fix (build 174): when the AI drifts and emits **[severity:medium]** (markdown bold wrapping the severity marker), the bold alternative matches first and would render the inner text as plain string, losing the severity dot. The fix recursively calls renderTextWithCitations() on the bold inner content. Save/restore of TEXT_PATTERN.lastIndex around the recursive call is required because the regex is module-scoped + /g-stateful — not saving would corrupt the outer iteration.

Why module-scoped state: lastIndex reset at function entry handles the common case. For recursive parses, save/restore around the call is required.

The MCP Tool Definitions Sent on Every Free-Form Request

The AI sees these two tool definitions on every free-form request. The schemas are sent in the system primer’s tool-block:

splunk_run_saved_search

{
    "name": "splunk_run_saved_search",
    "description": "Dispatch a saved search from the LogServ App's catalog.",
    "input_schema": {
        "type": "object",
        "properties": {
            "name": {"type": "string", "description": "Saved-search name from the catalog (e.g., logserv_hana_failed_auth)"},
            "earliest_time": {"type": "string", "description": "Splunk earliest token (-24h, -7d, etc.)"},
            "latest_time": {"type": "string", "description": "Splunk latest token (now, etc.)"},
            "render_hint": {"type": "string", "enum": ["table", "timechart", "kpi", "pie"]},
            "top_n": {"type": "integer", "minimum": 1, "maximum": 50, "description": "Width of categorical aggregates the AI receives in Tier 2 summary. Default 10."}
        },
        "required": ["name"]
    }
}

splunk_run_query

Same schema shape but accepts a query (SPL string) instead of name. SPL is run through utils/splGuard.ts before dispatch to block write/delete/alert operators.

The SPL Static-Analysis Guard

utils/splGuard.ts exposes analyzeSpl(spl: string): { blocked: boolean; reason?: string; operator?: string }. Blocked operators:

const BLOCKED_OPERATORS = [
    'collect', 'outputlookup', 'outputcsv',
    'delete', 'sendalert', 'sendemail',
    'script', 'run', 'tscollect',
];

Detection runs after lowercasing + stripping leading whitespace per pipe segment. Quoted strings are stripped before matching to avoid false positives on legitimate SPL where these tokens appear inside string literals.

Audit on block: security_blocked_spl event with spl (truncated to 1000 chars) + operator field. The synthetic error tool_result lets the AI see the failure and recover by writing a different query.

Scope: only AI-authored splunk_run_query SPL. Pre-authored saved searches dispatched via splunk_run_saved_search are NOT subject to the guard — their SPL is reviewed at build time.

Audit Event Schemas

All audit events extend a BaseAuditEvent shape:

interface BaseAuditEvent {
    category: string;       // discriminator
    timestamp: string;      // ISO-8601 UTC
    user: string;           // Splunk username
    sessionId: string;      // Per-tab session UUID
    seq: number;            // Monotonic per-session sequence
}

Per-category extras:

Category Extra fields
local_only (LocalOnlyEvent) promptId: string, spl: string, rowCount: number, executionMs: number, ok: boolean
vendor_tier1 (VendorTier1Event) provider, model, inputTokens, outputTokens, vendorCostEstimateUsd, outboundBytes, promptLength, turnCount, powerMode?: boolean
vendor_tier2 (extends Tier1Event) aggregateKind: string, tier2RedactionsApplied: number
vendor_tier2_elevation previousTier, newTier
security_blocked_spl (SecurityBlockedSplEvent) spl (truncated to 1000 chars), operator: string
rate_limited_prompt (RateLimitedPromptEvent) cap, attemptedCount, windowSeconds
user_prompt_jailbreak_flag (UserPromptJailbreakFlagEvent) promptHash (FNV-1a 32-bit hex), promptLength, matchedGroups: string[], charClassFingerprint: string
session_tool_cap_hit (SessionToolCapHitEvent) cap, attemptedCount, toolName
daily_spend_cap_hit (DailySpendCapHitEvent) cap, currentSpend, attemptedSpend
audit_forwarder_failure destinationUrl (sanitized — query strings stripped), reason, batchSize
forwarder_disabled_acceptance / ai_assistant_enable_acceptance host (Splunk-stamped IP), tcVersion, optInChoice, disclaimerHash

All schemas live at components/ai/audit/auditTypes.ts.

Splunk reserved-field gotcha

The category field on these events is a Splunk reserved name. The viewer’s SPL uses | eval category=json_extract(_raw, "category") after spath because spath does NOT overwrite reserved fields. Replicate this pattern in custom queries.

Adding a New Predefined Prompt

  1. Edit default/data/mcp/logserv_intent_map.json — add a new entry to the prompts array. Required fields: promptId, savedSearch, label, description, spl, renderHint. Optional: chartHint, chartPalette, dashboard, interpretation, nextSteps.
  2. Edit default/savedsearches.conf — add a stanza with the same name as the prompt’s savedSearch field. The SPL must match byte-for-byte.
  3. Bump the intent map’s top-level version field.
  4. Run yarn intentMap.consistency-test (or equivalent in your CI) — the test asserts byte-equality between the intent map’s SPL and savedsearches.conf for every prompt.
  5. Pre-deploy SPL dry-run. Dispatch the new prompt’s SPL via services/search/jobs/oneshot against live data: 
    curl -sk -u admin:<pw> --data-urlencode "search=<your spl>" \
    -d output_mode=json -d earliest_time=-30d -d latest_time=now \
    https://localhost:8089/services/search/jobs/oneshot
    Confirm the result has rows with the expected fields. ~50% of new prompts fail static review with field-extraction issues that only surface against live data.
  6. Bump UI App [install] build in app.conf to bust Splunk Web’s static-asset cache.

Intent map JSON schema

interface IntentMapPrompt {
    promptId: string;
    savedSearch: string;
    label: string;
    description: string;
    spl: string;
    renderHint: 'table' | 'timechart' | 'kpi' | 'pie';
    chartHint?: 'timechart' | 'kpi' | 'pie';
    chartPalette?: 'volume' | 'errors' | 'errors-2' | 'errors-3' | 'auth' | 'status' | 'categorical' | 'neutral';
    dashboard?: string | string[];
    interpretation?: string;
    nextSteps?: Array<string | { text: string; savedSearch?: string; spl?: string }>;
}

Splunk risky-command safeguard

Splunk Web’s Search app has a safety modal on these commands: map, runshell, script, delete, crawl, tscollect, loadjob, outputlookup, outputcsv, sendalert, sendemail. URLs to /app/search/search?q=... containing one of these triggers the modal.

Scan all SPL — including nextSteps.spl deep-dive strings — for risky commands before shipping. Rewrite map-style cross-row dispatches to first-class subsearch syntax: [ search ... | top 1 X | fields X ].

Adding a New Audit Event Category

  1. Edit components/ai/audit/auditTypes.ts — add a new TypeScript interface extending BaseAuditEvent with the category-specific fields. Add the type to the AuditEvent union.
  2. Add the new category id to the discriminator string-literal type.
  3. Find the call site that should emit the event. Construct the event object and call ctx.actions.recordAudit(event).
  4. Edit components/AuditLogViewer.tsx — add the new category id to AUDIT_CATEGORIES (the filter chip set) and to CATEGORY_GRADIENTS (the chip gradient lookup).
  5. Bump the UI App build.

The audit writer (components/ai/audit/AuditWriter.ts) sends events to BOTH the local _ai_assistant_audit index AND the optional HEC forwarder. No code change needed at the writer layer for new categories.

System Primer Architecture

Two primer variants ship: SYSTEM_PRIMER_TIER1 and SYSTEM_PRIMER_TIER2. Both live in hooks/useAIAssistant.ts as exported constants. They share most of their content; the differences:

  • Tier 1 primer instructs the AI that summaries it receives are count + execution-time only, and to write SHAPE-aware narrative (“X rows returned for the search’s rolling window”) rather than inventing concrete values.
  • Tier 2 primer informs the AI that summaries include aggregated metadata (per-column cardinality, top-N values + counts, numeric stats, time range), and instructs it to ground every claim in the actual values from the aggregates. PII redaction tagging is also explained.

Both primers include:

  • The TOOL_RESULT_DATA boundary block (<TOOL_RESULT_DATA>...</TOOL_RESULT_DATA> sentinels)
  • The TIME-WINDOW REASONING block (build 171)
  • The canonical saved-search catalog (48 entries)
  • The LogServ data model (sourcetypes + key fields, for ad-hoc SPL)
  • The decision rule for splunk_run_saved_search vs splunk_run_query
  • The SYNTHESIS RULES section: lettered finding format, severity-dot rendering, citation format, etc.

Power Mode augmentation: when the user has Power Mode on, POWER_MODE_RULE_SUFFIX is appended to the primer at dispatch time via unwrapVisible(content) + suffix then markVisible() (the brand type can’t widen back automatically).

Pre-Resolving Drill-Down URLs at the Orchestrator Layer

The drill-down chip URLs (↗ Dashboard, ↗ Run SPL) are pre-resolved at the AIAssistant.tsx layer, where each tool result’s exact dispatch window is known. The resolved URLs are stored on the citationLookup Map for the chat parser to consume:

const citationLookup: Map<string, { toolUseId: string; splUrl?: string }> = new Map();
toolResultsInOrder.forEach((tr) => {
    if (tr.displayName) {
        const splUrl = tr.spl
            ? buildSplunkSearchUrl(tr.spl, tr.earliest, tr.latest)
            : undefined;
        citationLookup.set(tr.displayName, { toolUseId: tr.toolUseId, splUrl });
    }
});

The chat-side renderer (ChatMessage.tsx) is dumb — it just renders <a href={entry.splUrl}> when the lookup carries a URL. This avoids duplicating SPL-lookup logic at every consumer.

Same pattern applies to the dashboard chips (build 156): the dashboardLinks are resolved once via resolveDashboardLinks(slugs) at the orchestrator and consumed as {slug, name, url} triples.

Legal Acknowledgement Modal Trigger Logic

Two compile-time legal modals gate Settings save flows:

  • Enable Acceptance Modal — gates enabled = true saves.
  • Forwarder Disabled Acceptance Modal — gates audit_forwarder_enabled = false saves.

Both follow Splunk’s splunk_instrumentation optInVersion framework with one extension. The trigger logic is:

isEnableTcPending = (
    (saved.enabled === false && draft.enabled === true)  // deliberate toggle-on
    OR
    (currentOptInVersion > userAcknowledgedVersion)       // version bumped
)

The pure optInVersion-only path is what Splunk’s stock telemetry uses (acknowledge once per version, durable until bump). We added the deliberate-toggle-transition trigger because re-enabling AI Assistant after a disable is a deliberate user action that warrants re-acknowledgement of the legal posture each time.

telemetryConfApi.ts exposes readTcAcknowledgement(stanza) / writeTcAcknowledgement(stanza, choice, version) parameterized by stanza name. The two modal stanzas are constants: STANZA_FORWARDER_TC and STANZA_ENABLE_TC.

The conf-writer coerces yes/no to '1'/'0'. The audit event preserves the literal string in _raw, so the audit event is the canonical record while the conf is the state marker.

Auto-Mint MCP Token Roadmap

The current MCP authentication relies on either cookie-based session reuse (default) or admin-paste bearer tokens. A planned future release replaces the manual paste with auto-mint via a Data TA-side Python REST handler:

  1. The handler reads the MCP Server’s RSA public key from a local lookup.
  2. Mints a short-lived JWT using the Splunk admin’s session credentials.
  3. Returns the JWT to the React app via a new endpoint at /services/splunk_ta_sap_logserv/mint_mcp_token.
  4. The React app caches the JWT, refreshes when nearing expiry.

Status: roadmap, ~2-3 days of engineering effort, not yet shipped.

The Python REST handler skeleton would live in the Data TA at bin/mcp_token_minter.py. Would require:

  • An [expose:] stanza in the TA’s web.conf for the new endpoint
  • A restmap.conf entry registering the handler
  • The RSA public key in a controlled lookup or local/passwords.conf

For implementation: see the existing bin/splunk_ta_sap_logserv_settings.py REST handler for the structural template (persistent endpoints require explicit sys.path setup; no import_declare_test).

Why Primer Rules Beat Other Fixes (TIME-WINDOW REASONING)

Three alternative fixes were considered before settling on primer rules for the time-window reasoning behavior:

  1. Surface the dispatch window in Tier 2 metadata even for non-_time results. Would let the AI see the window directly without reasoning about it. Higher engineering effort (~4-6 hours) than primer rules; deferred unless rules alone don’t move the needle.
  2. Re-order the regex alternatives so severity matches first. Doesn’t help: bold wraps around the severity, so even if severity tries first, it can’t match inside an unmatched bold span.
  3. Update the primer to forbid bold around severity markers. Helps for future prompts but doesn’t repair existing UI. Parser must be tolerant either way.

Primer rules were the cheapest test of “is this a primer-rules issue?” and it worked: same prompt, same data, dramatically better self-correction behavior in one turn instead of needing a follow-up prompt to re-rank cumulative-noise findings.

This is the canonical pattern: when the AI has the data but isn’t using it correctly, primer rules are the cheapest first attempt. When the AI doesn’t have the data, metadata expansion is the next step.

Splunk Static-Asset Cache Busting

Splunk Web caches the React bundle’s asset URL under a hash that includes the [install] build integer in app.conf. Re-deploying the same build number with new bytes serves stale assets even after Splunkd restart. Always bump the build number for any meaningful code change.

The 3-part SemVer in [id] version is independent and only changes on user-facing version bumps. AppInspect’s pretrained / Splunkbase precert rules require 3-part SemVer, so directory naming uses a 4-part vMAJOR.MINOR.PATCH.SNAPSHOT form for snapshot-tracking purposes only — the app.conf [id] version stays 3-part.

Live UI Refresh on Settings Save

When the admin saves a config change in the General tab that affects what users see (master enabled toggle, mcp_required, power_user_roles, etc.), the React app re-runs loadAIAssistantConfig() immediately and re-applies the new config to the running session. No page reload required.

Implementation: App.tsx accepts an onConfigSaved callback that re-loads config from the storage REST API. The callback is threaded App → AppShell → AIAssistantSettings → GeneralPanel, called after writeAIConfig succeeds. Affected props are recomputed on the next React render and the dependent UI re-renders accordingly (e.g., the ✦ AI Assistant button in the top-right nav appears / disappears).

Ended: Developer Reference