Skip to content

O365 Management Activity: Reduce log size (CIM & ESCU compatibility)

Disclaimer

By using SPL2 templates for data processing (the “templates”), you understand and agree that templates are provided “as is”. Splunk disclaims any and all warranties, express or implied, including without limitation the implied warranties of merchantability, fitness for a particular purpose and warranties arising out of course of dealing or usage of trade or by statute or in law. Splunk specifically does not warrant that templates will meet your requirements, the operation or output of templates will be error-free, accurate, reliable, complete or uninterrupted.

Use case

Use this template to reduce the size of O365 Management Activity logs by removing proven noise fields while preserving full compatibility with both the Splunk Common Information Model (CIM) and Enterprise Security Content Updates (ESCU).

Use this template for environments using Splunk Enterprise Security.

Template details

Compatibility

This template is compatible with Splunk Add-on for Microsoft Office 365 v4.0.0.

Template description

This pipeline reduces the size of O365 Management Activity events while preserving compatibility with the Splunk Common Information Model (CIM) and Security Detections (ESCU). This pipeline processes events with the o365:management:activity sourcetype and removes 107 proven noise fields from the events.

The pipeline features dual output branching:

  • Full/raw events are sent to $low_priority_destination (for example, S3 archive for compliance and backup)
  • Processed/reduced events are sent to $destination (for example, Splunk indexers for analysis)

Fields removed by this template

The following fields are removed from events:

Identity and Context fields:

  • AADGroupId
  • AadAppId
  • ActorContextId
  • ActorUserId
  • ActorYammerUserId
  • TargetContextId
  • UserKey

Application and Service fields:

  • AddOnGuid
  • AddOnName
  • AddOnType
  • AppAccessContext
  • AppDistributionMode
  • AppName
  • ApplicationDisplayName
  • AssertingApplicationId
  • ClientApplication

Event Metadata fields:

  • AzureActiveDirectoryEventType
  • EventSource
  • SecurityComplianceCenterEventType
  • SkypeForBusinessEventType
  • Version

SharePoint and OneDrive fields:

  • ChannelName
  • ChannelType
  • CustomUniqueId
  • CustomizedDoclib
  • DestFolder
  • DestinationFileExtension
  • DestinationFileName
  • DestinationRelativeUrl
  • IsDocLib
  • ItemCount
  • ItemName
  • ListBaseTemplateType
  • ListBaseType
  • ListColor
  • ListIcon
  • RelativeUrl
  • RemoteItemWebUrl
  • WebId
  • YammerNetworkId

Teams and Communication fields:

  • CommentId
  • Comments
  • CommunicationType
  • MessageID
  • MessageURLs{} (all elements)
  • MessageVersion
  • ParentCommentId
  • ParentMessageId

Exchange and Mail fields:

  • CrossMailboxOperation
  • InternalLogonType
  • LogonType
  • LogonUserSid
  • MailboxGuid
  • MailboxOwnerMasterAccountSid

DLP and Compliance fields:

  • PolicyDetails{}.Rules{}.Actions{} (all nested elements)
  • PolicyDetails{}.Rules{}.OverriddenActions{} (all nested elements)
  • PolicyDetails{}.Rules{}.ConditionsMatched.SensitiveInformation{}.SensitiveInformationDetailedClassificationAttributes{} (all nested elements)
  • PolicyDetails{}.Rules{}.ConditionsMatched.SensitiveInformation{}.SensitiveInformationDetections (all nested elements)
  • SensitiveInfoDetectionIsIncluded

Forms and Shifts fields:

  • FormType
  • FormsUserType
  • OpenShift
  • Shift
  • ShiftRequest
  • ScheduleId

Nested Value fields:

  • ExtendedProperties{}.Value (values from all array elements)
  • ExtendedProperties{}.Value{} (only when Value is an object/array)
  • ModifiedProperties{}.NewValue{} (only when NewValue is an object/array)

Other fields:

  • AdditionalData{} (all elements)
  • AdditionalInfo
  • AlertLinks{} (all elements)
  • AlertType
  • ClientRequestId
  • CmdletVersion
  • DatabaseType
  • Description
  • DoNotDistributeEvent
  • EffectiveOrganization
  • EntityType
  • ExceptionInfo
  • ExternalAccess
  • ExtraProperties{} (all elements)
  • FromApp
  • GroupName
  • HighPriorityMediaProcessing
  • ImplicitShare
  • IncidentId
  • InsightData{} (all elements)
  • InsightId
  • IntraSystemId
  • Members{} (all elements)
  • NonPIIParameters
  • OperationScope
  • OrganizationName
  • OriginatingServer
  • ResultCount
  • SharePointMetaData.SharedBy{} (all nested elements)
  • SharingType
  • SourceApp
  • StartTime
  • SupportTicketId
  • TabType
  • TemplateTypeId
  • TenantName
  • UniqueSharingId
  • UserServicePlan

Total: 107 fields removed

Supported sourcetypes

This template processes events with the o365:management:activity sourcetype. Events with other sourcetypes are passed through without processing.

Template outline

Template consists of a custom function followed by a pipeline that uses dual output branching.

Functions

The following table shows all custom functions:

Function name Description
noise_reduction This function removes proven noise fields from original event while preserving CIM and ESCU compatibility. The reduced event is stored back to _raw.

Pipeline This pipeline applies dual output branching:

  1. Original untouched event is sent to $low_priority_destination (archival/backup).
  2. Processed event with noise reduction is sent to $destination (primary indexing).

Configuration instructions

If any field listed in the template description is present in the event, the noise_reduction function deletes it. If you don’t want to delete all the fields, you can modify the field names provided in json_delete accordingly.

You can find examples in the following section.

Configuration example scenarios

Scenario 1: Do not remove ActorContextId field from event

Perform the following steps to keep the ActorContextId field:

  1. In the noise_reduction function, remove "ActorContextId" from the $conservative_fields array.
  2. Execute the pipeline preview and confirm that the field is not removed.
  3. Save the changes.

Scenario 2: Remove additional field not in the default list

Perform the following steps to remove an additional field (for example, CustomField):

  1. In the noise_reduction function, add "CustomField" to the $conservative_fields array.
  2. Execute the pipeline preview and confirm that the field is removed.
  3. Save the changes.

Scenario 3: Disable dual output branching (single destination only)

Perform the following steps to send events only to the primary destination:

  1. Modify the pipeline to remove the branch statement: 
    $pipeline = | from $source
| noise_reduction
| into $destination;
  2. Execute the pipeline preview and confirm events go only to $destination.
  3. Save the changes.

Performance impact

Metric Impact
Space reduction about 15 to 25% (varies by O365 workload)
Processing overhead Minimal (less than 5% CPU increase).
CIM impact None – all required CIM fields preserved
ESCU impact None – all detection fields preserved

When to use this template

Use this CIM & ESCU compatibility template when:

  • You are using Splunk Enterprise Security
  • You need ESCU detection compatibility
  • You want proven, safe noise reduction
  • You need to archive full events for compliance

Consider the CIM only compatibility template when:

  • You are NOT using Enterprise Security
  • You want maximum space reduction (129 fields removed)
  • You only need CIM compatibility