O365 Management Activity: Reduce log size (CIM only compatibility)¶

Disclaimer

By using SPL2 templates for data processing (the “templates”), you understand and agree that templates are provided “as is”. Splunk disclaims any and all warranties, express or implied, including without limitation the implied warranties of merchantability, fitness for a particular purpose and warranties arising out of course of dealing or usage of trade or by statute or in law. Splunk specifically does not warrant that templates will meet your requirements, the operation or output of templates will be error-free, accurate, reliable, complete or uninterrupted.

Use case¶

Use this template to achieve maximum size reduction of O365 Management Activity logs by removing noise fields AND ESCU-specific fields. This template maintains compatibility with the Splunk Common Information Model (CIM) but BREAKS compatibility with Enterprise Security Content Updates (ESCU).

Use this template for environments NOT using Splunk Enterprise Security.

ESCU Compatibility

This template removes ESCU-specific fields and will break Enterprise Security detections. Use the CIM & ESCU compatibility template if you need ESCU compatibility.

Template details¶

Compatibility¶

This template is compatible with Splunk Add-on for Microsoft Office 365 v4.0.0.

Template description¶

This pipeline provides maximum size reduction for O365 Management Activity events while preserving compatibility ONLY with the Splunk Common Information Model (CIM). This pipeline processes events with the o365:management:activity sourcetype and removes 127 total fields (107 noise fields + 20 ESCU-specific) from the events.

The pipeline features dual output branching: - Full/raw events are sent to $low_priority_destination (for example, S3 archive for compliance and backup) - Processed/reduced events are sent to $destination (for example, Splunk indexers for analysis)

Fields removed by this template¶

The following fields are removed from events:

Noise Fields (107 fields - same as CIM & ESCU compatibility template)¶

Identity and Context fields:

AADGroupId
AadAppId
ActorContextId
ActorUserId
ActorYammerUserId
TargetContextId
UserKey

Application and Service fields:

AddOnGuid
AddOnName
AddOnType
AppAccessContext
AppDistributionMode
AppName
ApplicationDisplayName
AssertingApplicationId
ClientApplication

Event Metadata fields:

AzureActiveDirectoryEventType
EventSource
SecurityComplianceCenterEventType
SkypeForBusinessEventType
Version

SharePoint and OneDrive fields:

ChannelName
ChannelType
CustomUniqueId
CustomizedDoclib
DestFolder
DestinationFileExtension
DestinationFileName
DestinationRelativeUrl
IsDocLib
ItemCount
ItemName
ListBaseTemplateType
ListBaseType
ListColor
ListIcon
RelativeUrl
RemoteItemWebUrl
WebId
YammerNetworkId

Teams and Communication fields:

CommentId
Comments
CommunicationType
MessageID
MessageURLs{} (all elements)
MessageVersion
ParentCommentId
ParentMessageId

Exchange and Mail fields:

CrossMailboxOperation
InternalLogonType
LogonType
LogonUserSid
MailboxGuid
MailboxOwnerMasterAccountSid

DLP and Compliance fields:

PolicyDetails{}.Rules{}.Actions{} (all nested elements)
PolicyDetails{}.Rules{}.OverriddenActions{} (all nested elements)
PolicyDetails{}.Rules{}.ConditionsMatched.SensitiveInformation{}.SensitiveInformationDetailedClassificationAttributes{} (all nested elements)
PolicyDetails{}.Rules{}.ConditionsMatched.SensitiveInformation{}.SensitiveInformationDetections (all nested elements)
SensitiveInfoDetectionIsIncluded

Forms and Shifts fields:

FormType
FormsUserType
OpenShift
Shift
ShiftRequest
ScheduleId

Nested Value fields:

ExtendedProperties{}.Value (values from all array elements)
ExtendedProperties{}.Value{} (only when Value is an object/array)
ModifiedProperties{}.NewValue{} (only when NewValue is an object/array)

Other fields:

AdditionalData{} (all elements)
AdditionalInfo
AlertLinks{} (all elements)
AlertType
ClientRequestId
CmdletVersion
DatabaseType
Description
DoNotDistributeEvent
EffectiveOrganization
EntityType
ExceptionInfo
ExternalAccess
ExtraProperties{} (all elements)
FromApp
GroupName
HighPriorityMediaProcessing
ImplicitShare
IncidentId
InsightData{} (all elements)
InsightId
IntraSystemId
Members{} (all elements)
NonPIIParameters
OperationScope
OrganizationName
OriginatingServer
ResultCount
SharePointMetaData.SharedBy{} (all nested elements)
SharingType
SourceApp
StartTime
SupportTicketId
TabType
TemplateTypeId
TenantName
UniqueSharingId
UserServicePlan

ESCU-specific fields (20 fields - REMOVED ONLY IN CIM ONLY COMPATIBILITY MODE)¶

Warning: Removing these fields breaks ESCU detections

The following fields are required by Enterprise Security Content Updates (ESCU). Removing them causes ESCU detections to fail.

Alert and Incident fields:

AlertEntityId
Category
Status

Application fields:

ApplicationId
ClientAppId

Network fields:

ClientIPAddress

Correlation fields:

CorrelationId
InterSystemsId
SessionId

Data and parameters:

Data
EventData
Parameters
CreationTime

Message and Value fields:

MessageId
NewValue
OldValue

SharePoint fields:

SiteUrl
Source
SourceFileExtension
Folder

Device fields:

DeviceProperties{} (all elements)

Total: 127 fields removed (107 noise fields + 20 ESCU-specific)

Supported sourcetypes¶

This template processes events with the o365:management:activity sourcetype. Events with other sourcetypes are passed through without processing.

Template outline¶

Template consists of a custom function followed by a pipeline that uses dual output branching.

Functions

The following table shows all custom functions:

Function name	Description
`noise_reduction`	This function removes 107 noise fields AND 20 ESCU-specific fields for maximum reduction. Maintains CIM compatibility only. The reduced event is stored back to _raw.

Pipeline

This pipeline applies dual output branching:

Original untouched event is sent to $low_priority_destination (archival/backup).
Processed event with aggressive noise reduction is sent to $destination (primary indexing).

Configuration instructions¶

If any field listed in the template description is present in the event, the noise_reduction function deletes it. If you don’t want to delete all the fields, you can modify the field names provided in json_delete accordingly.

You can find examples in the following section.

Configuration example scenarios¶

Scenario 1: Keep ESCU field CorrelationId for partial ESCU support

Perform the following steps to keep the CorrelationId field:

In the noise_reduction function, remove "CorrelationId" from the $escu_specific_fields array.
Execute the pipeline preview and confirm that the field is not removed.
Save the changes.

Note

Keeping some ESCU fields may restore compatibility with specific detections, but full ESCU support requires the CIM & ESCU compatibility template.

Scenario 2: Remove additional custom field

Perform the following steps to remove an additional field (for example, CustomField):

In the noise_reduction function, add "CustomField" to either $conservative_fields or $escu_specific_fields array.
Execute the pipeline preview and confirm that the field is removed.
Save the changes.

Scenario 3: Disable dual output branching (single destination only)

Perform the following steps to send events only to the primary destination:

Modify the pipeline to remove the branch statement:

$pipeline = | from $source
| noise_reduction
| into $destination;

Execute the pipeline preview and confirm events go only to $destination.
Save the changes.

Performance impact¶

Metric	Impact
Space reduction	about 15 to 25% (varies by O365 workload)
Processing overhead	Minimal (less than 5% CPU increase)
CIM impact	None – all required CIM fields preserved
ESCU impact	None – all detection fields preserved

When to use this template¶

Use this CIM only compatibility template when:

You are NOT using Splunk Enterprise Security
You want maximum space reduction (127 fields removed)
You only need CIM compatibility
You don’t need ESCU detections
You need to archive full events for potential reprocessing

Consider the CIM & ESCU compatibility template when:

You are using Enterprise Security
You need ESCU detection compatibility
You want safe, proven noise reduction (107 fields removed)

Migration from CIM only to CIM & ESCU compatibility¶

If you start with the CIM only compatibility template and later need ESCU support:

Keep archived events: Full events in $low_priority_destination can be reprocessed.
Switch to CIM & ESCU compatibility pipeline: Deploy CIM & ESCU compatibility mode for new events.
Reprocess historical data: (Optional) Reprocess archived events using CIM & ESCU compatibility pipeline.
Enable ESCU: Activate Enterprise Security detections.

The dual output branching architecture ensures you always have access to full events for reprocessing.