Skip to content

Okta System Logs: Reduce log size

Disclaimer

By using SPL2 templates for data processing (the “templates”), you understand and agree that templates are provided “as is”. Splunk disclaims any and all warranties, express or implied, including without limitation the implied warranties of merchantability, fitness for a particular purpose and warranties arising out of course of dealing or usage of trade or by statute or in law. Splunk specifically does not warrant that templates will meet your requirements, the operation or output of templates will be error-free, accurate, reliable, complete or uninterrupted.

Use case

Reduce the size of Okta System logs by removing unnecessary fields.

Template details

Compatibility

This template is compatible with Splunk Add-on for Okta Identity Cloud v4.0.0.

Template description

This is pipeline that reduces the size of Okta System log events while preserving compatibility with the Splunk Common Information Model (CIM) and Security Detections along with the fields which are important from customer perspective. This pipeline takes events with a source type OktaIM2:log and perfoms the Noise reduction in the two following configuration modes (through the commented code):

Configuration mode Definition Fields removed
escu_cim_only (default) Retains only ESCU + CIM fields actor.detailEntry
admin_interest
authenticationContext.authenticationProvider
authenticationContext.authenticationStep
authenticationContext.credentialProvider
authenticationContext.credentialType
authenticationContext.interface
authenticationContext.issuer
authenticationContext.issuer.id
authenticationContext.issuer.type
client.geographicalContext.geolocation.lat
client.geographicalContext.geolocation.lon
client.geographicalContext.postalCode
client.id
client.zone
debugContext.debugData.action
debugContext.debugData.adAttrName
debugContext.debugData.addedObjects
debugContext.debugData.appContextName
debugContext.debugData.appUserId
debugContext.debugData.appVersion
debugContext.debugData.appname
debugContext.debugData.approvertype
debugContext.debugData.assertion
debugContext.debugData.attributesAdded
debugContext.debugData.attributesDeleted
debugContext.debugData.attributesModified
debugContext.debugData.authCode
debugContext.debugData.authMethodFirstEnrollment
debugContext.debugData.authMethodFirstType
debugContext.debugData.authMethodFirstVerificationTime
debugContext.debugData.authType
debugContext.debugData.authenticationContext
debugContext.debugData.authenticatorKey
debugContext.debugData.authnRequestId
debugContext.debugData.authorizationServer
debugContext.debugData.authorizationServerName
debugContext.debugData.category
debugContext.debugData.changedAttributes
debugContext.debugData.clientAddress
debugContext.debugData.clientAuthType
debugContext.debugData.clientId
debugContext.debugData.clientName
debugContext.debugData.clientSecret
debugContext.debugData.compositeRequestEnabled
debugContext.debugData.concurrencyPercentage
debugContext.debugData.countryCallingCode
debugContext.debugData.countryCodeIso2
debugContext.debugData.currentRefreshTokenHash
debugContext.debugData.customMessage
debugContext.debugData.customObjectId
debugContext.debugData.customObjectType
debugContext.debugData.defaultAuthorizationServer
debugContext.debugData.delauthtimeout
debugContext.debugData.delauthtimespentatagent
debugContext.debugData.delauthtimespentatdomaincontroller
debugContext.debugData.delauthtimetotal
debugContext.debugData.deletedObjects
debugContext.debugData.detailedmessage
debugContext.debugData.deviceCategory
debugContext.debugData.deviceFingerprint
debugContext.debugData.devicePlatform
debugContext.debugData.distinguishedname
debugContext.debugData.emailProvider
debugContext.debugData.emailRequestId
debugContext.debugData.featurename
debugContext.debugData.flowContinuation
debugContext.debugData.grantType
debugContext.debugData.grantedScopes
debugContext.debugData.groupAppAssignmentId
debugContext.debugData.idpIdsWithSecrets
debugContext.debugData.importLastToken
debugContext.debugData.importTrigger
debugContext.debugData.importType
debugContext.debugData.initiationType
debugContext.debugData.invalidatedAppId
debugContext.debugData.isComposite
debugContext.debugData.isReauth
debugContext.debugData.jobId
debugContext.debugData.jobid
debugContext.debugData.kid
debugContext.debugData.lifecycle
debugContext.debugData.limitations
debugContext.debugData.linkedObjectAdded
debugContext.debugData.manualintervention
debugContext.debugData.oktaAttrName
debugContext.debugData.oldDisplayName
debugContext.debugData.oldRefreshTokenHash
debugContext.debugData.onOrBefore
debugContext.debugData.operationRateLimitScopeType
debugContext.debugData.operationRateLimitSecondsToReset
debugContext.debugData.operationRateLimitSubtype
debugContext.debugData.operationRateLimitThreshold
debugContext.debugData.operationRateLimitTimeSpan
debugContext.debugData.operationRateLimitTimeUnit
debugContext.debugData.operationRateLimitType
debugContext.debugData.origin
debugContext.debugData.phoneNumber
debugContext.debugData.pluginVersion
debugContext.debugData.policy
debugContext.debugData.privilegeGranted
debugContext.debugData.privilegeRevoked
debugContext.debugData.protocol
debugContext.debugData.providerMessage
debugContext.debugData.providerStatus
debugContext.debugData.radiusRequestId
debugContext.debugData.rateLimitBucketUuid
debugContext.debugData.rateLimitPercentage
debugContext.debugData.rateLimitScopeType
debugContext.debugData.rateLimitSecondsToReset
debugContext.debugData.reasoncode
debugContext.debugData.redirectUri
debugContext.debugData.requestId
debugContext.debugData.requestUri
debugContext.debugData.requestedScopes
debugContext.debugData.responseMode
debugContext.debugData.responseTime
debugContext.debugData.responseType
debugContext.debugData.risk
debugContext.debugData.rule
debugContext.debugData.sameBrowser
debugContext.debugData.sessionRevocation
debugContext.debugData.signOnMode
debugContext.debugData.smsProvider
debugContext.debugData.state
debugContext.debugData.suspiciousActivityBrowser
debugContext.debugData.suspiciousActivityEventCity
debugContext.debugData.suspiciousActivityEventCountry
debugContext.debugData.suspiciousActivityEventId
debugContext.debugData.suspiciousActivityEventIp
debugContext.debugData.suspiciousActivityEventLatitude
debugContext.debugData.suspiciousActivityEventLongitude
debugContext.debugData.suspiciousActivityEventState
debugContext.debugData.suspiciousActivityEventTransactionId
debugContext.debugData.suspiciousActivityEventType
debugContext.debugData.suspiciousActivityOs
debugContext.debugData.suspiciousActivityTimestamp
debugContext.debugData.tempPasswordSet
debugContext.debugData.threatSuspected
debugContext.debugData.threshold
debugContext.debugData.timeSpan
debugContext.debugData.timeUnit
debugContext.debugData.totalObjects
debugContext.debugData.totalTime
debugContext.debugData.transactionCompleted
debugContext.debugData.transactionId
debugContext.debugData.transactionStarted
debugContext.debugData.unchangedObjects
debugContext.debugData.updatedObjects
debugContext.debugData.userId
debugContext.debugData.usercomment
debugContext.debugData.verification
debugContext.debugData.warningPercent
debugContext.debugData.zoneData
event_type_description
event_type_tags
reason
release_note_date
request.ipChain{}.geographicalContext
request.ipChain{}.geographicalContext.city
request.ipChain{}.geographicalContext.country
request.ipChain{}.geographicalContext.geolocation.lat
request.ipChain{}.geographicalContext.geolocation.lon
request.ipChain{}.geographicalContext.postalCode
request.ipChain{}.geographicalContext.state
request.ipChain{}.source
securityContext.asNumber
securityContext.asOrg
securityContext.domain
securityContext.isProxy
securityContext.isp
security_interest
targetADAgentAlternateId
targetGroupPushMappingAlternateId
targetResourceRequestDisplayName
targetResourceSetAlternateId
targetRoleAlternateId
targetaccesstokentype
targetactivedirectoryDisplayName
targetactivedirectoryId
targetidtype
targetrefreshtype
version
enhanced_security Retains ESCU fields + CIM fields + extra fields debugContext.debugData.clientSecret
debugContext.debugData.importType
debugContext.debugData.operationRateLimitSecondsToReset
debugContext.debugData.operationRateLimitThreshold
debugContext.debugData.operationRateLimitTimeSpan
debugContext.debugData.operationRateLimitTimeUnit
debugContext.debugData.rateLimitBucketUuid
debugContext.debugData.rateLimitPercentage
debugContext.debugData.rateLimitScopeType
debugContext.debugData.rateLimitSecondsToReset

Supported sourcetypes

This template processes events only with the OktaIM2:log sourcetype. Events with other sourcetypes are passed through without processing.

Template outline

Template consists of few custom functions followed by a pipeline that uses these functions.

Functions

The following table shows all custom functions:

Function name Description
retains_only_escu_cim_fields This function removes unnecessary fields (Retains only ESCU + CIM fields) as mentioned above from original event and stores reduced event back to _raw.
retains_escu_cim_and_security_fields This function removes unnecessary fields (Retains ESCU + CIM + Important Security fields)as mentioned above from original event and stores reduced event back to _raw.

Pipeline

This pipeline has the following stage:

  • applies function for events having OktaIM2:log sourcetype and just pass-thru for other events.

Configuration instructions

If any field listed in Template description are present in the event, then based on the configuration mode selected (either retains_only_escu_cim_fields or retains_escu_cim_and_security_fields) delete them all. If you don’t want to delete certain fields, you can modify the field names provided in json_delete accordingly.

You can find the examples in the following section.

Configuration example scenario

Scenario 1: use default configration mode which retains only ESCU and CIM fields

Perform the following steps:

  1. Comment out the retains_escu_cim_and_security_fields function if it is not already commented.
$pipeline = | from $source
| retains_only_escu_cim_fields
// | retains_escu_cim_and_security_fields
| into $destination;
  1. Execute the pipeline preview and confirm that the field is removed as per the selected configuration mode.
  2. Save the changes.

Scenario 2: use configration mode which retains ESCU, CIM and Important security fields

Perform the following steps:

  1. Comment out the retains_only_escu_cim_fields function if it is not already commented.
$pipeline = | from $source
// | retains_only_escu_cim_fields
| retains_escu_cim_and_security_fields
| into $destination;
  1. Execute the pipeline preview and confirm that the field is removed as per the selected configuration mode.
  2. Save the changes.