Skip to content

Overview ↵

About the Splunk Add-on for IBM WebSphere Application Server

Component Description
Version 5.2.0
Vendor Product(s) IBM WebSphere Application Server versions 8.5.x, 9.0.0, 9.0.5.6, 9.0.5.8, 9.0.5.12

The Splunk Add-on for IBM WebSphere Application Server allows a Splunk software administrator to collect data from WebSphere Application Servers. The add-on can collect JMX metrics, HPEL logging events, and server logs, including status, trace, error, and exception logs.

Release notes

For a summary of new features, fixed issues, and known issues, and for more information on release history, see Release notes for the Splunk Add-on for IBM WebSphere Application Server.

Compatibility

This add-on provides the inputs and Common Information Model (CIM)-compatible knowledge to use with other Splunk apps, such as Splunk Enterprise Security, the Splunk App for PCI Compliance, and Splunk IT Service Intelligence.

Source types and lookups

For more information about the source types and lookups for Splunk Add-on for Juniper, see Source types and Lookups.

Download the Add-on

Download the Splunk Add-on for IBM WebSphere Application Server from Splunkbase at http://splunkbase.splunk.com/app/2789.

Install and configure the add-on

To install and configure the Splunk Add-on for IBM WebSphere Application Server, see Installation overview.

Hardware and software requirements

For more information, see Hardware and software requirements.

Additional resources

Discuss the Splunk Add-on for IBM WebSphere Application Server on Splunk Community.

Release notes for the Splunk Add-on for IBM WebSphere Application Server

Version 5.2.0 of the Splunk Add-on for IBM WebSphere Application Server was released on September 16, 2024.

Compatibility

Version 5.2.0 of the Splunk Add-on for IBM WebSphere Application Server is compatible with the following software, CIM versions, and platforms.

Component Description
Splunk platform versions 9.0.x, 9.1.x, 9.2.x, 9.3.x
CIM 5.0.1
Platforms Platform independent
Vendor Products IBM WebSphere Application Server versions 8.5.x, 9.0.0, 9.0.5.6, 9.0.5.8, and 9.0.5.12

Note

The field alias functionality is compatible with the current version of this add-on. The current version of this add-on does not support older field alias configurations.

For more information about the field alias configuration change, refer to the Splunk Enterprise Release Notes.

New features

Version 5.2.0 of the Splunk Add-on for IBM WebSphere Application Server has the following new features.

  • Support for IPv6. Splunk Add-on for IBM WebSphere Application Server version 5.2.0 is now compatible with Splunk running on the IPv6 environment.
  • Python 3.9 support

Fixed issues

Version 5.2.0 of the Splunk Add-on for IBM WebSphere Application Server has the following fixed issues.

Known issues

Version 5.2.0 of the Splunk Add-on for IBM WebSphere Application Server has the following known issues.

Third-party software attributions

Some of the components included in this add-on are licensed under free or open source licenses. We wish to thank the contributors to those projects.

A complete listing of third-party software information for this add-on is available as a text file for download: Splunk Add-on for IBM WAS third-party software credits.

Release history for the Splunk Add-on for IBM WebSphere Application Server

Latest version

The latest version of the Splunk Add-on for IBM WebSphere Application Server is version 5.2.0. See Release notes for the Splunk Add-on for IBM WebSphere Application Server for the release notes of this latest version.

Version 5.1.0

Compatibility

Version 5.1.0 of the Splunk Add-on for IBM WebSphere Application Server is compatible with the following software, CIM versions, and platforms.

Component Description
Splunk platform versions 8.1.x, 8.2.x, 9.0.x
CIM 5.0.1
Platforms Platform independent
Vendor Products IBM WebSphere Application Server versions 8.5.x, 9.0.0, 9.0.5.6, 9.0.5.8, and 9.0.5.12

Note

The field alias functionality is compatible with the current version of this add-on. The current version of this add-on does not support older field alias configurations.

For more information about the field alias configuration change, refer to the Splunk Enterprise Release Notes.

New features

Version 5.1.0 of the Splunk Add-on for IBM WebSphere Application Server has the following new features.

  • Support for IBM Websphere version 9.0.5.12
  • Compatibility with CIM version 5.0.1
  • Corrected the input for the ibm:was:profileCreationLog sourcetype to monitor only the expected files.

Fixed issues

Version 5.1.0 of the Splunk Add-on for IBM WebSphere Application Server has the following fixed issues.

Known issues

Version 5.1.0 of the Splunk Add-on for IBM WebSphere Application Server has the following known issues.

Third-party software attributions

Some of the components included in this add-on are licensed under free or open source licenses. We wish to thank the contributors to those projects.

A complete listing of third-party software information for this add-on is available as a text file for download: Splunk Add-on for IBM WAS third-party software credits.

Version 5.0.0

Version 5.0.0 of the Splunk Add-on for IBM WebSphere Application Server was released on September 24, 2021

Compatibility

Version 5.0.0 of the Splunk Add-on for IBM WebSphere Application Server is compatible with the following software, CIM versions, and platforms.

Component Description
Splunk platform versions 8.0.x, 8.1.x, 8.2.x
CIM 4.20.0
Platforms Platform independent
Vendor Products IBM WebSphere Application Server versions 8.5.x, 9.0.0, 9.0.5.6 and 9.0.5.8

Note

The field alias functionality is compatible with the current version of this add-on. The current version of this add-on does not support older field alias configurations.

For more information about the field alias configuration change, refer to the Splunk Enterprise Release Notes.

New features

Version 5.0.0 of the Splunk Add-on for IBM WebSphere Application Server has the following new features.

  • Support for IBM Websphere 9.0.5.6 and 9.0.5.8
  • Support for Splunk Universal Forwarder
  • Provides static file monitoring stanzas for each sourcetype by default
  • Removed programmatic creation of file monitoring stanzas. The stanzas will now need to be set manually in inputs.conf. The default inputs.conf has some default stanzas for each sourcetype that can be used.
  • Removed user interface - you can only use configurations with a deployment server or backend changes.
  • Multiple Server Support
  • CIM mapping and enhancements
    • ibm:was:gcLog sourcetype is now mapped to Performance.Memory data model instead of Compute_Inventory.All_Inventory
    • ibm:was:httpLog sourcetype is now mapped to Web:Web instead of Performance.All_Performance
    • Removed data model mapping from ibm:was:serverIndex sourcetype.
    • Threadpool MBean events are mapped to the JVM.Threading data model.
    • Added support for CIM v4.20.0
  • Migrated ibm_was_inventory CSV lookup to KV Store lookup
  • Removed Python2 support. This add-on only supports python3 for future releases.
  • Removed support for Splunk 7.x.
  • Removed prebuilt panels shipped with the add-on.

Fixed issues

Version 5.0.0 of the Splunk Add-on for IBM WebSphere Application Server has the following fixed issues.

Known issues

Version 5.0.0 of the Splunk Add-on for IBM WebSphere Application Server has the following known issues.

Third-party software attributions

Some of the components included in this add-on are licensed under free or open source licenses. We wish to thank the contributors to those projects.

A complete listing of third-party software information for this add-on is available as a PDF file for download: Splunk Add-on for IBM WAS third-party software credits.

Version 4.0.1

Compatibility

Version 4.0.1 of the Splunk Add-on for IBM WebSphere Application Server is compatible with the following software, CIM versions, and platforms.

Component Description
Splunk platform versions 7.0.x, 7.1.x, 7.2.x, 7.3.x, 8.0.x
CIM 4.14 and above
Platforms Platform independent
Vendor Products IBM WebSphere Application Server versions 8.5.5 - 9.0.0

Note

The field alias functionality is compatible with the current version of this add-on. The current version of this add-on does not support older field alias configurations.

For more information about the field alias configuration change, refer to the Splunk Enterprise Release Notes.

New features

Version 4.0.1 of the Splunk Add-on for IBM WebSphere Application Server has the following new features.

  • Default support for Python3

Fixed issues

Version 4.0.1 of the Splunk Add-on for IBM WebSphere Application Server has the following fixed issues.

Known issues

Version 4.0.1 of the Splunk Add-on for IBM WebSphere Application Server has the following known issues.

Third-party software attributions

Version 4.0.1 of the Splunk Add-on for IBM WebSphere Application Server incorporates the following third-party software or libraries.

Version 4.0.0

Compatibility

Version 4.0.0 of the Splunk Add-on for IBM WebSphere Application Server is compatible with the following software, CIM versions, and platforms.

Component Description
Splunk platform versions 7.0.x, 7.1.x, 7.2.x, 7.3.x, 8.0.x
CIM 4.14 and above
Platforms Platform independent
Vendor Products IBM WebSphere Application Server versions 8.5.5 - 9.0.0

New features

Version 4.0.0 of the Splunk Add-on for IBM WebSphere Application Server has the following new features.

  • Support for Python3

Fixed issues

Version 4.0.0 of the Splunk Add-on for IBM WebSphere Application Server has the following fixed issues.

Known issues

Version 4.0.0 of the Splunk Add-on for IBM WebSphere Application Server has the following known issues.

Third-party software attributions

Version 4.0.0 of the Splunk Add-on for IBM WebSphere Application Server incorporates the following third-party software or libraries.

Version 3.1.0

Compatibility

Version 3.1.0 of the Splunk Add-on for IBM WebSphere Application Server is compatible with the following software, CIM versions, and platforms.

Component Description
Splunk platform versions 6.0 and above
CIM 4.2 and above
Platforms Platform independent
Vendor Products IBM WebSphere Application Server versions 7.0.0 - 8.5.5

Migration Guide

The Splunk Add-on for IBM WebSphere Application server replaces the Splunk App for WebSphere Application Server in its entirety. There is no backwards compatibility between this add-on and the old app and its two add-ons. If you have the old app and add-ons installed, uninstall or disable them and begin collecting new and historical data with this new add-on instead.

Upgrade Guide

The line breaker rule for the http_access.log and http_error.log logs has changed in the Splunk Add-on for IBM WebSphere Application Server version 3.1.0. If you had enabled http access logging and http error logging in IBM WebSphere and collected http_access.log and http_error.log using the Splunk Add-on for IBM WebSphere Application Server 3.0.0, you will need to fix the line breaks in the older data after upgrading to the Splunk Add-on for IBM WebSphere Application Server 3.1.0.

New features

Version 3.1.0 of the Splunk Add-on for IBM WebSphere Application Server has the following new features.

  • Modifications to support integration with ITSI including two new data sources: gc.log and serverindex.xml. These new sources are mapped to the Application Server and OS ITSI data models. Other sources are also mapped to the Application Server and OS ITSI data models. In addition, some sources are mapped to the Inventory and Performance CIM data models. See the source types table for more information.
  • Added saved search Server Index - WAS Inventory Lookup to generate a lookup file that is used to correlate data from multiple logs and populate certain fields in the events.

Fixed issues

Version 3.1.0 of the Splunk Add-on for IBM WebSphere Application Server has the following fixed issues.

Known issues

Version 3.1.0 of the Splunk Add-on for IBM WebSphere Application Server has the following known issues.

Third-party software attributions

Version 3.1.0 of the Splunk Add-on for IBM WebSphere Application Server incorporates the following third-party software or libraries.

Version 3.0.0

Compatibility

Version 3.0.0 of the Splunk Add-on for IBM WebSphere Application Server has the same compatibility specifications as version 3.1.0.

Migration Guide

The Splunk Add-on for IBM WebSphere Application server replaces the Splunk App for WebSphere Application Server in its entirety. There is no backwards compatibility between this add-on and the old app and its two add-ons. If you have the old app and add-ons installed, uninstall or disable them and begin collecting new and historical data with this new add-on instead.

New features

Version 3.0.0 of the Splunk Add-on for IBM WebSphere Application Server has the following new features.

  • New Splunk-supported add-on that replaces the Splunk App for WebSphere Application Server and brings it up to date through version 8.5.5, including support for High Performance Extensible Logging (HPEL) in WAS 8.x.

Known issues

Version 3.0.0 of the Splunk Add-on for IBM WebSphere Application Server has the following known issues.

Third-party software attributions

Version 3.0.0 of the Splunk Add-on for IBM WebSphere Application Server incorporates the following third-party software or libraries.

Hardware and software requirements for the Splunk Add-on for IBM WebSphere Application Server

Dependencies

The Splunk Add-on for IBM WebSphere Application Server supports multiple data inputs, each capable of collecting different data from your WAS. For more information about which kind of data you can collect with which input, refer to the source types page.

If you want to collect data from the HPEL interface or by monitoring log files, you must install a Splunk Enterprise forwarder, Splunk Universal forwarder or single instance directly on the machine running your WebSphere application server, so it can access the logs locally.

Note

If you are using an Universal Forwarder, you must use Python version 3.7 or 3.9 to forward and collect HPEL logs.

If you want to collect JMX data, install the Splunk Add-on for Java Management Extensions (JMX) on your data collection node.

Note

To collect JMX data from an IBM WebSphere application server, you need to install the IBM JDK or JRE, which you can download here: http://www.ibm.com/developerworks/java/jdk/eclipse/. Put the JDK or JRE bin directory in the system path to make sure that the JMX data collection uses this IBM version of the Java runtime.

Sizing guidelines

The Splunk Add-on for IBM WebSphere Application Server has no specific sizing guidelines for JMX or file monitoring inputs. For HPEL interface inputs, the add-on can concurrently collect metrics from twenty WebSphere profiles without performance implications.

Splunk platform requirements

Because this add-on runs on the Splunk platform, all of the system requirements apply for the Splunk software that you use to run this add-on.

  • For Splunk Enterprise system requirements: see System Requirements in the Splunk Enterprise Installation Manual.
  • If you are using Splunk Universal Forwarder to forward data: see System Requirements in the Splunk Universal Forwarder Installation Manual.
  • If you are managing on-premises forwarders to get data into Splunk Cloud, see System Requirements in the Splunk Enterprise Installation Manual, which includes information about forwarders.

Installation overview for the Splunk Add-on for IBM WebSphere Application Server

To install and configure the Splunk Add-on for IBM WebSphere Application Server on your supported platform:

  1. Download the add-on from Splunkbase.

  2. Install the Splunk Add-on for IBM WebSphere Application Server.

  3. Configure your WebSphere application server to enable server file system logs, HPEL logs, and/or JMX data.

  4. On the part of your Splunk Enterprise architecture that is performing data collection for the add-on, configure the inputs that you want to use:

  5. If you plan to use the Splunk Add-on for IBM WebSphere Application Server with IT Service Intelligence (ITSI), enable the saved search for the Splunk Add-on for IBM WebSphere Application Server.

Ended: Overview

Installation ↵

Install the Splunk Add-on for IBM WebSphere Application Server

Installation instructions

See Installing add-ons in Splunk Add-Ons for detailed instructions describing how to install a Splunk add-on in the following deployment scenarios:

Deployment notes

There are three different inputs in this add-on. You can choose to use one or several of them. Keep these requirements in mind as you choose how to install the add-on in your environment.

  • The JMX input depends on the Splunk Add-on for JMX, which must be installed on a heavy forwarder or single-instance Splunk Enterprise.
  • The HPEL interface modular input must be enabled on a forwarder or single-instance Splunk Enterprise that is installed directly on the machine running your WebSphere application server. This input requires Python. You can configure the input either using the UI or using configuration files. Both Heavy forwarders and Universal forwarders are supported.
  • The file monitoring inputs must be enabled on a forwarder or single-instance Splunk Enterprise that is installed directly on the machine running your WebSphere application server. This input does not require Python. You can configure the input either using the UI or using configuration files. Universal forwarders or heavy forwarders are all supported.

If you have many WebSphere application server instances, Universal Forwarders are recommended. You can configure the Add-on on multiple Universal Forwarders using a Deployment Server.

Alternatively, consider using an aggregator between the WebSphere application servers and the Splunk platform. In this configuration scenario, you can install a Splunk forwarder and the Splunk Add-on for IBM WebSphere Application Server on the aggregation server rather than on each WebSphere application server to monitor the HPEL and local server logs.

Distributed deployments

Use the tables below to determine where and how to install this add-on in a distributed deployment of Splunk Enterprise.

Where to install this add-on

This table provides a quick reference for installing this add-on to a distributed deployment of Splunk Enterprise.

Splunk instance type Supported Required Comments
Search Heads Yes Yes Install this add-on to all search heads where IBM WebSphere knowledge management is required.
Indexers Yes Conditional Not required if you use heavy forwarders to collect data. Required if you use universal forwarders to collect data.
Heavy Forwarders Yes Conditional Required for the JMX input. Optional for HPEL and file monitoring inputs. The HPEL and file monitoring inputs must be enabled on a forwarder that is installed directly on the machine running your WebSphere application server.
Universal Forwarders Yes Conditional Not supported for JMX inputs. Supported for HPEL and file monitoring inputs.. The file monitoring inputs must be enabled on a forwarder that is installed directly on the machine running your WebSphere application server.

Note If you are using an Universal Forwarder, you must use Python version 3.7 to forward and collect HPEL logs.

This table provides a quick reference for the compatibility of this add-on with Splunk distributed deployment features.

Distributed deployment feature Supported Comments
Search Head Clusters Yes You can install this add-on on a search head cluster for all search-time functionality, but only configure inputs on forwarders to avoid duplicate data collection.
Before installing this add-on to a cluster, make the following changes to the add-on package:
- Remove the inputs.conf file.
Indexer Clusters Yes Before installing this add-on to a cluster, make the following changes to the add-on package:
- Remove the inputs.conf file.
Deployment Server Yes Note: Using a deployment server to deploy the configured add-on to multiple forwarders acting as data collectors causes duplication of data.

Upgrade to version 5.0.0 or later

Upgrade file monitoring settings

Previously, the add-on was creating file monitoring stanzas programmatically after asking for the IBM WAS installation directory as an input through the UI. We have removed the programmatic approach and have provided default stanzas for each sourcetype in inputs.conf for the user to use. These default stanzas will leverage the WASHOME environment variable for an IBM WAS system. Make sure that this WASHOME variable is properly configured on all the machines having IBM WAS server.

See the following example:

[monitor://$WASHOME\...\derby.log]

crcSalt = SOURCE

followTail = 1

sourcetype = ibm:was:derby

disabled = true

index = default

Remove your old monitoring stanzas and replace them with the new ones in order to use the field extractions provided by the add-on. Perform the following steps:

  1. Take the backup of your local/inputs.conf where the file monitoring stanzas are.
  2. Remove the old file monitoring stanzas.
  3. Enable the desired file monitoring stanzas in the local/inputs.conf which are already provided in the default/inputs.conf of the add-on. Example, if your IBM WAS is installed on windows OS; in local/inputs.conf, write the following to enable data collection for derby logs:

[monitor://$WASHOME\...\derby.log] disabled = true

For Unix or Linux based OSs:

[monitor://$WASHOME/.../derby.log] disabled = true

Note that there are different stanzas windows and “not” windows OSs due to their different ways of specifying a path.

  1. Make sure that the WASHOME environment variable is properly configured on the system. For windows OS, set the variable in System Properties -> Environment Variables -> System Variables.
  2. For other Unix and Linux based OS, set the variable in the .profile file on your system.
  3. Restart the Splunk instance after making the above changes.

Upgrade HPEL settings

Previously, the user was able to collect HPEL logs for only a single default server under a profile. In version 5.0.0, we have added support to collect HPEL logs from all servers under a WAS profile. As a result of which, the add-on will start collecting logs for all servers under the selected profiles instead of just the default one. If you want to avoid this, configure the newly added “excluded_servers” parameter in your local/ibm_was.conf under the was_hpel_settings stanza to selectively exclude servers. See Configure global settings, HPEL inputs, and server log inputs for more information about this parameter.

Also, the hpel_collection_enabled parameter under the was_hpel_settings stanza is deprecated from the v5.0.0 onwards. Use the “disabled” property for the ibm_was://was_data_input stanza in inputs.conf to toggle the HPEL data collection.

Upgrade lookups

In version 5.0.0, the CSV lookup ibm_was_inventory.csv has been migrated to KV Store. Run the following search on your search heads after the upgrade to migrate the existing data to new KVStore lookup:

| inputlookup ibm_was_inventory.csv | outputlookup ibm_was_inventory

For more details on how to migrate a CSV lookup to KV Store, see Migrate your Splunk app from using CSV files to KV store.

Ended: Installation

Configuration ↵

Configure IBM WebSphere to produce data for the Splunk Add-on for IBM WebSphere Application Server

The Splunk Add-on for IBM WebSphere Application Server allows you to collect three different data types from your WAS instances:

  • server file system logs
  • HPEL interface logs for applications
  • JMX metrics

You need to configure your IBM WebSphere application servers to produce these logs using the WebSphere administrative console.

Enable server file system logs

Enable PMI metrics

You can configure Performance Monitoring Infrastructure (PMI) metrics in IBM WebSphere to control the number of Mbeans the Splunk Add-on for IBM WebSphere Application Server can collect. Although the basic Mbeans used for AppServer data model mapping for ITSI can be retrieved without enabling PMI metrics in IBM WebSphere, you can collect additional Mbeans by configuring PMI according to your needs.

To enable all PMI metrics, perform the following steps:

  1. Log in to the WebSphere administrative console and navigate to Servers > Server Types > WebSphere application servers > .
  2. Select the Configuration tab then select Performance Monitoring Infrastructure (PMI).
  3. Check Enable Performance Monitoring Infrastructure and select All under Currently monitored statistic set.

For more information about enabling PMI, see https://www.ibm.com/support/knowledgecenter/en/SSEQTP_9.0.5/com.ibm.websphere.base.doc/ae/tprf_prfstartadmin.html.

Enable verbose garbage collection

  1. Log in to the WebSphere administrative console and navigate to the Java Virtual Machine: Servers > Server Types > WebSphere application servers > > Server Infrastructure > Java and Process Management > Process definition > Java Virtual Machine.
  2. Enable verbose garbage collection and specify the gc.log file path. The log path must be under the ${SERVER_LOG_ROOT} directory. For example: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1.

For more information, see the IBM documentation: http://www-01.ibm.com/support/docview.wss?uid=swg21114927.

To monitor verbose garbage collection files, you can enable the following default stanza provided in the add-on:

  • On Windows environments, enable the following stanza: [monitor://$WASHOME\...\*gc.*log]

  • On Linux/Unix environments, enable the following stanza: [monitor://$WASHOME/.../*gc.*log]

Note

If you were using the old stanzas, such as "[monitor://$WASHOME\...\*gc.log] (for Windows) and [monitor://$WASHOME/.../*gc.log] (for Linux) to monitor garbage collection logs, and you upgraded to IBM WAS version 8 or greater, use the default stanzas that correspond with IBM’s latest file name convention.

Enable http access logging and http error logging

  1. Login into the WebSphere administrative console.
  2. Navigate to Servers > Server Types > WebSphere application servers > >Troubleshooting > NSCA access and HTTP error logging.
  3. On the NSCA access and HTTP error logging screen, enable the following: - Enable logging service at server start-up - Enable access logging - Enable error logging

Turn on HPEL logging

To turn on High Performance Extensible Logging (HPEL) logging for your WebSphere applications, follow the instructions in the IBM documentation that match your WAS version. See instructions for version 8.5.5 at http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_compToHPEL.html?cp=SSAW57_8.5.5.

Enable JMX in WebSphere

To extract JMX metrics for your WebSphere application server using the Splunk Add-on for JMX, you must first enable the JMX connector in WebSphere. Follow the detailed walkthrough for instructions on how to do this in WebSphere version 8.5 at https://www.ibm.com/docs/en/was-nd/9.0.5?topic=services-java-management-extensions-jmx-connectors.

Configure JMX inputs for the Splunk Add-on for IBM WebSphere Application Server

The Splunk Add-on for IBM WebSphere Application Server relies on the Splunk Add-on for Java Management Extensions (JMX) to collect JMX metrics. The Splunk Add-on for IBM WAS provides a jmx_templates.conf that the Splunk Add-on for JMX can invoke.

Note

To collect JMX data from an IBM WebSphere application server, you need to install the IBM JDK or JRE, which you can download from https://www.ibm.com/support/pages/java-sdk. Put the JDK or JRE bin directory in the system path to make sure that the JMX data collection uses this IBM version of the Java runtime.

  1. Install the Splunk Add-on for JMX on the Splunk Enterprise instance responsible for JMX data collection, usually a heavy forwarder. This add-on can collect JMX metrics locally or remotely.

  2. Go to your WebSphere installation directory and navigate to $WAS_ROOT/WebSphere/AppServer/runtimes.

  3. Copy the following files to $SPLUNK_HOME/etc/apps/Splunk_TA_jmx/bin/lib: - com.ibm.ws.admin.client_*.jar - com.ibm.ws.ejb.thinclient_*.jar - com.ibm.ws.orb_*.jar

  4. After copying, verify the files successfully copied to your $SPLUNK_HOME/etc/apps/Splunk_TA_jmx/bin/lib directory before continuing. For example, if you run the command ls /opt/splunk/etc/apps/Splunk_TA_jmx/bin/lib/com.ibm*, you should see something similar to:

   /opt/splunk/etc/apps/Splunk_TA_jmx/bin/lib/com.ibm.ws.admin.client_8.5.0.jar
   /opt/splunk/etc/apps/Splunk_TA_jmx/bin/lib/com.ibm.ws.ejb.thinclient_8.5.0.jar
   /opt/splunk/etc/apps/Splunk_TA_jmx/bin/lib/com.ibm.ws.orb_8.5.0.jar

  1. Restart the Splunk platform.

  2. Go to Splunk Web and access the configuration pages for the Splunk Add-on for JMX, either by clicking on the name in the left nav, or going to Apps > Manage Apps, then clicking Launch app in the row for Splunk Add-on for JMX.

  3. Select Add Server to add a new JMX server.

  4. Enter a Name and an optional JVM Description for your server.

  5. For Connection Type, choose Use URL directly from the dropdown menu.

  6. Enter the URL in this format: service:jmx:iiop://<hostname>/jndi/corbaname:iiop:<hostname>:9100/WsnAdminNameService#JMXConnector.

Note

The is a hostname of the running application server. Use the hostname instead of the IP address. If the hostname cannot be resolved through DNS, add the hostname to the hosts file. You can find the hostname for the application server through WAS administration console.

600px-WASconfig1.png

  1. Click Create.

  2. Navigate to the task configurations by clicking Configurations > Tasks.

  3. Click Add Task to create a new JMX task.

  4. Enter a Name and optional Description for your task, then select the server that you just configured.

  5. On the Templates tab, select one or more of the predefined templates to collect the data that you want.

  6. On the Settings tab, set the source type to ibm:was:jmx.

  7. Click Create to enable your JMX input.

For more information about configuring JMX inputs, refer to Configure inputs for the Splunk Add-on for JMX in the Splunk Add-on for Java Management Extensions manual.

Configure global settings, HPEL inputs, and server log inputs for IBM WebSphere Application Server

Configuring your HPEL inputs and your file monitor inputs for your server log files requires for several steps. You can perform all of this configuration using configuration files.. You can use a deployment server for configuring the add-on on your forwarders or configure it directly on the forwarders using the configuration files.

Prerequisites:

  • Ensure you have configured your IBM WebSphere application servers to enable logging for the logs you want to collect with the add-on as described in Configure IBM WebSphere to produce data for the Splunk Add-on for IBM WebSphere Application Server.
  • If you are using a Universal Forwarder, you must be using Python version 3.7 or 3.9 to forward or collect HPEL logs.
  • In the newer versions of the Splunk Universal Forwarder versions 9.x and above, the management port is not enabled by default, this is required for the IBM WebSphere Application Server add-on. To activate the management port in Splunk Universal Forwarder follow these steps:
  1. For Linux, add the following to splunkforwarder/etc/system/local/server.conf: [httpServer] mgmtMode = tcp
  2. In splunkforwarder/etc/system/local/web.conf, add the following (update the management port): [settings] mgmtHostPort = 127.0.0.1:8089
  3. For Windows, add the following to SplunkUniversalForwarder\etc\system\local\server.conf: [general] python.version=unspecified

Add the following line to set the PYTHONPATH of your Python installation in the .conf file: SplunkUniversalForwarder\etc\splunk-launch.conf PYTHONPATH= < path to your python installation>

Configure HPEL and server log inputs using configuration files

Follow these steps to configure your global settings and your HPEL and server log inputs using the configuration files.

Configure global settings and HPEL settings in local/ibm_was.conf

  1. Copy $SPLUNK_HOME/etc/apps/Splunk_TA_ibm-was/default/ibm_was.conf to $SPLUNK_HOME/etc/apps/Splunk_TA_ibm-was/local/.

  2. Provide the root installation directory of your IBM WebSphere application server for the argument was_install_dir.

  3. All other parameters are optional. Refer to the table for information about each one.

Section Argument Description Default
Global settings index The index in which to store data collected with the Splunk Add-on for IBM WebSphere Application Server. main
Global settings was_install_dir Required. The installation directory of your IBM WebSphere application server. On Windows, the WebSphere installation path should include all spaces. For example: C:\Program Files (x86)\IBM\WebSphere. None
Global settings log_level The logging verbosity for the add-on. INFO
HPEL settings excluded_profiles Profiles to exclude from HPEL data collection separated by commas. For example, MyProfile.*,OtherProfile. None
HPEL settings excluded_servers Servers to exclude from HPEL data collection separated by commas specified in the format <Profile>:<ServerDir>. For example, ProfileA:ServerA1,ProfileB:ServerB3 None
HPEL settings start_date HPEL logs start date (UTC) in MM/dd/yy H:m:s:S format. For example, 6/29/15 00:00:00:000. Note that you can configure this only before you enable the input for the first time. 1 day ago
HPEL settings level Set a single log level to collect from the HPEL log data. This argument overrides any values in min_level and max_level. None
HPEL settings min_level Set a minimum log level to collect from the HPEL log data. Ensure the min_level is set to a lower level than max_level to define a valid range. INFO
HPEL settings max_level Set a maximum log level to collect from the HPEL log data. Ensure the max_level is set to a higher level than min_level to define a valid range. FATAL
HPEL settings duration The collection interval for the HPEL input. 60
  1. To collect the data for this sourcetype you need to enable IBM WAS Input from either the UI or backend:

  • From UI: Data inputs > Splunk Add-on for IBM WebSphere Application Server > Enable 'was_data_input'.

  • From Backend: You can enable this stanza from $SPLUNK_HOME/etc/apps/Splunk_TA_ibm-was/local/inputs.conf > [ibm_was://was_data_input]

Configure the file monitoring stanzas for server logs in local/inputs.conf

From the version 5.0.0, we have provided default file monitoring stanzas for each sourcetype in default/inputs.conf. You can enable these stanzas in the local/inputs.conf to start the data collection. These default stanzas will leverage the WASHOME environment variable of an IBM WAS server for the file and directory path in them. Please note that we have provided separate stanzas for windows OS and “non” windows OSs because these OSs use different separators in their file paths.

Examples:

Windows OS

[monitor://$WASHOME\...\derby.log]

crcSalt = SOURCE

followTail = 1

sourcetype = ibm:was:derby

disabled = true

index = default

Unix and Linux based OSs

[monitor://$WASHOME/.../derby.log]

crcSalt = SOURCE

followTail = 1

sourcetype = ibm:was:derby

disabled = true

index = default

In the local/inputs.conf, enable the stanzas as per your OS.

Make sure that the WASHOME environment variable is properly configured on the system having the IBM WAS server. For windows OS, set the variable in System Properties -> Environment Variables -> System Variables For other unix and linux based OS, set the variable in the .profile file on your system.

The setting followTail = 1 will let you skip over data in files, and immediately begin indexing current data, for example, it will not ingest the already present data in files but will only ingest new data to those files after enabling the stanza.

Validate the inputs

To validate that all the inputs you configured are working correctly, go to the Search and Reporting app and search for the source types listed on the source types page that match the inputs that you configured.

Configure monitor inputs for the gc.log and serverindex.xml logs

If you plan to use the Splunk Add-on for IBM WebSphere Application Server with IT Service Intelligence (ITSI), you need to create inputs for the gc.log and serverindex.xml logs. Although you can use the setup page to generate inputs for most of the local server logs, you must manually create monitor inputs for the gc.log and serverindex.xml logs. You can use either Splunk Web to create the monitor inputs or configure inputs.conf directly.

Configure Monitoring through Splunk Web

Configure file monitoring inputs on your data collection node for the gc.log and serverindex.xml logs.

  1. Log into Splunk Web.

  2. Select Settings > Data inputs > Files & directories.

  3. Click New.

  4. Click Browse next to the File or Directory field.

  5. Navigate to directory of your gc.log, and click Next.

  6. In the Sourcetype field, select ibm:was:gcLog, and click Next.

  7. Click Review.

  8. After you review the information, click Submit.

  9. Repeat this procedure for the serverindex.xml and assign a source type of ibm:was:serverIndex.

Configure inputs.conf

You can configure the monitor inputs in the inputs.conf file instead of using Splunk Web. After setting up the add-on, using either the setup page or using the configuration files and running the python script, a local/inputs.conf file gets generated for you. Edit this file to add the file monitor inputs.

  1. Using a text editor, open the $SPLUNK_HOME/etc/apps/Splunk_TA_ibm-was/local/inputs.conf file.

  2. Add the following stanzas and lines, replacing server_name, cell_name, and node_name with the appropriate values for your environment, and save the file:

    [monitor:///opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/<server_name>/gc.log]
    sourcetype = ibm:was:gcLog
    disabled = 0

    [monitor:///opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/$<cell_name>/nodes/$<node_name>/serverindex.xml]
    sourcetype = ibm:was:serverIndex
    disabled = 0
  1. Restart the Splunk platform in order for the new inputs to take effect.

Validate inputs for gc.log and serverindex.xml

After you configure monitoring, verify that data from the two sources is being ingested into the Splunk platform by using the following search commands and verifying that one or more events is returned.

sourcetype=ibm:was:gcLog sourcetype=ibm:was:serverIndex

Enable saved search for the Splunk Add-on for IBM WebSphere Application Server

The Splunk Add-on for IBM WebSphere Application Server includes a preconfigured lookup generation saved search that you need to enable if you are using this add-on with Splunk IT Service Intelligence. This saved search is based on the data collected through JMX and file based logs. You need to configure JMX inputs, configure server log inputs, and configure monitor inputs for the gc.log and serverindex.xml logs in order to collect the data. After the data has been indexed by the Splunk platform, manually run the saved search in order to populate the lookup file then set a frequency to run it that matches the frequency of configuration changes in your environment.

Saved search name Description
Server Index - WAS Inventory Lookup Generates the ibm_was_inventory lookup file. Populates the appserver_port_number and application_server fields in the events.

You can review and enable the saved search either in Splunk Web or in the configuration files.

Access and enable saved search in Splunk Web

To access and enable the saved search in Splunk Web, perform the following steps:

  1. Go to Settings > Searches, reports, and alerts.

  2. Set the app context to Splunk Add-on for IBM WebSphere Application Server.

  3. Click Enable next to Server Index - WAS Inventory Lookup.

Access and enable saved search in savedsearches.conf

To access and enable the saved search in the configuration files, perform the following steps:

  1. Go to $SPLUNK_HOME/etc/apps/Splunk_TA_tomcat/default/savedsearches.conf.

  2. Copy the file to /local.

  3. In the local copy, change disabled = 1 to disabled = 0.

Ended: Configuration

Troubleshooting ↵

Troubleshoot

Splunk restarts after installing or upgrading the Add-on when the JMX addon is not installed

The IBM WAS add-on will trigger a Splunk restart in case the JMX add-on is not installed on the same Splunk instance because of the jmx_templates.conf file. The jmx_templates.conf present in the IBM WAS add-on is responsible for triggering the restart if the JMX add-on is not installed. Install it to avoid a restart.

Ended: Troubleshooting

Reference ↵

Lookups for the Splunk Add-on for IBM WebSphere Application Server

The Splunk Add-on for IBM WebSphere Application Server has one KV store lookup.

Name Description
ibm_was_inventory Generated from Server Index - WAS Inventory Lookup saved search. Links the events from JMX and the events from file based logs and populates the appserver_port_number and application_server fields in all events.

Note

If you are upgrading to version 5.0.0, see Upgrade the Splunk Add-on for IBM WebSphere Application Server page to migrate your old CSV lookup to KV store.

Source types for the Splunk Add-on for IBM WebSphere Application Server

The Splunk Add-on for IBM WebSphere Application Server supplies or expects the following source types, depending on the data sources and collection methods that you configure: JMX events, HPEL logs, and other log files.

Collection method Source type Description CIM data models ITSI data models
Splunk Add-on for JMX ibm:was:jmx JMX Events for WebSphere JVM Application Server, OS
Local HPEL log collection ibm:was:hpel HPEL logging. Supported for IBM WebSphere Application Server version 8.X only. None None
Local log file monitoring ibm:was:manageprofiles logs under manageprofile directory None None
Local log file monitoring ibm:was:serverStatus server status logs None None
Local log file monitoring ibm:was:orbtrc ORB trace logs None None
Local log file monitoring ibm:was:serverExceptionLog Server exception logs None None
Local log file monitoring ibm:was:textLog Logs in TextLog directory None None
Local log file monitoring ibm:was:derby Derby database logs None None
Local log file monitoring ibm:was:ffdc First Failure Data Capture logs None None
Local log file monitoring ibm:was:startServerLog Start server logs None None
Local log file monitoring ibm:was:stopServerLog Stop server logs None None
Local log file monitoring ibm:was:systemOutLog System out logs None None
Local log file monitoring ibm:was:systemErrLog System error logs None None
Local log file monitoring ibm:was:nativeStdOutLog Native stdout logs None None
Local log file monitoring ibm:was:nativeStdErrLog Native stderr logs None None
Local log file monitoring ibm:was:profileCreationLog Profile creation logs None None
Local log file monitoring ibm:was:wsadminTraceout wasadmin trace logs None None
Local log file monitoring ibm:was:profileManagementLog Profile management logs None None
Local log file monitoring ibm:was:addNodeLog App node addition logs None None
Local log file monitoring ibm:was:activityLog Activity logs None None
Local log file monitoring ibm:was:httpErrorLog http_error.log None None
Local log file monitoring ibm:was:ivtClientLog IVT Client Logs None None
Local log file monitoring ibm:was:javacore Javacore logs None None
Local log file monitoring ibm:was:httpLog http_access.log Web Application Server, OS
Local log file monitoring ibm:was:serverIndex serverindex.xml None Application Server, OS
Local log file monitoring ibm:was:gcLog Garbage collection log (gc.log) Performance Application Server, OS

Ended: Reference