Skip to content

Troubleshoot the Splunk Add-on for Google Cloud Platform

General troubleshooting

For troubleshooting tips that you can apply to all add-ons, see Troubleshoot add-ons in Splunk Add-ons. For additional resources, see Support and resource links for add-ons in Splunk Add-ons.

Data not showing up

If you are upgrading from a version of the Splunk Add-on for Google Cloud Platform earlier than 1.2.0, you must run the following upgrade script in order for your data to be properly sent. python3 $SPLUNK_HOME/etc/apps/Splunk_TA_google-cloudplatform/bin/tools/migrate_pusbub_input.py

Accessing logs

You can access internal log data for help with troubleshooting. Data collected with these source types does not appear in any dashboards.

Data source Source type
Logs from *google_cloud_pubsub*.log google:gcp:pubsub:log
Logs from *google_cloud_monitoring*.log google:gcp:monitoring:log
Logs from *google_cloud_billing*.log google:gcp:billing:log
Logs from *google_cloud_pubsub_lite*.log google:gcp:pubsublite:log
Logs from *google_cloud_pubsub_based_bucket*.log google:gcp:pubsubbasedbucket:log
Logs from *google_cloud_bucket_metadata*.log google:gcp:buckets:log
Logs from *google_cloud_resource_metadata*.log google:gcp:resource_metadata:log
Logs from *google_cloud_resthandler*.log google:gcp:resthandler:log

Configure log levels

  1. Click Splunk Add-on for Google Cloud Platform in your left navigation bar on Splunk Web’s home page.
  2. Click Configuration in the app navigation bar.
  3. Click the Logging tab.
  4. Adjust the log levels for each of the Google Cloud Platform services as needed by changing the default of INFO to one of the other available options, DEBUG or ERROR.
  5. (Optional) If you are using pub/sub, restart your Splunk instance to apply changes.

These log level configurations apply only to runtime logs. Some REST endpoint logs from configuration activity log at DEBUG, and some validation logs log at ERROR. These levels cannot be configured.

Large pub/sub subscriptions

For large pub/sub subscriptions, we recommend cloning existing inputs that are ingesting the same subscriptions to increase data throughput and performance. These identical inputs can be in the same instance or in different instances.

To manage a large number of subscriptions to one Splunk instance, aggregate subscriptions belonging to the same Google Cloud Service account into one input to save resources.

Exceed Request Limit

If you see any insufficient tokens for quota group errors such as the following, then you have exceeded the Google Cloud Monitoring request limit, which is 50000 per day. You should limit your request or apply for higher quota in Google: 

<"Insufficient tokens for quota group and limit DefaultGroupCLIENT_PROJECT-1d using the limit by ID 342432.">

Python version issues

If you are upgrading from versions 1.2.0 or lower, upgrade your Splunk Enterprise deployment to work with Python 3

  1. Plan your Splunk Enterprise upgrade to work with the Python 3 migration.
  2. Clean up all pycache files from your add-on’s directory location. For example, $SPLUNK_HOME/etc/apps/Splunk_TA_google-cloudplatform

Billing data not ingesting

If you encounter the following message in your internal logs:

Billing ingestion error. You must use the Cloud BigQuery Billing input in order to ingest billing data. You must delete your existing billing inputs, upgrade to version 3.2.0 or later of this add-on, and recreate your billing inputs using the Cloud BigQuery Billing input in order to ingest billing data.

Google has deprecated regular file export of your Cloud Billing data to CSV and JSON. To export your Cloud Billing data for analysis, use Cloud Billing export to BigQuery.

See the Configure Cloud BigQuery Billing inputs for the Splunk Add-on for Google Cloud Service topic in this manual.

Missing fields in Events/Larger events not structured in Pub/Sub input

If you get a larger size of events and face an issue with some missing fields in the Events or Events not structured correctly then perform the steps below:

  1. Click on Settings
  2. Click on Source types under the DATA section
  3. Unclick Show only popular checkbox if it is already checked
  4. Search google:gcp:pubsub:message sourcetype
  5. Click on Edit, it will open the Edit Source Type: google:gcp:pubsub:message dialogue box
  6. Click on Advanced
  7. Increase the TRUNCATE value based on Event size and click on the Save button
  8. Perform 4 to 7 steps for google:gcp:pubsub:audit:change sourcetype

Account Type information not getting updated in Google Credentials Configuration Tab after editing the existing credentials

Reload the configuration page to reflect the updated Account Type value will.

Manually enable/disable the scripted input for Cloud Storage Bucket

In case of a migration failure for specific cloud storage bucket inputs, the best practice is to initiate a rerun of the migration script. To manually enable or disable the input migration script $SPLUNK_HOME/etc/apps/Splunk_TA_google-cloudplatform/bin/google_cloud_storage_bucket_conf_migrator.py for Cloud Storage Bucket input, navigate to Settings>Data inputs>Scripts.