Configure NGINX logging and monitoring¶

You need to set up the NGINX logging and monitoring to enable the Splunk Add-on for NGINX to collect data from the NGINX server including access log, error log, and performance metrics.

Configure NGINX access log¶

NGINX writes information about client requests in the access log right after the request is processed. By default, the access log is located at /var/log/nginx/access.log, and the information is written to the log in the predefined combined format. You can override the default settings and change the format of logged messages by editing the NGINX configuration file (by default it is the /etc/nginx/nginx.conf file). The Splunk Add-on for NGINX can ingest the NGINX access log in both the predefined combined format and the custom key-value pair format. Splunk recommends using the custom key-value pair format because it contains more information and is easier to parse.

Default NGINX access log¶

For information about setting up the default NGINX access log, refer to the NGINX documentation: https://www.nginx.com/resources/admin-guide/logging-and-monitoring/#access_log.

Custom NGINX access log¶

Edit the NGINX configuration file (by default it is the /etc/nginx/nginx.conf file) and use the log_format directive to define the format of logged messages based on your requirements.

Here is an example of logging in raw format for the nginx:plus:access source type:

log_format main '$remote_addr $server_name $remote_user [$time_local] "$request" '
                '$status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" $server_port '
                '$upstream_bytes_received "$sent_http_content_type" $host "$https" "$http_cookie"';

Here is an example of logging in KV format for the nginx:plus:kv source type:

log_format kv 'site="$server_name" server="$host" dest_port="$server_port" dest_ip="$server_addr" '
              'src="$remote_addr" src_ip="$realip_remote_addr" user="$remote_user" '
              'time_local="$time_local" protocol="$server_protocol" status="$status" '
              'bytes_out="$bytes_sent" bytes_in="$upstream_bytes_received" '
              'http_referer="$http_referer" http_user_agent="$http_user_agent" '
              'nginx_version="$nginx_version" http_x_forwarded_for="$http_x_forwarded_for" '
              'http_x_header="$http_x_header" uri_query="$query_string" uri_path="$uri" '
              'http_method="$request_method" response_time="$upstream_response_time" '
              'cookie="$http_cookie" request_time="$request_time" category="$sent_http_content_type" https="$https"';

Note

Use kv format instead of a raw format for the access log.

See the full list of variables at https://nginx.org/en/docs/varindex.html that you can capture in the log.

For more information about configuring ngx_http_log_module, see the official NGINX documentation at https://nginx.org/en/docs/http/ngx_http_log_module.html.

Set up NGINX error log¶

NGINX writes information about encountered issues of different severity levels to the error log. For information about setting up the NGINX error log, see https://www.nginx.com/resources/admin-guide/logging-and-monitoring/#error_log.

Set up NGINX live activity monitoring¶

NGINX Plus provides a real-time live activity monitoring interface that shows key load and performance metrics of your server infrastructure. These metrics can be represented as a RESTful JSON interface and live JSON data can be ingested into Splunk. You need to enable statistics collection in the NGINX Plus configuration file. For information about setting up live activity monitoring, see https://www.nginx.com/resources/admin-guide/Monitoring.

Configure the NGINX App Protect Security log¶

Security logs (also known as Request logs or Traffic logs) contain information on HTTP requests and responses, how App Protect processes them, and the final decision made based on the configured policy parameters. The policy configuration defines the information contained in the Security log, such as whether requests are passed, blocked or alerted, due to violations, attack signatures, and other criteria.

For information about setting up the default NGINX App Protect Security log, see the NGINX documentation at https://docs.nginx.com/waf/#security-logs.

Edit the /etc/app_protect/conf/log_default.json file and change the format from default to splunk.

For example:

{
  "filter": {
    "request_type": "illegal"
  },
  "content": {
    "format": "splunk",
    "max_request_size": "any",
    "max_message_size": "5k"
  }
}